Backtesting, Overfitting and the Evidence Problem

A backtest is a simulation of what a set of trading rules would have earned on past data. It is the cheapest experiment in finance, and the most often abused. A researcher with a laptop can test thousands of rule variations in an afternoon, and some will look excellent by chance. Campbell Harvey, Yan Liu and Heqing Zhu counted several hundred factors proposed in the academic literature to explain stock returns and argued that, given how many had been tried, a new factor should clear a t-statistic of about 3.0, not the conventional 2.0 ("…and the Cross-Section of Expected Returns", Review of Financial Studies 29(1), 2016, 5–68; 316 factors examined; t-ratio cut-off of about 3.0 — NBER WP 20592 abstract and journal record). David McLean and Jeffrey Pontiff studied 97 published anomalies and found their returns were about 26% lower out of sample and about 58% lower after publication ("Does Academic Research Destroy Stock Return Predictability?", Journal of Finance, 2016; figures checked against the paper in S04). Those are the results for peer-reviewed research. Vendor backtests, course sellers' screenshots and "AI bot" marketing face no such review (blueprint rule 4, categories (d) and (e)). This section explains how backtests mislead: look-ahead bias, survivorship, data snooping and selection, unrealistic fills, ignored costs and borrow, regime dependence. It covers the statistical tools that correct for them: the multiple-testing haircut, the deflated Sharpe ratio and the probability of backtest overfitting of David Bailey and Marcos López de Prado, walk-forward and out-of-sample tests, and paper versus live records. It also sets out how this knowledge base grades evidence. The companion worked example simulates 1,000 strategies with no edge at all and finds the best one shows a Sharpe ratio of 1.82 over five years and a t-statistic above 4, then loses money out of sample. A second file gives 20 questions to ask of any backtest.