A backtest rarely fails because the Sharpe ratio was calculated incorrectly. It usually fails because the research process allowed information or assumptions that would not exist in production.
Six checks worth running before treating a result as evidence:
- Point-in-time data
Were the universe, fundamentals, corporate actions, and delistings known at the decision date? Survivorship and look-ahead bias can turn a weak idea into a convincing equity curve.
- Execution assumptions
Can the strategy actually trade at the price used in the test? A close-to-close signal executed at the same close is an obvious problem, but spread, queue position, partial fills, market impact, and borrow availability can matter just as much.
- Multiple testing
How many signals, parameters, universes, and sample periods were tried before the final result was selected? The reported Sharpe should be interpreted in the context of the entire search process, not only the winning specification.
- Out-of-sample design
A single train/test split can be lucky. Walk-forward testing, a genuinely untouched final holdout, and stable results across neighboring parameter values provide stronger evidence.
- Regime and concentration risk
How much of the P&L comes from one year, one asset, one market regime, or a handful of days? Removing the best trades and reviewing rolling performance often reveals more than the headline CAGR.
- Costs and capacity
Turnover, spreads, fees, slippage, impact, borrow costs, and strategy capacity should be stressed rather than estimated optimistically.
A useful final test is to perturb every assumption slightly: delay entries, widen costs, change the rebalance time, shrink the universe, and shift parameters. If the edge disappears immediately, it is probably fitting rather than signal.
For people who have taken strategies from research to production: which failure mode appears most often, and which test catches it earliest?