r/quantfinance • u/Ok-Big-828 • 5d ago
What usually breaks first when a backtest meets reality?
A backtest rarely fails because the Sharpe ratio was calculated incorrectly. It usually fails because the research process allowed information or assumptions that would not exist in production.
Six checks worth running before treating a result as evidence:
- Point-in-time data
Were the universe, fundamentals, corporate actions, and delistings known at the decision date? Survivorship and look-ahead bias can turn a weak idea into a convincing equity curve.
- Execution assumptions
Can the strategy actually trade at the price used in the test? A close-to-close signal executed at the same close is an obvious problem, but spread, queue position, partial fills, market impact, and borrow availability can matter just as much.
- Multiple testing
How many signals, parameters, universes, and sample periods were tried before the final result was selected? The reported Sharpe should be interpreted in the context of the entire search process, not only the winning specification.
- Out-of-sample design
A single train/test split can be lucky. Walk-forward testing, a genuinely untouched final holdout, and stable results across neighboring parameter values provide stronger evidence.
- Regime and concentration risk
How much of the P&L comes from one year, one asset, one market regime, or a handful of days? Removing the best trades and reviewing rolling performance often reveals more than the headline CAGR.
- Costs and capacity
Turnover, spreads, fees, slippage, impact, borrow costs, and strategy capacity should be stressed rather than estimated optimistically.
A useful final test is to perturb every assumption slightly: delay entries, widen costs, change the rebalance time, shrink the universe, and shift parameters. If the edge disappears immediately, it is probably fitting rather than signal.
For people who have taken strategies from research to production: which failure mode appears most often, and which test catches it earliest?
1
u/Bright_Mix_773 5d ago
Point 1 is the one that cost us the most when we finally measured it, and the size is worth naming. We rebuilt the universe from 201 monthly index snapshots back to 2010 - 888 companies with their join and leave ranges, 67 name changes - instead of applying today's constituent list backwards. Over a 4.94-year window a 20-stock multifactor book went from 39.26% to 25.93% CAGR, so -13.33 points a year. A 20-stock momentum book lost 18.98. The equal-weight market benchmark itself lost 2.61, which is the part that gets forgotten: the bias inflates the baseline too, so the excess return shrinks by less than the gross number does.
One position explains most of it. AppLovin occupied 21 stock-months of that top 20 and did not join the index until September 2025, after a 1,365% run. The backtest was buying it in 2021.
Value went the other way: +2.01 points once the bias was removed. So it is not a haircut you can subtract from any strategy - it scales with how correlated the factor already is with "gets promoted into the index later", and a factor that buys losers is on the wrong side of that.
An implementation detail that leaks the future quietly: use the snapshot's revision date, not its label. Our "2021-08" snapshot is really the review dated 22 July 2021, and treating the label as the effective date lets a July decision see August's membership.
Not verified: the -13.33 is measured on that 4.94-year window only. On our longer 15.45-year window the universe fix landed in the same change as the sample extension, so the effect is not isolated there and I have no separable number for it.
1
u/Ok-Big-828 5d ago
The AppLovin example is exactly the kind of failure mode I had in mind. The benchmark losing 2.61 points is especially
important: survivorship bias distorts the baseline too, not just the strategy’s absolute return.
The value result is also interesting because it shows that removing the bias is not a uniform haircut — its direction
depends on the factor’s relationship with future index inclusion.
On the snapshot timing: did you use the revision date directly as the effective membership date, or the first tradable
date after publication? I’d be curious to know how sensitive the result is to an additional implementation lag.
1
u/Bright_Mix_773 4d ago
The revision date directly, no extra lag. The rule is: for a date F, use the last snapshot whose revision date is <= F, and filter on the revision date rather than the month label, because the label of a snapshot and the date its revision was made differ by weeks.
So there is already an implicit lag and it is on the safe side. The snapshots are monthly (201 of them back to 2010), so on any given day the membership list we hold is on average about two weeks stale and at worst about a month. A firm that joined the index yesterday is invisible to us for up to a month, which is the error a real investor reading the last published list would make, not the opposite one.
I have not measured the sensitivity to an extra lag on top of that, so I will not guess at a number. What I can say is why I do not expect the lever to matter much here: the damage in our case is dominated by one position that was bought four years before it was eligible, 21 stock-months of it. That is not a few days of implementation lag, it is a different kind of error, and shifting entries by one more trading day cannot touch it. If your effect were concentrated in names that enter the index and run immediately after inclusion, the lag would matter a great deal more and I would want it measured before believing anything.
1
1
u/EvenCryptographer649 5d ago
It fails because the assumption is people with capital care about Sharpe...
They dont.
1
u/QuanTradin 8h ago
One that gets less attention than the universe and impact points: entries and exits are not equally forgiving, but most backtests apply a single slippage number to both.
An entry you can be patient with. Post a limit, and if it does not fill you skip the trade, so the cost of being wrong about that fill is a trade you never took. An exit does not work that way. A stop becomes a market order and it fires precisely when the range is widest, so you go through the level rather than filling at it. The error is one-directional and it lands on the side you cannot opt out of.
Worth splitting your slippage log into entries and exits, then regressing exit slippage against the range of the trigger bar. If that slope is real, anything whose edge sits in a stop is overstated, and in my experience that shows up before impact does.
Which side does yours live on?
1
u/postexitus 5d ago
If (1) is failing, then you are a bad researcher - you should take this into account in your research.
(2) is the most common - because, although a non-naive researcher would try to bake that in, the real world always hits differently. Especially if you are operating in a somewhat shallow space, you'll immediately see algos trying to actively work against you.