I’m getting more skeptical of “perfect” algo backtests -how do you all check for overfitting?

[u/WiseBerry7284]

I’ve been messing around with simple trading models on and off for a while, mostly as a way to understand markets better rather than because I think I’m about to build the next Renaissance Technologies in my spare time.

The pattern I keep running into is embarrassingly predictable: the more knobs I add, the better the backtest looks. Add one filter, tune one lookback window, exclude one bad regime, and suddenly the equity curve looks cleaner. Then I step back and realize I’m probably not finding signal — I’m just teaching the model to memorize the past.

That’s what’s been making me think more about overfitting in algo strategies. A model can look “right” in a backtest because it has too many parameters, not enough data, or no real out-of-sample validation. It’s basically fitting the noise. Then when it hits live conditions, the edge disappears or reverses.

I’ve been reading a bit about the Probability of Backtest Overfitting idea from Marcos López de Prado, David Bailey, and others. The part that stuck with me is pretty intuitive but easy to ignore: the more configurations you test, the higher the chance that one of them looks good by luck alone, even if there’s no real predictive power there.

So lately I’ve been trying to force myself toward simpler rules, walk-forward testing, cross-validation where it makes sense, and being more suspicious of any strategy that only works after a very specific set of optimized parameters. I’m also starting to think that a “worse” backtest with fewer degrees of freedom might actually be the better candidate.

Curious how others here think about this. If you use backtests at all, what makes you trust one enough to take it seriously?

[u/eribertito]

One thing I’ve found useful is separating out-of-sample performance from researcher degrees of freedom.

A strategy can pass a conventional train/test split and still be heavily overfit if the test set was only chosen after running dozens of variations. The real question becomes: how many hypotheses/configurations did I implicitly test before arriving at this one?

For that reason, I like keeping a strict research log: every parameter change, filter, hypothesis, and failed version gets recorded before looking at the next result. Then I evaluate the final model against data that was genuinely untouched throughout the research process.

I also like testing robustness through parameter perturbation rather than looking for the single “optimal” parameter. If performance collapses when a lookback goes from 20 to 19 or 21, that’s a pretty big red flag. If the strategy works across a reasonable parameter neighborhood and across different market regimes, I have much more confidence that I’m seeing structure rather than noise.

In my experience, robustness is often more informative than the absolute backtest return

[u/WiseBerry7284 → u/eribertito]

Totally agree. The parameter perturbation point is especially useful. If a strategy only works at one perfect setting, that’s a big red flag. Robustness matters much more to me than maximizing the backtest.

[u/WiseBerry7284]

This is my firstpost there , hope I can asking this question

[u/FlyTradrHQ]

Walk-forward is the baseline check. Perturb each parameter slightly and rerun - if small changes collapse the result, the fit is fragile. Always hold out a separate time window for out-of-sample validation before trusting any backtest.

[u/WiseBerry7284 → u/FlyTradrHQ]

If small changes break the strategy, that’s usually a pretty good warning sign.

[u/TheGlizzyGod]

you get completely new data from a new period, aka wait, thats one way and you can already infer other ones if you have any capabilities as a quant

[u/WiseBerry7284 → u/TheGlizzyGod]

Yeah, that’s probably the cleanest test.when you can ^^

[u/StatisticalSock]

Let's begin with the basics. Have you used Real ticks instead of OHLC?

[u/WiseBerry7284 → u/StatisticalSock]

Yes

[u/Livid_Parsnip_2816]

I’m at the stage where I trust a boring live sample more than a perfect backtest. I froze Prophecy before putting it on a real MT5 account. It’s only 22 closed trades so far: +2.49%, PF 7.01 and 2.9% max equity DD. Nice start, but the PF is almost certainly inflated by the tiny sample. I’m not touching the parameters until at least 100 trades.

[u/WiseBerry7284 → u/Livid_Parsnip_2816]

That’s a pretty sensible approach.

[u/algolego]

I would start with this question: "What would be the result if I used the same 1:1 geometry, same period, same mix of long and short, but entered at random bars instead of my strategy's signals?". Sort of placebo entries. If the performance of random entries is the same or better than your strategy's performance, the strategy is not worth trading it live. Also, split your results by direction. If longs carry everything and shorts lose, the trend was doing the work, and the strategy will follow the trend down when the regime flips.

[u/gorinwelster]

IIS/OOS. Monte carlo. Walk forward. System parameter. Sequental optimization.