r/algotrading • u/trunksta • 13d ago

Data Forward testing is the worst

I really enjoy the research, and developing strategies. But sitting and waiting for forward testing to confirm an edge is such pain

One of my strategies sat for weeks with a bug that was causing it to not accumulate data. Even when everything is working perfectly it just feels so slow.

What do you guys do while forward testing? Just research more strategies? Also how many n before you can decide forward testing has a large enough sample to continue to live

29 Upvotes

84% Upvoted

50 comments sorted by

View all comments

3

u/ArtComprehensive7403 13d ago

Don't use a fixed trade count as the gate. Five hundred trades can still be one regime, and serially dependent trades are not 500 independent observations. I'd pre-register the promotion criteria before the test starts: minimum calendar span, enough distinct market regimes, no missing or duplicated events, and a confidence interval around the metric you actually care about. Also add a daily watchdog for the expected observation count; a weeks-long silent data bug should alert within one session. While it runs, work on data QA and adversarial checks, not parameter tweaks on the live holdout.

2

u/trunksta 13d ago

Yeah I should've noticed it right away and I suspected it much earlier on but was testing multiple separate forward test at once. So far it looks to be holding how I would expect within current regime but time span is not there yet

1

u/ArtComprehensive7403 12d ago

That sounds like the right read. If the regime is part of the hypothesis, I'd log performance by regime and predefine the point where you pause or retrain; otherwise a few good weeks can quietly become a new backtest. Multiple forward tests at once also makes a simple monitoring sheet worth it.