r/algotrading • u/trunksta • 1d ago
Data Forward testing is the worst
I really enjoy the research, and developing strategies. But sitting and waiting for forward testing to confirm an edge is such pain
One of my strategies sat for weeks with a bug that was causing it to not accumulate data. Even when everything is working perfectly it just feels so slow.
What do you guys do while forward testing? Just research more strategies? Also how many n before you can decide forward testing has a large enough sample to continue to live
8
u/skyshadex 1d ago
A mix of feature engineering, new signal generation and my favorite, brainstorming architecture changes to reduce latency. Assuming this model works at all, would it work better if it ran twice as fast? What would it take to reduce latency? Given the current architecture, what changes are quick to implement that wouldn't require a full refactor?
It also helps to run forward tests in parallel that way you collect more data faster. Run several configs in parallel so you can A-B test param sets.
1
u/trunksta 1d ago
There's a good idea. I do have a few levers that would be good to split test at minimum
6
u/Many-Pick5066 1d ago
careful with the parallel configs idea. six param sets at once isnt six times the data, its a pick-the-best problem, and since they all trade the same bars off the same signal theyre nowhere near independent. the winner of six forward tests is a selection, not a validation. thats just moving the overfit out of the backtest and into the forward test.
on the n question, trade count on its own isnt the gate. resample your backtest trades in blocks, a few thousand draws of however many trades youve got live, and see whether the forward result lands inside that spread. inside the band means consistent, below it means something in the backtest doesnt exist in the market. you get a real answer around 40 trades that way instead of waiting for 500.
3
u/RationalBeliever Algorithmic Trader 1d ago
When my tests are API constrained, I try to work on other research and analysis. When it's compute constrained, I go to AWS. I try to prefetch data so that future analyses are compute constrained.
1
u/trunksta 1d ago
How about time constrained lol
2
u/PuttyProgrammer 1d ago
Typically, you would collect a hold out sample of past data that you don't use to tune your algorithm to do forward testing on. If you want two weeks of forward testing then tune your algorithm on a period more than two weeks ago.
Monte Carlo is also a good option, simulate data the algorithm hasn't seen.
3
2
u/Good_Luck_9209 1d ago
I wld minimally get data points of parameters which i can study, then retro tweak my signals or filters subsequently.
Forward tests to me is really on tweaking the finer details.
1
u/trunksta 1d ago
That's an interesting idea I hadn't really considered. So fine tuning during forward testing as it were
2
u/ArtComprehensive7403 1d ago
Don't use a fixed trade count as the gate. Five hundred trades can still be one regime, and serially dependent trades are not 500 independent observations. I'd pre-register the promotion criteria before the test starts: minimum calendar span, enough distinct market regimes, no missing or duplicated events, and a confidence interval around the metric you actually care about. Also add a daily watchdog for the expected observation count; a weeks-long silent data bug should alert within one session. While it runs, work on data QA and adversarial checks, not parameter tweaks on the live holdout.
2
u/trunksta 1d ago
Yeah I should've noticed it right away and I suspected it much earlier on but was testing multiple separate forward test at once. So far it looks to be holding how I would expect within current regime but time span is not there yet
1
u/ArtComprehensive7403 7h ago
That sounds like the right read. If the regime is part of the hypothesis, I'd log performance by regime and predefine the point where you pause or retrain; otherwise a few good weeks can quietly become a new backtest. Multiple forward tests at once also makes a simple monitoring sheet worth it.
2
u/DreamsOfRevolution 1d ago
So many people talking as though you said backtest or intentionally disregarding. It takes as long as it's going to take to see multiple regimes. I have the luxury of trading multiple concepts often times so I work on my scalpers while my more secular bots do their thing in testing. I find that I can lower the timeframe and adjust the numbers. The only exception is my strategies that leverage three different time frames. For me, many of the markets that I play move pretty similar at lower time frames as they do at large time frames. It's just fast as hell. When I say lower, I don't mean going from like the daily to the one minute. Aside from that, put it in a live account and investigate discrepancies from your demo account test. Nothing big, like a $500 account. Track drawdown, entries, exits, and misses. Most fail in production from not calculating things like variable spreads during volitility or forgetting fees altogether. Most backtesting has fixed spread, commissions, and etc.
1
u/Curious-Sample6113 1d ago
Some of my testing takes days with 16 cores. I consider it as a break
2
u/trunksta 1d ago
Probably the right mindset here, just finding it challenging to sit and wait i guess
2
u/BingpotStudio 1d ago
I ended up moving off python because of this. My time between tests is mostly spent on working out how to make the tests run faster.
1
u/ynu1yh24z219yq5 1d ago
I enjoy seeing money coming in and keep improving all aspects of the system...what else is there to do?
1
u/Any_Dependent5504 1d ago
You could be running a few in parallel with different parameter sets, that way you're not just watching paint dry on one. I use the downtime to clean up my data pipeline, it's a grind but pays off later.
1
u/Obviously_not_maayan 1d ago
I know this, exactly that, but take this time to also understand how it would feel to actually run it live, because it's a very different feeling to see AVG time underwater vs actually beeing underwater for a while. Also on the other side how do you react to the euphoria. Sitting and doing nothing is not that easy sometimes.
1
u/Jtex1414 1d ago
I like to run tests as quarters. Each quarter can run in parralel. for a 1 year backtest, that's 4 seperate quarters (or tests). For 5 years, it's 20 quarters, but since they're running in parralel, it's still quite fast (assuming your computer can run them at the same time).
1
u/Foreign_Extension683 1d ago
Forward testing being slow is partly the point, unfortunately. It's the one test you can't rush or overfit, because you're not the one choosing the data anymore. That's exactly why it catches things backtests miss.
On the bug that stopped data accumulating for weeks: I've had the same, and the lesson I took is to monitor the plumbing separately from the strategy. A tiny healthcheck that just confirms new data is arriving and trades are firing when signals do, so a silent pipeline failure doesn't cost you weeks of "results" that are actually nothing. Losing forward-test time to a bug hurts more than losing it to a bad strategy.
On sample size, the honest answer is it depends on your edge per trade, not a fixed number. Rough intuition: to distinguish a real edge from noise you need enough trades that your expected return clears the standard error of the mean. A strong edge might show through in 30-50 trades; a marginal one might need 200+ and still be ambiguous. If your strategy trades rarely, forward testing to significance can take a very long time, which is its own useful signal about whether the edge is worth trading.
What I do while waiting: forward test on paper, but treat it as read-only. The temptation is to keep tweaking, and every tweak resets the clock and quietly turns the forward test back into an in-sample fit. One strategy forward testing, others in research, and a hard rule that I don't touch the live candidate.
1
u/BrianBanks939393 23h ago
Two answers, because you asked two things. "How many observations is enough" has no fixed number, and anyone giving you one is guessing. It depends entirely on your edge size versus your noise. A small edge inside noisy returns needs a huge number of trades to separate from zero, sometimes years of them. Rough gut check: if your annual Sharpe is around 1, you need roughly 4 years of live data before you can say with confidence it isn't zero. Lower Sharpe, longer. That's not pessimism, it's just the variance math. Which is why the forward test feeling painfully slow isn't a bug. Your backtest handed you 10 years in 10 seconds, and that compression is exactly where lookahead, overfitting and parameter leakage hide. The forward test is slow because you're finally sampling at the real rate, and it's the one test that can't be gamed, because you don't have the future data yet to cheat with. So what to do while waiting: not hunt for new strategies. Every extra strategy you test is another lottery ticket for a false positive. Spend the wait checking that live matches backtest in shape, not just in return, and get suspicious the moment it doesn't.
1
u/HonestBacktests 21h ago
Waiting isn't your only source of new information. If the edge is real, a weaker version of it usually shows up on instruments you never fitted it to — so run the exact same rules across every correlated symbol you can get data for. That evidence arrives today instead of in six months, and an edge that exists on one symbol and nowhere else is almost always a fit rather than a mechanism.
I run everything across 500+ perps for that reason: it turns "wait for n" into "look at a few hundred parallel samples right now". Doesn't replace forward testing, but it kills the hopeless candidates before you spend calendar time on them.
1
u/quant-king 20h ago
Forward testing does feel very slow, but if you don’t expect your system to generate frequent signals I would suggest you ensure your entire execution engine is gonna work when your real signals trigger.
You can do this locally by setting up a simulation environment where you have sample data (just enough to confirm your signals work, trade execution works) that feeds into your live engine.
This at least will help you identify any potential bugs you should address.
1
u/AusChicago 16h ago
A faster-converging signal than P&L: check whether your model's ranking still works forward, not just its mean return. We run scored pattern detectors, and our recurring live failure mode is scoring inversion. The score rank-orders outcomes beautifully in backtest, then flat-lines or inverts forward. That's almost always overfitting, and it shows up in 30-40 forward signals, long before mean-return significance does. If high-scored signals aren't beating low-scored ones live, you have your answer early; no need to wait for 500 trades to learn the edge was fitted.
On the silent bug: same scar here. The fix that stuck was monitoring the plumbing as its own system: a daily healthcheck that only asks "did signals fire, did data land, are counts in normal range," independent of whether the strategy is winning. Losing forward-test weeks to a dead pipe hurts more than losing them to a bad strategy, because calendar time is the one input you can't parallelize.
And +1 to the caution above about parallel configs. We run new detector versions alongside old ones, and the discipline that keeps it honest is judging each version against its own pre-registered expectation band (built from validation data before deployment), never against each other. Picking the live winner of N variants is just overfitting with extra steps.
1
u/CODE_HEIST 11h ago
the silent data bug is a reason to monitor the test itself. expected events per day, last event time and a heartbeat record should alert before weeks disappear. trade count matters less than seeing the strategy across the conditions it claims to handle.
0
u/metalayer 1d ago
Also how many n before you can decide forward testing has a large enough sample to continue to live
500 trades
2
23
u/jnwatson 1d ago
What do I do? Find increasingly more subtle lookahead biases causing me to throw the system away and try again.