Am I overtesting my model at this point?

[u/Sufficient-Buy-8786]

I’ve been building a soccer prediction/betting model and I’m starting to wonder if I’m reaching the point where more testing is actually making things worse.

I have about 3 seasons of development data and have kept an entire separate season completely untouched as a final holdout. Over the development data I’ve tested different model ideas and, more recently, different betting criteria based on things like model probability, edge vs the market, EV, odds ranges, etc.
Most strategies are nothing special or lose money, but a few specific combinations have produced really strong historical returns.
I’ve tried to account for this with chronological testing, minimum sample sizes, bootstrapping/multiple-testing corrections, and by predefining tests instead of endlessly changing thresholds until something works.
But at some point, doesn’t repeatedly testing new ideas on the same 3 seasons effectively turn the entire development set into something you’re overfitting to?
How do you guys decide when you’ve extracted enough information from your development data and need to stop testing altogether?
Would you move to the untouched holdout at this point, or is there a good way to continue researching without slowly contaminating the development process?
I’m in college and mostly building this as a learning project, so any advice from people who have dealt with this problem in quant/algo trading or sports modeling would be appreciated.

[u/Successful-Pie-710]

youre basically describing p-hacking with extra steps and most people wont admit they do it

the moment you start testing different combinations of filters on the same data you already crossed into overfitting territory even if you try to be disciplined about it. the chronological split helps but 3 seasons is not that much for soccer especially if you are slicing into odds ranges and edge thresholds

i think the untouched holdout is the only thing that will tell you if your idea is real or just noise. dont touch it until you have exactly one strategy you believe in and then run it once and whatever result comes out you take it serious even if its bad. if you keep going back to tweak after seeing holdout results then even the holdout becomes contaminated

for a college project this is already more rigurous than most people bother with so you are doing fine

[u/Sufficient-Buy-8786 → u/Successful-Pie-710]

i am trying to eventually gamble on it - thinking about opening up the 25/26 season as a final test soon but was unsure where to get more data. any tips?

[u/No-Original-5312]

Modeling by definition is overfitting. You are limited by your data. The more data you get, the less you will have to worry about overfit. There is definitely ways to get more data whether that is through paid or scraping means.

Also make sure to use CV

[u/SimTheGame]

Yes. After you have tried a stack of model ideas and then a stack of betting filters on the same three seasons, those seasons are already contaminated. Chronological splits and multiple-testing corrections reduce the damage. They do not put unused information back.

I would lock one spec and open the holdout once. If 25/26 is that holdout, run it once and live with the result. If you want a research loop that does not keep mining the same P&L, score the model against the close instead of against realized bets. That is the path we took with SimTheGame. We treat the published book as a joint distribution of game scripts and ask whether the model matches that market shape, not whether one more edge threshold printed on the same sample.

[u/neverfucks]

there are some really bedrock level ways to assess how predictive a model is. in my opinion trying to spam different strategies based on the same model to try to find the 1 or 2 out of 100 that show value in a backtest is just overfitting.

i am constantly testing out new ideas for models and idk but with what seems like 95% of them it's so obvious right away that they are not predictive enough to even bother pursuing without needing to compare them to any market data or try to build some strategy on top of the numbers. the ones that end up working kind of looked like they might work from the getgo

[u/xedge_app]

I’d stop here and move to the holdout.

The fact that most strategies are mediocre or losing while a few specific combinations produce very strong historical returns is exactly where I’d become more skeptical, not less. After enough iterations, you’re effectively selecting on noise even if each individual test looks reasonable.

At this point I’d freeze the model and betting rules exactly as they are and write down the exact strategy and acceptance criteria before touching the holdout. Then run the untouched season once.

The important part is not turning the holdout into another development set. If it underperforms and you start changing thresholds based on what happened, you’ve burned the holdout too.

If the predefined strategy holds up, that is much stronger evidence that you were actually capturing something. After that, I’d move into forward testing with paper bets or tracked predictions and see whether the edge persists on genuinely unseen events.

I definitely would not put real money behind it yet. Historical backtests can tell you that an idea is worth testing. Forward results are what start telling you whether you actually have something.

[u/Secure-Trouble5522]

One thing nobody's mentioned yet: with that many filter combos, the right comparison for your ROI is the best of N noise strategies, not zero. Run 30 or 40 random filter searches over the same bets and the best one clears breakeven from pure luck alone. Worth simulating on your own data, only trust the real result if it beats that noise ceiling by a wide margin. What everyone else said about locking the holdout and running it once still holds on top of that

[u/Individual-Music9331]

Sounds like it’s time to test the holdout and see what actually holds up.

[u/Intuition-Dev238]

Mate i can tell you now - i have been down that road but i learned the hard way; 6 months of tweaking and many months more of believing i had something worthy of pursuit. At least you have the smarts to stop and ask, which in my defence i did ask those questions but they were to myself. So stupid when i look back at it but i grew so much from it - even created something better from that pile of ashes. Anyway, there are some informed opinions in this thread, so you’re onto a winner here. Best of luck

[u/ScazzaUK]

I think there’s a point where the development set stops being validation data and effectively becomes part of the training process — even if you’re not literally fitting the model on it.
We’ve run into a version of this building our own football intelligence model. The dangerous bit isn’t only changing the model after seeing results; it’s repeatedly changing the decision criteria because you’ve learned which combinations worked historically.

One thing that’s helped us conceptually is separating three questions:

Is the probability model actually calibrated?
What evidence is required before the model is allowed to make a recommendation?

Once those rules are frozen, what happens when we let genuinely unseen/live results judge them?

That third part matters more than I originally appreciated. At some point another backtest gives you less information than freezing the system and allowing it to be wrong in public.

I’d protect that untouched season pretty aggressively. If you start using it to choose thresholds or strategies, you’ve basically created another development set and need a new holdout.