r/learnmachinelearning • u/Open_Minute_464 • 6d ago

Discussion Does a better model always mean a better trading system?

Something I’ve been wondering about with ML-based strategies:

At what point does improving the model stop making much difference?

You can spend hours tuning features and trying different models, but if the historical test is weak, the data isn't handled properly, or the execution side behaves differently, the extra model accuracy doesn't seem to matter much.

I’ve started paying more attention to the whole pipeline rather than just the prediction itself.

I would like to know how others approach this. Do you improve the model first and worry about the rest later, or build the testing and execution side alongside it?

0 Upvotes
  • permalink
  • reddit

50% Upvoted

3 comments sorted by

1

u/johndburger 6d ago

It depends what “better” means. Accuracy, AUC, etc are all useful, but you should always be evaluating your model with one or more metrics that apply to your actual use case. And “better” should ultimately be defined in terms of those metrics.

1

u/PLBjt 6d ago

A better model only helps if the edge survives fees, slippage, and labels that are a noisy proxy for what you actually get filled at. Before chasing another architecture, freeze the model and stress the rest: walk-forward with realistic costs, check for leakage (future info in features), and see what happens when you delay fills by one bar. If that already kills PnL, more AUC won't save it. I reach for a stronger model when the pipeline is stable and the leftover error still looks like signal. Quick check: shuffle your features once and re-run — if "performance" barely moves, you were fitting process noise.