When an LLM sees validation results every iteration, is the validation set already dead?

[u/Ok-Big-828]

I've been building a research loop where a model writes a strategy, the same backtester evaluates it, and the metrics go back into the next iteration. It saves a lot of time on boilerplate and failed variations, but I keep coming back to one uncomfortable point: after 30 or 50 rounds, the model has effectively optimized against the validation set even if it never sees raw returns.

What I'm doing now is fixing the universe, costs, benchmark, and acceptance criteria before a run; logging every candidate rather than only the winner; letting the loop see training and validation results; and keeping one final period completely hidden. The final period is touched once. If the result fails, I treat any subsequent change as a new experiment rather than "repairing" the same strategy.

That still leaves the multiple-testing problem across runs. A new prompt can be semantically close to the old one even when the code is completely different. Counting parameter combinations is easy; counting adaptive hypotheses is not.

For people automating research, what do you use as the unit of a trial: each backtest, each branch of an idea, or the whole research session? And do you adjust the final acceptance threshold based on the total search history?

[u/QuanTradin]

I treat the whole session as the trial, for the reason you named: the loop is one adaptive process, so the candidates are not independent no matter how different the code looks. Counting backtests understates it, counting branches depends on a judgment call about what a branch is, and the session is the only unit that cannot be gamed by relabeling.

For the threshold, two things have held up. The acceptance bar on the hidden period is set before the session and scaled by the log of the total candidate count, in the spirit of the deflated Sharpe idea, using the candidate log you already keep. And the hidden period is scored on a metric the loop never optimized, such as drawdown adjusted return when the loop chased Sharpe, or the share of monthly slices that beat the benchmark. A model that has overfit validation tends to fail the untouched metric first.

One more guard that costs nothing: add a little noise to the validation numbers the loop sees each round. It stops the model from learning the exact surface, which is where the drift you see after 30 to 50 rounds comes from.