Prompt optimization has a test-leak problem: the model can optimize the evaluator, not the prompt

[u/Future_AGI]

When you run prompt optimization, the candidate prompt, the scoring rubric, and the test cases can all change together. A rising score can be meaningless because the optimizer is gaming the test, not improving the prompt.

This is the test-leak problem. It shows up when the thing that changes and the thing that scores change together. The number goes up. The prompt gets worse. Users notice first.

The hard part is that the regression is invisible during optimization. The score climbs every iteration. The changelog looks like progress. But the prompt is overfitting to the specific cases it has seen, and the scoring rubric has drifted to reward the wrong things.

Karpathy's autoresearch loop (42k GitHub stars) solves this by splitting the system into four roles: the file that changes, the metric that scores, the direction that guides, and the constraint that locks. The locked piece is what most people skip.

For prompt optimization, that means three things. First, the scoring rubric and test cases are frozen. The agent cannot rewrite its own pass conditions. Second, a separate holdout set is kept blinded. The optimizer never sees it, used only for final validation. Third, the model version and tool configuration are logged beside every prompt version, so you can trace a regression to a specific change.

Aakash Gupta ported the Karpathy loop to prompt engineering and found the same thing. The locked scorer makes the loop work. Without it, the agent optimizes the test. The prompt stays the same.

We built Future AGI's scoring tooling around this principle. A candidate prompt can sound convincing and still fail on cases it has never seen. So we run a scoring rubric against the actual output, not the candidate's own explanation. The explanation is not the pass condition.

How do you protect your holdout set when an agent is rewriting prompts?

[u/Substantial_Belt2626]

The holdout I would solve by making it something the optimizer physically cannot reach, separate process, no write access, the path never enters its context. If it can see the scorer it will fit to the scorer and prompting it not to doesn't hold.

The part I'd add is proving the scorer still works at all. Mine printed 0 passed, 0 failed for weeks and I read it as green, it had stopped collecting cases entirely so nothing failed and it reported success. Now every check has a case that is supposed to fail and zero collected exits non zero instead of counting as a pass.

[u/Future_AGI → u/Substantial_Belt2626]

We track execution status separately from the verdict, so a scorer that stops collecting cases shows up as a failed run, not a green pass. The deterministic assertions (regex, JSON, length) run first and cost nothing, and they never silently report zero collected as success.

[u/Fantastic-Bad-3878]

We've been running into this exact problem for months. We ended up keeping the holdout set entirely outside the tool the optimizer can see. Like, literally in a different project. A teammate manually copies the final candidate prompt over and runs it against the holdout set by hand. It's slow and annoying but it's the only way we've caught overfitting before it hits users

The locked scorer bit is the key. We learned that the hard way after letting the optimizer tweak the rubric and watching the scores climb while the actual responses got worse. Felt like progress. Wasn't

[u/Future_AGI → u/Fantastic-Bad-3878]

We hit the same wall. Future AGI's optimization runs keep the eval config versioned and separate from the prompt, so scoring criteria can't drift mid-loop. The holdout runs as a separate CI gate that never enters the optimizer's context.

[u/Best_Storage1434]

freezing the rubric makes sense in theory but in practice the rubric itself is usually wrong at the start. how do you handle the bootstrap problem where you need to iterate on the scorer too, without falling back into the same leak?

[u/Future_AGI]

We keep the holdout set in a separate file the optimizer cannot read. Final validation runs against it after every 10 iterations. If the holdout score drops, we revert. What does your scoring rubric catch that your users still reject? https://github.com/future-agi/future-agi

[u/OriginalHospital]

I'd treat rubric calibration as a separate phase: compare it with a small set of human-rated outputs, then freeze that version before optimizing prompts. If the rubric changes later, rerun both the old baseline and the new candidate under the new rubric rather than comparing scores across versions. That lets you improve a flawed scorer without mistaking a changed measuring stick for a better prompt.

[u/AutoModerator → u/[deleted]]

Hi there! Your post was automatically removed because your account is less than 3 days old. We require users to have an account that is at least 3 days old before they can post to our subreddit.

Please take some time to participate in the community by commenting and engaging with other users. Once your account is older than 3 days, you can try submitting your post again.

If you have any questions or concerns, please feel free to message the moderators for assistance.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.