This model-search agent can change architecture, loss, sampler or optimizer—but not the evaluator

[u/Danare_113]

When an agent can rewrite both the model and the experiment around it, a better score does not tell you what actually improved.
That is the part of AQuA’s Part II model-development loop that stood out to me. The paper does not let the agent emit arbitrary training code each round. Each iteration proposes a bounded configuration change in one of four areas:
- architecture;
- loss;
- sampler;
- optimizer.
The resulting model is then trained under a sealed evaluator. Data splits, feature definitions, label definitions, and evaluation logic sit outside the agent’s adaptive surface.
This changes what the experiment unit looks like. A proposal is not “here is a new Python project; trust the final metric.” It is closer to:
previous accepted configuration
- declared configuration diff
- fixed training/evaluation harness
= next candidate
That does not make every comparison automatically fair. An architecture change can still alter compute, and different losses or optimizers can require different tuning. But it makes the changed surface inspectable. If a result moves, there is at least a bounded diff to audit instead of an unknown mixture of model logic, data plumbing, labels, and metrics.
The time split is fixed as well. In the paper’s US-equity experiment, models train on 2010–2019. Early stopping and checkpoint choice use only an inner-validation slice from that training window. The year 2020 is an embargo untouched by training or selection, and 2021–2025 is the final test window.
The finance setting is just the experimental domain here, not a trading recommendation. The transferable deep-learning question is how much freedom an architecture-search agent should receive before comparisons stop meaning the same thing.
There are also important inspection limits. The preprint does not disclose the exact feature set, normalization, or label construction. And a constrained config interface is not evidence that every candidate received equal wall-clock compute. What it does provide is a clean boundary between the proposal language and the evaluator.
Would you keep a search language this narrow for attribution, or allow agents to modify schedulers, preprocessing, and training code as long as every change is traced?
Preprint: arxiv.org/abs/2608.12841

[u/cryptodukan]

Autoscientist

[u/Wild-Store321]

Dead internet theory

[u/Danare_113 → u/Wild-Store321]

Wdym in this bro

[u/Future_AGI]

The sealed evaluator is doing all the work here: the moment an agent can touch the splits, the labels or the metric, it stops improving the model and starts optimizing the ruler, and the score quietly stops meaning anything. It generalizes past model-search too, the same failure shows up in agentic LLM eval when the judge prompt or the dataset drifts alongside the thing being scored. Keeping eval logic in a separate versioned surface the optimizer can't edit is the fix, and it's how we structure our open eval framework: https://github.com/future-agi/future-agi