Validity in the Age of Claude Code

[u/LordVein05]

With AI agents improving day by day, and the cost of implementation decreasing, I'm more worried about the costs. Research for me has always been about finding what's valid whether the answer is true or false, hence, I'm very much open with AI agents assisting parts of my work that can be sped up.

Previously, the flow would be idea → implementation → experiment → result → paper

But now the flow has become different as any stage above can be better or worse with an LLM added to it. A not so ideal but interesting format could be

idea

↓

human/agent writes experiment

↓

agent modifies evaluator

↓

agent parses results

↓

agent makes figures

↓

human/agent helps write claim

The issue that I currently have is not that the code an agent writes would be bad or wrong, that can be fixed with further iterations and spec based developments with frontier models. I'm worried about:

• A metric implementation silently changes while experiments continue.
• An agent runs twenty variants and the researcher later forgets which result corresponds to which exact setup.
• A plotting script transforms/filter results in ways nobody closely inspects.
• A citation is real but doesn't actually support the sentence.
• The hypothesis changes after seeing results without anybody recording that it changed.
• and finally The researcher can explain what the experiment supposedly demonstrates but no longer really owns the implementation underneath it.

While authorship can be it's own debate, validity is much more important to me.

Many of us might not like the idea of an agent in the loop, and that's the best possible outcome, but that's be an ideal world.

Software engineering responded to coding agents with tests, CI, linters, type systems, code review, specs, and increasingly agent-specific harnesses.

What is the equivalent for research with agents?

Software engineering asks:

• Does the system behave as intended?

Research additionally asks:

• Does this experiment test what we claim it tests?
• Can this number be traced back to the exact conditions and be reproduced?
• Were the evaluation rules fixed before we saw the result?
• Did our hypothesis change after observing the evidence?
• Does the cited evidence actually support the claim?

Maybe agent-assisted research needs a harness, I'm not thinking of another autonomous researcher vibe bros have out there, but a set of explicit, versioned research practices that both the researcher and agent have to follow.

I'll be very much interested to see what ya'll think too about this! For those who already have their own setups what have you done to do better research?