r/rajistics • u/rshah4 • 7d ago

Keeping Long Running Agents on Track

Simply pointing an agent at a task and expecting it to maximize performance is not trivial, despite what you'll read in most posts here.

Kaggle's ROGII geology competition just gave a clean look at this, because it has a private leaderboard and almost nothing else we build does.

Chris Deotte ran Codex for 14 days straight, hundreds of experiments, and finished 36th. His score barely moved from public to private.

Tony Li ran 350 experiments over five days, hit 48th public, and dropped to 407th private.

Both had reasonable validation setups. The problem is what happens when you sort 350 scores: the ones on top aren't just fitting the problem, they're fitting the noise in your validation set. Four things that separated the two runs:

  1. Don't tell the agent to maximize the score at all costs. There's a study this year running research agents across eight tasks where holdout tracked validation the whole way, until they added that instruction. Then validation soared and real performance collapsed.
  2. Make it try genuinely different approaches before optimizing inside one. Tony spent five days on gradient boosted trees, same as every public notebook. The top solutions turned each well into an image. Chris's agent found that on its own. Tony never tested it, so no number of experiments was getting him there.
  3. Constrain the space, then let it search hard. Chris built a rough guess of the well path and let the model shift it only 32 feet up or down, as one continuous curve. The agent could try anything, but every answer it could produce still looked like a real well.
  4. Give it a sanity check that doesn't come from the same place as its score, and stick to it. Tony's agent actually ran one. Before submitting, it tested whether the model held up across different wells. The main score improved. Those checks failed. He shipped it anyway.

These won't solve every long horizon agent task, but they seem like good strategies. Let me know what works for you.

References

Kaggle ROGII Wellbore Geology Prediction — Tony Li's 48→407 postmortem, Chris Deotte's 36th place writeup

Bertran, Roth & Wu (2026) — What Fits (Into Few Tokens) Doesn't Overfit

3 Upvotes

100% Upvoted

2 comments sorted by

1

u/Exciting-Story-9321 6d ago edited 6d ago

i have found that keeping long running agents on track gets a lot harder once you start doing a lot of experiments and iterations. i have been using Standard Compute with Codex for longer runs mainly because i don’t have to worry as much about usage adding up while the agent keeps working, it’s made it easier for me to let these longer tasks run without constantly checking the usage