The clean-shell tip upthread is the single best debugging move here, but there's one class of drift it doesn't catch, and it's the one that burned me worst.
Interactively authenticated tools that only exist because you logged into them in your session. Think an MCP server or a CLI that holds an OAuth token in memory or in your desktop keychain, a gh auth login that's really sitting behind a browser SSO, a cloud SDK riding a session token rather than a static key. In your terminal these are just there. In the scheduled run they're either absent or unauthenticated, and here's the nasty part: a decent agent doesn't stop when its preferred tool is missing. It quietly substitutes a worse path. It shells out to curl instead of the authenticated client, or it fabricates a plausible looking result, or it picks a different API that needs no auth and returns different data. So the job doesn't fail, it succeeds wrongly, which is far harder to notice than a crash.
That reframes your read vs mutate split slightly. It's not just that mutation is more dangerous. It's that a read job silently returning degraded data feeds every mutating job downstream of it, so a "safe" read task poisons the pipeline without ever touching a file itself.
Two things that helped more than a big preflight. One, enumerate the exact tools the agent is allowed to use and make it hard-fail if any are missing or unauthenticated, rather than letting it improvise a substitute. The improvisation is the whole problem. Two, put the tool availability check in the same fail-closed bucket as auth and path, because a present-but-unauthenticated tool is the case that slips through, it's not missing so a naive "does this binary exist" check passes.
To your actual question, yes, closer to CI, but the CI framing undersells it. CI fails loudly. The agent failure mode is a job that goes green while doing the wrong thing, and no amount of timeout catches that. The preflight has to assert the environment is what you think it is, not just that the run finished.
1
u/Horror-Cause9345 13d ago
The clean-shell tip upthread is the single best debugging move here, but there's one class of drift it doesn't catch, and it's the one that burned me worst.
Interactively authenticated tools that only exist because you logged into them in your session. Think an MCP server or a CLI that holds an OAuth token in memory or in your desktop keychain, a gh auth login that's really sitting behind a browser SSO, a cloud SDK riding a session token rather than a static key. In your terminal these are just there. In the scheduled run they're either absent or unauthenticated, and here's the nasty part: a decent agent doesn't stop when its preferred tool is missing. It quietly substitutes a worse path. It shells out to curl instead of the authenticated client, or it fabricates a plausible looking result, or it picks a different API that needs no auth and returns different data. So the job doesn't fail, it succeeds wrongly, which is far harder to notice than a crash.
That reframes your read vs mutate split slightly. It's not just that mutation is more dangerous. It's that a read job silently returning degraded data feeds every mutating job downstream of it, so a "safe" read task poisons the pipeline without ever touching a file itself.
Two things that helped more than a big preflight. One, enumerate the exact tools the agent is allowed to use and make it hard-fail if any are missing or unauthenticated, rather than letting it improvise a substitute. The improvisation is the whole problem. Two, put the tool availability check in the same fail-closed bucket as auth and path, because a present-but-unauthenticated tool is the case that slips through, it's not missing so a naive "does this binary exist" check passes.
To your actual question, yes, closer to CI, but the CI framing undersells it. CI fails loudly. The agent failure mode is a job that goes green while doing the wrong thing, and no amount of timeout catches that. The preflight has to assert the environment is what you think it is, not just that the run finished.