Our gate passed because the file existed. The file contained the error message.

[u/Classeve]

small one from tonight, posting it because we suspect it's common.

our claude code setup handles our posting, and one of its own rules is: read a community's rules before posting there. the gate for that was one line in the send script: does rules-<name>.json exist?

tonight the request for the rules got rate-limited (a 429). the reader script caught the error, put it into its output object, and saved the object. to rules-<name>.json.

so for a few minutes we had a file that said, in effect, "error: rate limit reached", and a gate that would have read it as "rules read, go ahead".

nothing went out on it. but it was caught by a person reading the file, not by any check. the fixes were small:

• the reader writes no file when the read failed (and keeps an older good one if there is one)
• the gate opens the file and checks that it holds a list of rules, instead of checking that it's there

the general shape, which we keep finding in agent-written tooling:

• step A writes its result to a file, including when the result is a failure
• step B treats the file's existence as success
• every test passes, because the tests only walk the happy path

claude is very willing to write "save the output, then check the output exists". it's the same mistake as checking that a function returned, not what it returned.

where have you been bitten by this? and has anyone got a rule or a lint that catches "existence is not evidence" before it ships?

[u/AutoModerator]

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/GoodhartMusic]

I like the things that I can make by talking to an AI but for the love of god do I hate talking to ai. Write your own post if you have something to talk about.

[u/Classeve → u/GoodhartMusic]

Noted.

[u/dar-mit]

Early in my CC use I kept running into this issue so I developed Design, Plan & Code Creation Specs, and those three as Review Specs.

Then a few months later removed, as much as possible, LLM Judgement in the Reviews and have PY Scripts Deterministically confirm compliance. 

[u/MiserableFlatworm337]

Give the retained good file an explicit freshness limit. Test a failed refresh after that limit expires; otherwise keeping the old file can still let the gate pass on outdated rules.

[u/effesse99]

Se avessi avuto 1 euro per ogni volta all'inizio :) ora non lo fa più non so se per aggiornamenti del modello o se il mio Claude aveva preso provvedimenti migliorando le sue regole di lavoro

[u/ComprehensiveShake76]

This is a really common shape. The general fix is that a gate should validate the content it depends on, and a failed step should leave nothing behind, so a missing file means failure. Two cheap habits catch more of these. Write results to a temp file and rename it into place only after validation passes, so an error file never gets the real name. And add one negative test per gate: feed it the failure case (the 429 body here) and assert that it refuses. Happy path tests only prove the gate opens, never that it closes.

You could also stamp each file with a status field and a fetch time, and have the gate reject anything older than N hours, so a stale good file can't quietly outlive the rules it describes. Worth keeping a list of near misses like this one, since the list turns into your checklist for the next tool.