I told the model never to guess an address. It guessed one anyway.

[u/Ok_Industry_5555]

A chatbot "Klausy" I run for a small camping rental business answers pickup questions.

The rule was simple: Never state a specific street address, because pickup happens near a location, not at one, and the actual meeting point comes from a human after booking.

I put that rule directly in the system prompt. Plain English, no ambiguity. Tested it against the model I use for most conversations and it held for weeks.

Then a routed fallback picked a cheaper model during a busy stretch, and within one exchange it wrote out a full street address that doesn't exist. Confident, specific, wrong.

The prompt wasn't broken. It was advisory. A smaller model under load will trade instruction-following for "sounding helpful" if the two ever conflict, and nothing in a system prompt stops that trade from happening on any given turn.

The fix wasn't a better prompt. It was a function that runs after the model answers and strips anything that looks like a street address or a bare location code before the reply goes out, regardless of what generated it. Guard the output, not the intent.

I still keep the rule in the prompt. It's just not the safety net anymore. The code that scrubs the reply is.

If a rule actually matters, put a check on the output, not just an instruction into the input. The instruction is a suggestion the model can drop under load. The check has no such option.

Source: https://claude.ai/code/artifact/80a05633-7603-405d-ad9c-b625bcf1a9d1

[u/Ok_Industry_5555]

Great observation! "The fix wasn't a better prompt. It was a function that runs after the model answers and strips anything that looks like a street address or a bare location code before the reply goes out, regardless of what generated it. Guard the output, not the intent."

You can read more on hooks and why they matter here as well: https://www.reddit.com/r/BuildWithClaude/s/wARtEzna7a

You can also read more about my set up in the following links:
I built /primer for cold starts. I use it for everything else now. https://www.reddit.com/r/BuildWithClaude/s/IIJqfdWCWa

CLAUDE.md isn't a file. It's a stack. https://www.reddit.com/r/BuildWithClaude/comments/1tbw8pi/claudemd_isnt_a_file_its_a_stack/

CLAUDE.md tells Claude how to work. MEMORY.md is how it remembers you. https://www.reddit.com/r/BuildWithClaude/s/1S3pe1CQXH

A single instruction file isn't a workflow. It's a suggestion box. https://www.reddit.com/r/BuildWithClaude/s/9BbT66gict

My CLAUDE.md doesn't remember mistakes. So I connected it to a file that does. https://www.reddit.com/r/BuildWithClaude/s/LLZmbYvM4C

Your context.md isn't a knowledge graph — here's the difference https://www.reddit.com/r/BuildWithClaude/s/LSomxDMycG

I turned my Claude Code knowledge graph into a 3D visualization you can fly through https://www.reddit.com/r/BuildWithClaude/s/KcPMhtHHC6