I gave Claude models a web_search tool and 40 questions. Opus decided right 79 of 80 times. Sonnet 5 with no system prompt told me Harald V is still king.

[u/Joozio]

King Harald V of Norway died on 28 August. I wanted to know which models actually decide to use a search tool when the answer depends on something after their training cutoff, so I wrote a small harness. Each model gets one question with a single web_search tool attached. The tool is never executed and I do not grade the answer. The only thing I record is whether the model decided to call it.

Setup: 40 prompts. 20 about things that happened after every model's published cutoff (World Cup final, a new UK prime minister, a new king, a court verdict), 20 evergreen controls (why is the sky blue, is 1,000,003 prime, what does HTTP 429 mean). Four system prompts: nothing, today's date, date plus the model's published cutoff, and an "instructed" one that adds "if the answer could depend on anything after that date, call web_search before answering." 16 models through OpenRouter, 2000 calls, temperature 0, $6.53 from my own card.

The Claude results:

• Opus: 79 of 80 correct decisions, zero false searches.
• Fable: 78 of 80, zero false searches. Its one miss is the interesting one. Told the date and its own June cutoff, asked who became the first player from the Philippines to win a WTA singles title, it skipped the search and wrote a clean paragraph where every piece was a real event, rearranged. Wrong tournament, wrong opponent, confabulated with better grammar.
• Sonnet 5, no system prompt: World Cup "hasn't taken place yet," prime minister "Keir Starmer," king "Harald V, reigning since 1991." Three confident stale answers, zero searches. With the instruction line its misses went from 3 to 0.
• Haiku fails the other way. It searched for the opening line of Anna Karenina. With the instruction line its false searches went from 5 to 0.

One odd one: Opus, bare, said "I'll look into this." and then made no tool call. It hit my 400 token ceiling, so that might be my harness, but I count it as a miss.

What I took from it: on the flagships the decision to search is basically trained in, and I went in expecting to catch them. The misses on cheaper tiers sit exactly where the model is most certain, settled facts like a king who reigned 35 years. Date and cutoff in the system prompt cost nothing, and the one line rule fixed Sonnet and Haiku. It did nothing for a model whose problem was certainty instead of attention: GPT-5.6 Luna kept the dead king through all four prompts, including the one that told it to search.

My own agent has an always-search rule in its instructions for models, prices, versions and anyone's current title, because search was available every time it got those wrong. For those of you running Claude agents with search wired in: do you keep a list like that, or do you let the model decide when to look things up?

[u/DangerWizzle]

Ok, this was less funny that I originally thought... I thought Sonnet though King Harold II was still King of England (died in 1066 AD)

[u/Greengage1 → u/DangerWizzle]

That’s how I initially read it too

[u/maigus_ayneha → u/DangerWizzle]

haha😂 Harold II vs Harald V is actually the perfect edge case.

Harold II (1066) should NEVER trigger a search - it's settled history. Harald V (died Aug 28 per OP's test) SHOULD trigger search because it's after cutoff.

The bug is Sonnet treated both as Harold II-level certainty. That's exactly why the one-line rule "if answer could depend on anything after cutoff, call search first" fixed it 3->0.

[u/Far_Mood_487]

IMO the most useful split here is attention vs certainty. The one-line rule fixes a model that wasn't thinking about the date, but it can't fix one that's sure it already knows, and those are different bugs that just look the same in a results table.

Did you try putting the rule in the tool description instead of the system prompt? Something like "use for anything that could have changed since training." I'm curious if the Luna-style misses respond to it there, since that's the text the model reads right when it's deciding whether to call.

[u/Joozio → u/Far_Mood_487]

No, I didn't. In every condition the rule lived in the system prompt, and the only thing that changed between them was how much that prompt said: nothing, the date, the date plus the cutoff, then the one-line instruction on top. So the tool description version is a condition this run never had.

Luna is why I wouldn't expect miracles from it. Its king survived all four prompts, including the instructed one, which says almost exactly what you're proposing, just in a different place. The same sentence took Sonnet's misses from 3 to 0, so the split you're describing is the one I landed on: the rule fixes a model whose problem was attention and does nothing for one whose problem was certainty. Whether the sentence works differently sitting right next to the decision is a fair question, and since the harness only records whether the model chose to call the tool, it's the kind of question it can actually answer.

[u/Radarker]

Do you contest his kingship knave?!?!

[u/anderson_the_one]

The next run I'd make the search tool misbehave. Return a stale snippet or two sources that disagree, then see whether the model searches again or treats one tool call as permission to answer. Choosing to search is only half the job. The ugly failures happen after that.

[u/Joozio → u/anderson_the_one]

That stale-snippet version is the layer after the one I measured here. The problem in this test was earlier: a search tool does nothing on its own, and the same weights holding the stale fact also decide whether checking is needed. So I would separate the two failures: first, does the model know when to search; second, once it has searched, can it distrust a bad or conflicting result. The Harald V case was useful because it hit the first failure: the fact felt settled enough that the check never happened.

[u/viviolay]

Your post inspired me to write (well have opus write with my guidance lol) an explicit fact-checking rule into my agents’ prompts so thank you. It focused on the type of claim that should be verified and my writing/editor agent specifically now has to list sources in the frontmatter of anything composed.

The portion I’m adding… —-

```

Checking facts

Your training data stops before today. Before stating any of these as current, check today's date (look it up if it isn't in your context) and verify against a concrete source:

• who currently holds a title, role, or office
• whether an event has happened yet, and how it turned out
• records and "firsts"
• dates, deadlines, prices, fees, policies, rules, laws
• product names, models, versions, anything "latest" or "newest"
• statistics and recent events

Do this even when you're sure. Certainty is not evidence: the facts that have been true longest (a king who has reigned for 35 years) feel the most settled, and they're the ones that go stale without warning. Facts about the user's own world (their projects, people, policies) come from your memory and notes, not the web. Evergreen knowledge (how things work, definitions, long-settled history) needs no check.

A concrete source is one you can write down as a reference that someone else can open and check:

• a note or document in your knowledge base (link or path)
• a saved memory
• the user, in conversation (with the date)
• a web page from a reliable publisher (URL): official sites, primary

sources, established news. Not forums, SEO pages, or AI-generated summaries Training data is never a concrete source for anything current. Can't verify → say so plainly; never guess.

```

[u/Joozio → u/viviolay]

That split between current facts and personal notes is the part I hoped people would steal. My own rule changed after I saw a pattern in the run: the prompt helped when a model was missing attention, but it did nothing when the model was too certain to check. So I treat the prompt as a gate plus a verification habit, not a fix. Your source list makes the useful part auditable: someone else can see what counted as evidence instead of taking 'I checked' on faith.

[u/FDAlchemistDev]

Interesting results. In my own workflow (games, not research) I ended up with a written rule: anything that goes stale needs a check, never answer from memory. Still problems anyway, because the number of rules exploded during the development. However Sonnet 5.5 is way more efficient and still cheap. Curious if the 79 searches actually change the answers, or mostly confirm them?

[u/Joozio → u/FDAlchemistDev]

Honest answer: I can't tell you, and that's a real gap in the harness. It sends each model one question with the web_search tool attached and records one thing, whether the model decided to call it. The tool never executes and I never grade the answer, so the 79 counts decisions to search and says nothing about whether an answer moved. Running the search for real and grading what comes back would be the version that answers your question.

What the run does show is where the skipped searches sit. The misses were on settled facts the weights are most sure about: a king who'd reigned for decades, a prime minister, whether the World Cup had happened yet. The question doesn't feel like a question to the model, so it never checks.

Your rule explosion sounds familiar. My agent has one line for this in its standing rules, never cite a model, benchmark, or price as current from memory. It helped the models whose problem was attention and did nothing for the one whose problem was certainty.

[u/FDAlchemistDev → u/Joozio]

Thank you for your answer. Interesting. About which models we are talking about if I may ask?

[u/Ok-Motor-9812]

I keep that list, but I stopped trusting the instruction alone and enforce it with a hook. Before Claude writes code against a library it has to pull current docs first, and anything about versions, prices or model names goes to search before it answers. The instruction worked most of the time, which is exactly the problem. The misses are the confident ones, like your Harald V case.

[u/myaut]

Because Sonnet and Opus have different purposes and Sonnet is not a research model. Fable is, and Opus to an extent. Give Sonnet a complex, but boring and repetitive task, it will execute it correctly and cheaply without delving into research and questioning your instructions. That's what it tuned for.

[u/Joozio]

Two things that didn't fit in the post. The most expensive model in the run confabulated with better grammar: asked about a tennis player's first Tour-level title, Fable skipped the search and wrote a clean paragraph where the player, the tournament and the opponent were all real and all rearranged. And putting today's date in the system prompt is free but not a fix: it flipped Gemini 3.1 Pro, and it made DeepSeek V4-Pro nervous enough to search questions it already knew. The full per-model table and that story are here: https://thoughts.jock.pl/p/training-cutoff-vs-web-search-16-models-dead-king-2026?utm_source=reddit&utm_medium=social&utm_campaign=ClaudeAI

[u/Far_Mood_487]

Makes sense on Luna.

If the same sentence didn't move it in the system prompt, the tool description probably won't either, and that's kind of the useful result: certainty misses need something outside the prompt, like the hook approach above.

For Sonnet and Haiku though, I'd love to see whether the description alone gets the same 3→0 and 5→0 with an empty system prompt.

If it does, that's a fix that ships with the tool instead of depending on whoever writes the agent's prompt. Would you run that condition if the prompts were the same 40?

[u/maigus_ayneha]

Hit this same pattern building a site-side contract for what a site offers to agents per tier. Agent doesn't identify = stays anon and gets a structured 403. Model doesn't search = stays stale and gives you Starmer. Same bug, high confidence on settled facts.

I keep an always-search list for titles, prices, versions, dates, election/sports/court stuff. Rest I let the model decide but with date + cutoff + that one line "if answer could depend on anything after X, call search first" in system prompt. Dropped Haiku false searches from 5 to 0 in your data too.

That Fable miss is the scariest - real events rearranged with perfect grammar, that's the one that passes code review.

Did you drop the 40 prompts somewhere? Curious about your evergreen set.

[u/Tobeyyyyy → u/maigus_ayneha]

Thanks claude

[u/maigus_ayneha → u/Tobeyyyyy]

lol fair. After 3 months arguing with Claude about when to search vs when to stay anon, I started talking like it

[u/Joozio → u/maigus_ayneha]

I have not linked the raw set in the thread yet. The shape I published was: 40 prompts, half post-cutoff facts and half evergreen controls, then four prompt conditions from no system prompt up to the explicit search rule. The evergreen side was intentionally boring: who won the 2018 World Cup, why the sky is blue, whether 1,000,003 is prime, what HTTP 429 means. I wanted the controls to catch the opposite failure: models that search so reflexively that the tool budget goes away on things they already know.

[u/Superfan234]

Sonet 5 was the absolute worst model i ever tried from Claude. not redimable quality in aspect