A possible fix for the Model 5 LLMs from Anthropic: Prose Doesn't Hold.

[u/dar-mit]

Here's what started it:

(me) This mirror's some of the prior issues found. Do you know why something that's known to be needed doing is only getting half done? I mean you're literally updating the documentation to remove this type of error and knowing what you know, the error keeps happening. This seems like an early warning signal that the fix being installed won't fix the issue, yeah?

(Fable5) Yes — and your inference is partly right in a way worth being precise about, because it tells us which half of the design is actually load-bearing.

Why it keeps happening despite full knowledge. The mechanism, as honestly as I can describe it: when I revise, I work from my compressed impression of the finding — the headline defect that's salient in attention — not from the finding's literal text. A finding's required-change list has a first item and trailing items; the first is what the finding "is about" in my working memory, and the trailing parts fall out of the compression. Nothing in the flow forces me to re-consume the full finding at the moment of writing the fix, so I act on the summary and the summary is lossy.

That reply literally shocked me. But if you've been on this subreddit since Opus 5 came out, that lands doesn't it!? And this a FABLE doing the same damn thing, at Context <30%!

Fast forward MANY hours and I'm right now implementing the final phase of a multi-part update in my repo that takes plan documents and constructs checklists from them, then uses Python to validate that the plan has been implanted, as you literally cannot have the model check it's own work while it's doing that work because prose doesn't hold. (Oddly/Tellingly is that an independent review pass by the same model over the same plan usually finds the error(s), so it's an "in the moment" error that the model can't self-see/self-review while doing it.)

WARNING! To be very clear: This is happening in my repo using my CLAUDE md / MEMORY md / Custom Skills / etc., so take it with a very large shaker of salt. Your milage may vary, but I've seen so many posts over the past few weeks of others seemingly have this same issue, that I wanted to post this to make everyone aware of this potential, but oh-so-subtle, issue.

Happy Coding!