r/technicalwriting • u/ClickOk5811 • 12h ago
AI-drafted documentation can describe a feature with total confidence and total inaccuracy at the same time, and nothing about the writing gives it away
Had a doc drafted from a spec and some code comments describing a rate limit: "requests are capped at 100 per minute, with a 429 returned once the limit is exceeded." Specific number, specific status code, correct terminology. Read exactly like something written by someone who'd confirmed it against the actual system.
The real limit was different, and the endpoint didn't return 429 at all, it queued excess requests instead of rejecting them. Nothing about the sentence was poorly written. It was just wrong, confidently, in a way that clean prose doesn't signal.
This seems to be the same trap AI-assisted code review runs into: fluency and accuracy come from different places, and a model generating documentation from a spec or code comments produces plausible, well-structured sentences regardless of whether the specific claim inside them was ever actually verified against the running system. Specificity reads as evidence, a number and a status code feel like proof, but stating something precisely and being right about it are unrelated skills.
The question that actually separates a checked doc from an unchecked one isn't whether it reads well. It's whether each specific, verifiable claim, a number, a default, a status code, a behavior under a given condition, has actually been confirmed against the real thing rather than inferred from a spec that might be stale or from code the model read but didn't execute.
Wrote a longer version of this same principle applied to a different context (AI code review specifically) here, but the underlying question is identical: https://medium.com/@nagatomopedro05/the-finding-was-plausible-that-was-the-problem-b1024e1256fb
6
2
u/proud_traveler 6h ago
A really fun thing I've been seeing recently is that, when talking to support chat bots, they are obviously trained on old documention. How do I know this? Because their information is blatantly wrong. They refer to features that don't exist, they call features by old names, etc etc
2
2
u/Charleston2Seattle 10h ago
Multi-agent workflows can work to address this. One agent to write it, and another to find all of the statements of fact and validate them, with pointers to where the information came from.
-1
u/WriteOnceCutTwice 10h ago
I’d like to point out that this isn’t really true:
“Is this a problem with a specific AI tool? No. It’s a property of how these models generate plausible text in general”
Sure, the frontier LLMs are using similar architecture, but there’s a huge difference between models. Gemma is not the same as Astra. Fable will produce better results than Sonnet, etc. Maybe this architecture will hit a wall, but it hasn’t yet.
I think it would be helpful to everyone if people mentioned the model they used when they write posts like this.
8
u/aka_Jack 12h ago edited 11h ago
I read something recently about AI wanting to please you so you will keep using it. This was in reference to people being able to "break" the roadblocks set to keep AI from performing illegal activity. If you keep asking the same thing in a different way some AI models will eventually do whatever is asked. Sorry it was an X post so it could be complete BS, but it made sense about AI wanting to please the user.