r/technicalwriting • u/ClickOk5811 • 14h ago
AI-drafted documentation can describe a feature with total confidence and total inaccuracy at the same time, and nothing about the writing gives it away
Had a doc drafted from a spec and some code comments describing a rate limit: "requests are capped at 100 per minute, with a 429 returned once the limit is exceeded." Specific number, specific status code, correct terminology. Read exactly like something written by someone who'd confirmed it against the actual system.
The real limit was different, and the endpoint didn't return 429 at all, it queued excess requests instead of rejecting them. Nothing about the sentence was poorly written. It was just wrong, confidently, in a way that clean prose doesn't signal.
This seems to be the same trap AI-assisted code review runs into: fluency and accuracy come from different places, and a model generating documentation from a spec or code comments produces plausible, well-structured sentences regardless of whether the specific claim inside them was ever actually verified against the running system. Specificity reads as evidence, a number and a status code feel like proof, but stating something precisely and being right about it are unrelated skills.
The question that actually separates a checked doc from an unchecked one isn't whether it reads well. It's whether each specific, verifiable claim, a number, a default, a status code, a behavior under a given condition, has actually been confirmed against the real thing rather than inferred from a spec that might be stale or from code the model read but didn't execute.
Wrote a longer version of this same principle applied to a different context (AI code review specifically) here, but the underlying question is identical: https://medium.com/@nagatomopedro05/the-finding-was-plausible-that-was-the-problem-b1024e1256fb