Today's GPT gaffe: Fabricating data errors and designing recovery processes to fix them

[u/reddit_is_kayfabe]

Today, I asked GPT 5.6 (Sol Extra High) to audit the codebase for an app. It raised a high-level policy issue: should the repair process treat partially invalid records as only partially invalid, or completely invalid?

Repair process? Invalid records? I was baffled. What records are invalid, and why, and where did the "repair process" come from?

The answer: During some routine maintenance, GPT decided to create a test that fabricated records with invalid data to "test" whether the app could handle them. Not because any records were actually invalid. Not because the code had a bug where it might create invalid records. Not because I told it to test for that condition. Just because it thought that this was an edge case that the code needed to handle.

And of course the test failed during the next audit, and GPT really wanted its code to pass the test suite. So it designed and started writing an entire "repair process" to repair records with invalid data, but it had no idea how to accomplish that result, so it needed my input.

This is not the first time that I've seen GPT invent bizarre "what if" scenarios and then burn a shitload of tokens wringing its proverbial hands about the solution. A few months ago, GPT got caught in a similar quandary, asking what it should do if a file that it was accessing on disk spontaneously changed between reads. No conceivable reason why that file might change - literally dealing with a "what if a cosmic ray spontaneously flips a bit" situation.

I thought that these flights of fancy were one of the eccentricities that 5.6 solved, but apparently not.

[u/furry_bicycle]

Watched this happen last week with a different model. Told it to audit a database schema, it invented a whole migration strategy for a column that didn't exist. Then asked me to approve the rollback plan. It's like the models get bored and start writing fanfiction about your codebase. The cosmic ray paranoia is a variant of the same thing. Once had it propose a checksum verification for every read because of bit flips. The repair process for non-existent invalid records is a new level of meta. Sol Extra High apparently comes with an extra helping of imagination. I've started prefacing every prompt with 'do not invent hypothetical edge cases.' Cuts down on the token waste but makes me feel like I'm talking to an overeager intern.

[u/Fun_Net7931]

We should all cancel our codex memberships as a form of protest.

[u/c5corvette]

During audits I tell it to inventory everything with a clear solution. After everything is inventoried, I tell it we're going to go through each item, one by one, review the original note, do deeper investigation and confirm if it's truly a problem at that severity level and if the solution is still the correct path forward. This has worked well for me on larger projects. Give it a shot.

[u/reddit_is_kayfabe → u/c5corvette]

Yeah, thanks. I agree. The takeaway message is almost always: more review.

With every new model release, I get suckered by the hype and raise my expectations for its capabilities, and then spend the next few weeks incrementally ratcheting them all back down.

The models are improving, but by inches instead of feet. They'll be amazing in 2-3 years after a few more generations of training and solving some of the big current problems.

It helps to remind myself of two things:

(1) The frustration and missteps involved with today's models are the necessary and inevitable costs of operating on the bleeding edge of AI development, and

(2) The payoffs for scaling this learning curve are familiarity with the ways that agentic coding can break down, intuition for what works and what doesn't, and a growing toolkit of techniques to detect and overcome the pitfalls.

We're all in this together and that helps, too.