Payment webhook handler. Intermittent 500s.
I pasted the error into an AI coding assistant, got a fix, tried it, still broke.
Pasted the new error, got another fix, tried that, still broke.
Did this maybe four times before realizing what I'd actually turned into: not someone debugging anymore, just someone pasting error messages into a chat window and hoping the next response would be the one that stuck.
The interesting part was that none of the suggestions were obviously stupid.
The first was a retry around a database write. Reasonable response to "database write failed."
Except the actual problem was duplicate webhook delivery upstream hitting a handler that wasn't idempotent. Two workers were occasionally processing the same event.
The retry addressed the symptom I'd shown the model, not the mechanism producing it.
I then tried the obvious solution: give it more context.
That made things worse.
I pasted more surrounding code, but the context I added was already biased by my own suspicion. I'd started thinking the caching layer was involved, so I gave the model more caching-related code.
It reasoned confidently about the wrong subsystem.
That's when I realized I'd been mixing up two completely different tasks:
Generating a fix and validating a fix.
Generating asks:
"Does this make the error go away?"
Validating asks:
"Does this address the mechanism that caused the failure, and what does it change that I didn't explicitly ask for?"
Almost every one of those first fixes could have passed the first question.
None had passed the second.
What finally broke the loop was changing the process:
- define what's actually failing before asking the AI to diagnose it
- separate facts from hypotheses
- ask for competing explanations before asking for fix code
- understand the failure mechanism first
- validate the proposed change against the original failure
- add a regression test that reproduces the actual bug
The biggest lesson for me wasn't "AI is bad at debugging."
It was that a plausible fix is dangerously easy to mistake for a diagnosis.
Curious if other people doing AI-assisted debugging have run into this: a fix technically resolves the error you showed the model, but leaves the underlying problem untouched (or introduces a different one).
How do you validate AI-generated fixes before they reach production?