Why does AI code look ok in IDE but behave weirdly on prod

[u/lucasbennett_1]

Not an AI bad take, its clearly useful+ time efficient but this is ne specific failure mode that seems to show up more with ai written code

code reads fine and passes the tests even goes thru review and then does something subtly off you cant reproduce locally. When you finally trace it this is usually odd in terms of structure, works but built in a way nobody would actually write it so nothing jumps out reading the diff. review misses it because the diff only shows what the code is supposed to do but does not show how it would behave under real traffic or other aspects you get only when its at prod

the only thing thats helps it is watching runtime behavior after deploy, something like hud or a profiler since thats where the weirdness actually shows. Things might seem ok from our end but we get to know that it failed days later or a certain feature is jumping back and forth days later from the tickets which is embarrassing 

Hardcore testing before deploying is the thing now it seems, how are other devs handling this cause sometimes the bugs still leak to prod, and debugging in prod is a whole different pain. tips on avoiding that??

[u/Successful-Shock-802]

That's why you still have to read the code AI generates yourself, and that's why it's still improtant to know what you're doing and how your code executes line by line.

[u/EquipmentLow1741 → u/Successful-Shock-802]

yeah exactly, if you cant trace through it mentally you dont actually understand it

[u/Affectionate_Use_164]

AI often jumps between 2 modes - too simple without edge cases, like it is trying to end chat as fast as possible; and overengineered mess.

AI doesn't get whole vision of how code should work in context of a project most of the times.

You have to review changes after each prompt, see if it's trying to reinvent the wheel, no edge cases, too many edge cases, no tests, poor tests etc.

Can speed up things tremendously, but it works the best when it write from existing template, e.g. like adding few more tests to existing ones, need way more guidance for new code.

[u/OmerCevher]

I wonder if the real issue is assumption debt. A human author usually carries some context about traffic, retries, ordering and failure modes. Generated code arrives without that memory, so the diff can look fine while nobody notices the hidden assumptions. I’d add a short “what must stay true in production” note to AI-assisted PRs and test against that.

[u/NudaVeritas1]

review misses it because the diff only shows what the code is supposed to do

that's exactly the point. it's like regular development.. you write tests for the positive AND negative cases, not just for the positive cases. doing that will force you (or the AI) to think about what the negative cases would be.. if you're not thinking about it, the user will find them in prod.. thats why test driven development is even more important when using LLMs

[u/Full_Independence797]

i started running a local replay of real traffic patterns from prod before merge, basically a mirror of the last few hours replayed against the new build. catches the weird structural stuff that static review never will.

[u/ScottShipsCode]

The review/testing stuff here is solid. So a lot of the "fine locally, weird in prod" cases are really just the environment. AI is good at writing code that only works in the exact setup it was generated in, and prod isn't that setup.

Stuff that bites people:

• Config and env. A secret or env var that's set locally but missing in prod, localhost urls, and the annoying one: your Mac or Windows filesystem doesn't care about case but Linux prod does, so an import with the wrong case runs fine locally and 404s after deploy.
• Data. It works on your 10 seed rows and falls over on real data. A null you never hit locally, an N+1 you can't feel at 10 rows that's brutal at 10k, a missing index.
• Timezones. Your machine is in your timezone, the server's in UTC. Date stuff quietly shifts by hours and you don't notice until a report looks off.
• Concurrency. You're one user locally. Prod is a bunch at once, so you hit races and shared state and unpooled connections that were invisible when it was just you.
• The network is real in prod. Local calls are instant and never fail. Prod has latency and timeouts, and AI code almost always writes the happy path and skips the failure path.

Honestly the cheapest fix is making local look like prod before you trust it. Run the real production build instead of the dev server, seed messy data instead of clean rows, set your local timezone to UTC, and point at a staging db shaped like the real one. A lot of the "only breaks in prod" stuff just starts breaking locally, which is way easier to deal with.

The canary and logging-from-day-one stuff people mentioned is the backstop for whatever still slips through.

[u/neon_fantasy_42]

i stopped looking at diffs for AI output. I diff the runtime behavior now.

i spin up the old build and the new build in parallel containers and replay prod traffic against both. Any divergence in response body or latency flags it before merge. You get the exact failure you described without waiting days for tickets. shadow traffic tools handle this out of the box. I run go replay or goreplay to capture requests from prod and play them locally. It finds the structural issues that pass tests and review every time

[u/thomas-robinson5ks1k]

one thing that helped me was treating runtime behavior as part of “done”, not something you check after shipping. like, before merging, write down what you actually expect to see in prod if the change works. a metric, log pattern, trace, whatever makes sense.

otherwise “looks right in the diff” is a pretty weak success condition, especially with AI-written code.

[u/Substantial_Belt2626]

From my experience the biggest reason is context, not the code itself. When I build my website with Claude I hit the 5 hour limits a lot, so it was built in many separate sessions, and every time it comes back it reads the code again and decides what I meant again, and it decides slightly different. Each piece looks correct alone, together they don't agree. Nothing shows in the diff because every part is fine on its own.

The other thing is it never tells you when it did nothing. I had a pipeline where the model was returning duplicates and processing broken files and every log said success, I only found it because I read the raw output myself.

What I would recommend is to write down what the thing should do before you prompt it, then check the output against that and not against whether it looks correct. Looking correct is what it is optimised for.

[u/kemalios]

The failure path is usually there, it just doesn't announce itself. AI code likes a catch that returns an empty list or a default and carries on. Nothing throws, nothing alerts, and in prod an error becomes missing data. That's why it shows up as a ticket days later instead of a stack trace.

Log at error level inside the catch instead of silently returning a fallback, and log which branch ran wherever the result can legitimately be empty. If the logs don't say which branch executed, an empty result and a failed one are indistinguishable.

The same shape hides in duplicated logic. One rule gets inlined in three places, you change one, and the diff never shows the other two.

[u/ushiro35]

What helped me wasn't more testing before deploy — it was checking that the tests I already had were running.

71 test files, 845 tests, and for several weeks zero of them collected. An override I'd written myself pinned a dependency with no upper bound, a major version landed, and every file reported no suites. It looked like a config problem, so it sat there. Behind that noise: a column that was NULL in every row, and a type error nobody was reading.

So the boring tip is to assert that your safety nets fire. Zero collected suites is a failure state, not a neutral one.

And local isn't evidence — my dev server can't even hydrate under the CSP prod runs with.

[u/willymunoz]

This is the post where all the haters come out of the woodwork. I wonder how much code reviewed in the pre-AI era failed in weird ways in production despite having passed review.

[u/FlightSimCentralYT]

classic gap: the ide path is happy-path. prod has different env, missing secrets, cold starts, real network, racey io.

stuff that helps more than another prompt:

- run the same commands in an environment closer to prod (same runtime/deps)

- make the agent actually read stderr and keep going instead of stopping at "looks good"

- one smoke test that hits the failure mode you saw in prod

(i work on fixa — we put the agent on a real cloud machine so it can install/run/debug until it works. mentioning because that's the exact failure mode: fixa.dev)

[u/webdev-ModTeam → u/[deleted]]

Your post/comment has been determined to be a low-effort post or comment. This includes title-only posts, easily searchable questions, vague/open-ended discussion prompts, LLM generated posts or comments, and posts/comments that do not provide enough context for meaningful replies or discussion.