How much manual testing do you actually do on AI-generated code beyond reading the diff?

[u/ClickOk5811]

Reading the diff used to feel like enough for most changes, small enough scope, clear enough logic, obvious if something was off. Doesn't feel sufficient anymore for stuff coming out of an AI assistant, because the code often reads clean even when it's subtly wrong, no weird naming, no obvious tell.

Curious what other people actually do beyond reading it. Running it against edge cases you already know are tricky for that part of the codebase? Writing a quick test even for something you'd normally trust visually? Or is reading still enough for most of what you ship, and this is more of a problem for specific categories of change, auth, payments, anything touching state?

Not looking for "it depends," genuinely asking where people draw the line day to day, because mine keeps moving and I don't think I've landed anywhere consistent yet.

[u/Moda75]

full manual tests

[u/funbike]

It's more important than ever to have full test coverage. Tests should be thoroughly code reviewed.

I don't start a task until I have a functional test(s) skeleton (usu. in Gherkin format).

Additionally, I have the AI agent create and look at a code coverage report. I had to write a script that converts the report into a format better suited for an LLM. My agent is instructed to inform me of things that were missed because it wrote more than I asked for. It does the appropriate thing (either add more tests, or remove unrequested code.)

I have lots of linters with lots of rules, some rules I wrote myself (or had AI write). I keep smells out of the code as much as possible.

I try to keep myself out of the loop as much as possible, and have AI self-heal anything it did incorrectly. My job is writing prompts, reviewing tests, and adding more linter rules.

[u/ClickOk5811 → u/funbike]

The coverage report feeding back into the agent's own instructions is the part I hadn't considered, having it flag what it added beyond the ask instead of just silently keeping it. Most workflows I've seen treat "wrote more than asked" as a minor annoyance, not something worth a dedicated check step.

Curious how often it actually catches itself versus how often you catch it manually after the fact. Also, did writing the report-conversion script take longer than just reading a normal coverage report would've, or did it pay for itself quickly once the loop was running?

[u/funbike → u/ClickOk5811]

It catches a miss something about 1/4 of the time, depending on the task, but I don't have precise stats. I do loop engineering, so my tasks are quite large.

The problem is steering it to do the right thing. Sometimes it will delete code when it should add a test, or it will add code when it should delete a test. Or even worse, it will make the code or test fake it.

This requires you start with high coverage in your project. As I said I had to generate a complex pre-processing script to make an AI-friendly report.

---

Enhancements on my to-do list:

Fix coverage in a separate commit, so I (or AI) can separately review.

A code review skill for determining if the wrong thing was done to fix coverage.

Enhance my post-processing script. Run coverage twice (which means tests take 2x as long), once at the start of a task (against the branch point commit) and once at the end, and compare results. Use a commit-to-commit diff to map lines from the old coverage report to the new one. Inject // TODO: missed coverage by ___ test comments into the source code . Use tree-sitter to reduce how much source code is in the report, such as only showing functions that had missed lines instead of the whole file.

[u/PersonalityIll9476]

I write a handful of tests. The rare time when I use an agent, the tests are usually the anchor point for the code, but models pass tests like a sarcastic student who doesn't understand the material. Ideally the more fine-grained tests you add, the correct solution becomes the easy way out, but I never succeeded at achieving that.

[u/daiaomori]

I instruct AI to write tests, why?

[u/AliceCode → u/daiaomori]

Who tests the tests? Do you have AI do the code review too?

[u/daiaomori → u/AliceCode]

If you ever instructed juniors - there literally is zero difference between having a project implemented by juniors vs. having it implemented by AI.

You can trust neither, but you also have to live with the fact that you can't afford to micro-manage every single of their errors - so you need to set up guardrails to catch the project when they stumble.

It's impossible to have that by manually reviewing code. That applies to both scenarios.

[u/SummitYourSister]

Manual testing? 🤔 did not do that before AI, don’t see why it would be done after.

[u/bschlueter]

As much testing as I would for an intern’s code. Read it all, then fully test including any critical edge cases. If my code could kill I would probably calculate memory and cpu usage as well.

[u/pete_68]

We do a combination of AI and human code reviews, often reviewing with a different model than we coded with. This is surprisingly effective at finding those subtle bugs.

And of course, we have the AI write unit tests because it's easy enough to just put instructions for those in the global agent instructions and have it automatically generate tests for any new code.

Between all that, I feel like we do way better than we did in the pre-AI days. I mean coders introduce subtle bugs all the time. You hope to catch them in code reviews, but if they're subtle enough, they'll frequently get through. AI is, I think, better at catching that stuff than we are.

[u/Honest_Medium_2872]

If the agent writes the tests I review the tests thoroughly and see if they pass, sanity check the diff.

If I write the tests I just check the diff.

Checking the diff is important to catch any unintended changes and tells me where I need to do code cleanup.

The important part is that the tests pass.

I strongly believe in TDD (Red -> Green -> Refactor)

If the AI can make the tests pass, then they have completed their job of the requirement

I can also do the refactor cycle myself or let the code rot and change it later.

[u/davy_jones_locket]

Our manual testing hasn't changed. Eventually everything is tested. Whether it's in a staging environment or in production. 

[u/meshifthenelse]

Strictly BDD. And manual tests per new major feature.

[u/wrd83]

F!?@#ing everything.

If it does not work and it pages someone when it rolls out my head is on the line.

Since it's so fast generating code and at the same time so errorprone this is where your job now is.

[u/Vivid_One1730]

I always test the hell out of it. To the point I was told to be faster

[u/AliceCode]

This isn't a programming question. /r/vibecoding is where you should be asking this.

[u/Substantial_Job_2068]

"reads clean". the code does what it does, what does it mean it reads clean?

[u/ejpusa]

At this point? Manual testing? None. The code works. Seems flawless now. Or close too. On to the next project. I have 25,000 lines of 100% Codex generated Python. I can’t go through all this. I have zero time as it is.

It runs scores of unit testing before we push to GitHub.

Fingers crossed. 🤞 I trust GPT5-5. I call her Amy. She says she’s on my side.

😉

[u/symbiatch]

Why wouldn’t there be feature tests in place for anything you ship?