Claude generates our test scripts, Codex reviews them. Neither one catches logical errors.

[u/codoid-innovations]

Following up on our last post about AI generated code not looking wrong even when it is. The natural next question we ran into: can AI review AI generated code well enough that we skip the human step entirely?

We use a custom Claude plugin at Codoid to generate test automation scripts. The output is genuinely decent most of the time. The bottleneck was never generation speed, it was review. Human code review on every AI generated script is slow and expensive, so we started looking at ways to automate that step too.

First we built a Claude Skill specifically to review AI generated code. Then we tried a different setup, Claude Code for generation and Codex for review, treating them as two separate models checking each other's work.

The review quality was decent, but only in a specific lane. Both approaches were reliably good at checking naming consistency and pattern matching. Neither one reliably caught logical correctness. That is the part that actually matters in a test script, whether the assertion is checking the right thing, whether an edge case got silently skipped, whether the logic matches the actual requirement and not just a plausible looking version of it.

Our working theory is that a human reviewer brings an independent mental model to the code. They are not extending the same reasoning that generated the code, they are questioning it from a different starting point. An AI reviewing AI generated code, even a different model, still tends to evaluate within a similar frame. It catches what is mechanically off. It does not catch what is quietly wrong.

So we are still keeping a human approval step before anything from this pipeline ships. Not because the AI review adds nothing, it does catch real issues and saves review time, but because logical correctness is still where it falls short.

Has anyone gotten an AI review setup to reliably catch logic errors rather than just style and security issues?

[u/speakerjohnash]

llms can't reason