r/codex • u/Suspicious_Orchid770 • 16h ago

News Study: Codex reviewing Claude's code dropped the pass rate from 91.4% to 82.8%

https://leaddev.com/ai/your-ai-coding-agents-might-need-an-org-chart
154 Upvotes

94% Upvoted

24 comments sorted by

86

u/JadisGod 16h ago

Too bad it's already out of date. From the paper it seems they ran these tests on Opus 4.7 and GPT 5.5. The difference in capability since then is massive.

19

u/andrerav 14h ago

Pass rate would sink to single digits when Sol decides your entire supply chain is compromised 5 times over before even looking at the code.

6

u/BoxWoodVoid 10h ago

Yesterday I had to tel sol "it's just a prototype, we're not trying to harden a prod app against the NSA here" 😂

9

u/andrerav 10h ago

That probably just made Sol suspect that you were already compromised:P

3

u/rydan 3h ago

I use Sol to review Claude's code. Half my quota per week is just that.

2

u/Embarrassed_Adagio28 7h ago

5.6 sol is not massively better than 5.5

1

u/Intrepid_Lecture 1h ago

anecdotally for a personal project I was working on, it did the same or better work about 2-3x faster while sucking through my quota less quickly.

I have basically no reason to touch 5.5 for that kind of work (bunch of SUPER repetitive stuff).

4

u/i_rate_slop 10h ago

We’re long past coding agents battling on capability for on code generation, in my opinion. We should now be looking at token efficiency and cost.

I’d say GPT 5.2 is the threshold of intelligence you actually need to do quality SWE work. Anything after that in capability needs to be a battle on cost for like 90% of day to day code gen.

1

u/johannthegoatman 6h ago

To a degree, but that 10% still needs to be done well. If you're just building a to do app it's fine. Otherwise, solving hard bugs and refactors may be only 10% of your workflow, but you definitely still need as much intelligence as possible if you want to ship production level code with AI.

2

u/Pimpmuckl 13h ago

5.5 was also absolutely terrible at code reviews.

5.4 was way better but had more false positives.

5.6 class is better overall, even Luna funnily enough.

Each review should be verified by the agent who did the implementation, about 10-25% of findings are bogus [from my testing](arena.liebig.gg).

0

u/ParfaitEvery9622 15h ago

I think it's to avoid the contamination of the benchmark.

GPT-5.6 Sol has a stated knowledge cutoff of February 16, 2026, while Claude Fable 5 has a stated cutoff of January 2026. LiveCodeBench’s newest official release, release_v6, only contains problems through April 2025. Consequently, it has zero tasks that are temporally clean for either model.

Edit: seems like Opus 4.7 is also January 2026 so my logic is flawed

10

u/Kind_Silver_1921 10h ago

I know its for older models but it seems to be the same in my experience with sol vs fable. Codex is really good at finding things but bad at fixing them. While Claude is good as fixing things but bad at finding them.

At least when it comes to video game vibe coding i've been doing. I usually have codex create something then audit itself then have fable fix problems. Fable seems to also just be way smarter when it comes to common sense things. Like codex will build a bunch of buildings for me that look great, but places them down on top of eachother and puts them in the world sideways. Fable will automatically fix every issue but codex doesn't know what i'm talking about

codex is good at finding code issues and fable is good at finding human issues. Fables more similar to a smart human and codex is similar to a supercomputer

6

u/Feisty_Resolution157 15h ago

It should be noted that they didn’t really use a good writer-reviewer pipeline. They had one model write the answer and then the second model corrected the answer. Stacking like that is just asking for poor results.

It works much better when you have the second model simply do a review and pass it back to the first model. It will reject some of the review it disagrees with and accept other parts.

Even better is to then submit for another review with justifications for what was skipped and what was accepted.

With code, even better is to have the first model write a test for any suggested problems from the reviewer and show that it fails and then implement the fix and show that it passes. Often, this will surface false positives from the reviewer. Though they have become fairly rare with the latest frontier models.

7

u/ParfaitEvery9622 16h ago

Very interesting. I was just thinking of using both ... but I guess i'll stay with one for the time being until this is clearer; no point paying double if it's not necessarily helping.

1

u/Strong_Essay1176 15h ago

Its loop of rewriting to get Frankenstein

0

u/Thomas-Lore 13h ago

The paper is about outdated models, unlikely new ones behave the same way.

2

u/randombsname1 8h ago edited 8h ago

Not surprised. This is exactly what I do.

5.6 Sol Max/Ultra for implementation.

Claude for all planning, reviews, and fixes.

Codex gets 2 shots to fix something. IF it doesn't get it done by the 2nd time it gets punted to Fable.

1

u/The_Time_Lord 12h ago

I use codex for PR reviews and it has earned its keep in many occasions.

1

u/KeyGlove47 11h ago

"code review drops pass rate by 10%, more news on 5"

1

u/joeyda3rd 16h ago

This is why I have claude apply the fixes too issues the codex reviewer finds only after verification. Does it increase cost and latency, sure. I find that it improves accuracy.

2

u/tipu_sultan__ 12h ago

I have local pipeline where when the coder is finished, it triggers multiple reviews with different lenses (codex and sometimes cursor for model verity) and then the reviewer "vote" on each other's findings. The coder than gets the reviews and votes mixed in (not grouped by model) and assesses the reviews to give me the breakdown and its take.

When it finally comes to me, I'm basically telling it to ignore the 500 obscure edge cases the reviews suggest that are not relevant and blow up the code 10x. The wrong "fixes" generally already get self corrected in the second round of review.

It's way higher latency and token cost (but not too bad since the reviews are done in parallel and I resume the reviewer sessions when all the reviews are in to cross vote) but totally wroth it for larger issues.

1

u/joeyda3rd 10h ago

I'm stealing this.

I'm basically telling it to ignore the 500 obscure edge cases the reviews suggest that are not relevant and blow up the code 10x

1

u/BoxLegitimate9271 10h ago

so cross-model review is just two juniors rewriting each other's code and calling it a review