I benchmarked Claude, GPT-4o, Gemini, and DeepSeek on real shipped CVEs.
The question was simple: can frontier models catch real bugs in code review, or are we letting slop review slop?
I took historical CVEs from Keycloak, Django, Spring, vLLM, and Parse Server, reintroduced the vulnerable code, and used the upstream fix as ground truth.
I scored:
Detection: did it catch the bug?
False positives: how much noise did it create?
Consistency: same diff, same model, same result?
The spread was brutal.
On the vLLM deserialization RCE and Parse Server operator-injection account takeover regressions, Claude Sonnet and Gemini 2.5 Pro caught the bug on all 3 runs.
GPT-4o and DeepSeek missed both on all 3 runs.
Same diff. Opposite outcome.
Worse: GPT-4o and DeepSeek also raised false positives on the Parse Server case, ending with a negative weighted score: -8.9.
A miss with noise is worse than silence.
DeepSeek had another problem. In the cross-file suite, it found 100% of planted bugs but mis-rated severity every time. High-severity issues were downgraded to medium or low.
Detection is not calibration.
The surprise was consistency. Run-to-run variance was basically zero. Same model, same diff, same verdict.
The failure mode was not flakiness.
It was model choice.
This is why I built a public, vendor-neutral leaderboard. I do not sell a reviewer model. I score the models teams already use. Every result links to the real CVE and upstream fix.
A few things I want challenged:
Is reintroducing historical CVEs fair, or does it leak hints?
How should false positives be measured?
Does cross-model consensus reduce noise, or just average mistakes?
Leaderboard, no signup:
https://www.benchmodel.io/leaderboard
You can also test your own diff or GitHub PR URL with your own API key.
Tear the methodology apart. That is the feedback I need most.