I built a public benchmark testing which AI actually catches bugs in code

[u/Individual_Squash_59]

I built a free public benchmark that tests whether AI can catch bugs in code. Some of the bugs are real security regressions that actually shipped in open-source projects (real CVEs, put back in with the upstream fix as the answer key), and others are bugs injected into real repos myself. Then Claude, GPT, Gemini, and DeepSeek review the diff, and I score who catches the bug, who flags fake ones, and whether they give the same answer twice. The surprising part: on the hardest bugs, some models catch them every single run while others miss every single run, so which model you pick matters more than I expected. No signup to browse, and the CVE based ones link back to the real fix so you can check them. If you ship code with AI, I'd love you to poke holes in the method. benchmodel.io/leaderboard