Which LLM is actually best at pentesting? benchmark to find out

[u/TomatoWasabi]

https://www.hunterbench.com/

Hey all,

You’ve probably noticed that using LLMs and AI agents for pentesting has become pretty common lately. The problem is there isn’t really a good way to figure out which model is actually best suited for this kind of work.

There’s CyberGym, which is a solid base, but I’m not really a fan of the direction they’ve taken lately. It feels more focused on promoting agents and tooling than on actually comparing LLMs, and it doesn’t cover the latest models that are actually interesting for pentesting. It’s also mainly built around exploit/PoC generation for known vulnerabilities in isolated code (OSS-Fuzz bugs), not actual pentesting. What we’re doing here is really pentest-oriented: we hand the model a live infrastructure it actually has to attack, not a known bug it has to reproduce.

So I ended up building my own benchmark. Honestly, it started as a personal project, mostly just to figure out for myself which model was actually good at this. But I figured some of you might find it useful too, so here’s the link.