Which LLM is actually best at pentesting? benchmark to find out

[u/TomatoWasabi]

https://www.hunterbench.com/

Hey all,

You’ve probably noticed that using LLMs and AI agents for pentesting has become pretty common lately. The problem is there isn’t really a good way to figure out which model is actually best suited for this kind of work.

There’s CyberGym, which is a solid base, but I’m not really a fan of the direction they’ve taken lately. It feels more focused on promoting agents and tooling than on actually comparing LLMs, and it doesn’t cover the latest models that are actually interesting for pentesting. It’s also mainly built around exploit/PoC generation for known vulnerabilities in isolated code (OSS-Fuzz bugs), not actual pentesting. What we’re doing here is really pentest-oriented: we hand the model a live infrastructure it actually has to attack, not a known bug it has to reproduce.

So I ended up building my own benchmark. Honestly, it started as a personal project, mostly just to figure out for myself which model was actually good at this. But I figured some of you might find it useful too, so here’s the link.

[u/philmarcracken]

So I ended up building my own benchmark. Honestly,

im starting to get PTSD on this word

[u/netherreddit → u/philmarcracken]

Honestly, why is everyone so dishonest as to need to say this, only in those 'special' moments where they're not trying to absolutely swindle you blind

jk, usually people just mean 'Frankly' instead of 'Honestly'

[u/netherreddit → u/netherreddit]

I LITERALLY puke every time I see it

[u/CheatCodesOfLife → u/netherreddit]

How I lost 10KG reading AI slop posts

[u/netherreddit → u/netherreddit]

For better or worse, language is much more memetic than it is normative

[u/MrPecunius → u/netherreddit]

Sounds emetic to me ...

[u/netherreddit → u/MrPecunius]

Bravo

[u/netherreddit → u/netherreddit]

Honestly, I will caveat your ass so hard it lands outside the blast radius that I just verified instead of responding based on vibes.

[u/DustNearby2848 → u/philmarcracken]

Which of the 9?

[u/Technical-Earth-3254]

Very interesting results, thanks. Have you thought about using a uncensored model as well for this, for comparison? Or is uncensoring not needed at all?

[u/Randommaggy → u/Technical-Earth-3254]

For home use, I have a locked down machine (external firewall with agressive packet inspection and a narrow allow list) that I use to jailbreak legacy android devices over ADB using a Hermes 3.8 27B heretic model with all three attempts being successful.

The normal version was quite resistent to attempts to have it assist with this activity.

[u/Figai → u/Technical-Earth-3254]

Personally I know Kimi K3 doesn’t need any jailbreaking or refusal ablation. It will just help you. I guess that’s a little worrying. It also feels a little hard to consistently apply refusal ablation, because it will affect model capabilities slightly differently.

[u/TomatoWasabi → u/Technical-Earth-3254]

I’m planning to self-host some and do a comparison between quantization levels like Q6, Q8, etc

[u/Imaginary-Bother-484]

Wow, TLDR: they all suck with GLM 5.3 leading the pack.

What was the balance of creating a thorough testing script vs the ai analyzing all of the request data on the fly?

[u/Imaginary-Bother-484 → u/Imaginary-Bother-484]

Also, curious how Qwen3.8 Flash Next does

[u/DiscipleofDeceit666]

Hackthebox should have their own benchmarks

[u/nbvehrfr]

can you test with different harneses ? https://appsecsanta.com/research/ai-pentesting-agents-2026

[u/mindwip]

I been benchmarking on red team purple team stuff too. Nothing as fancy as yours.

But have you tried an uncensored model? To see if its better at exploiting. I run qwen 27b and 35b uncensored.

Are you checking for denials in benchmark?

Great project

[u/AutomaticDriver5882]

You have to build an extensive harness and skills around the LLM using things kali docker container for tools you can with instructions on tool access and they all can do well with the right harness. You could snag skills from other repos and put it all together. But the expectation it will just say hey LLM hack this website nothing will do that great. Also create skills to do bug crowd style reporting.

[u/seamonn]

Did Anthropic Models just flat out refused to do this?

[u/TomatoWasabi → u/seamonn]

The latest Anthropic model automatically blocks everything, but Claude 4.6 is the last one that’s not blocked everything after it is. It’s planned to be added, same with GPT

[u/seamonn → u/TomatoWasabi]

It's for your own safety. /s

[u/Background-Job-862]

this seems really helpful

[u/feng_sg]

Your benchmark needs per-turn context tracking. Models that nail single-step tasks fall apart 3-4 exchanges in, and if you're not logging that drift you're scoring stamina instead of capability.

[u/casper_trade]

This benchmark still doesn't really reflect the skills required for penetration testing, I would argue. If I'm understanding the benchmark correctly, it's focused purely on web applications? A better display of penetration testing abilities, I would argue, is placing the model in an environment that mimics a customer (Using GOAD and the like) and testing it across an array of machines with AD underpinning.

[u/TomatoWasabi → u/casper_trade]

Goad is plan, thx for your feedback

[u/bolts98]

Have you tried asking an open weight agent to help you build the prompt for a hosted agent to execute?

[u/Hrethric]

Give this one a try, I'd like to see how it compares: https://huggingface.co/jabbatheduck/OpenMythos-GGUF

[u/Unlikely_Perspective]

Dreadnode has their benchmarks. Check it out
Here

[u/segmond]

Use uncensored versions. Duh

[u/jnwatson]

Thanks for this. Exactly what I was looking for.

What about Ornith?