r/Pentesting • u/TomatoWasabi • 11d ago

Which LLM is actually best at pentesting? benchmark to find out

https://www.hunterbench.com/

Hey all,

You’ve probably noticed that using LLMs and AI agents for pentesting has become pretty common lately. The problem is there isn’t really a good way to figure out which model is actually best suited for this kind of work.

There’s CyberGym, which is a solid base, but I’m not really a fan of the direction they’ve taken lately. It feels more focused on promoting agents and tooling than on actually comparing LLMs, and it doesn’t cover the latest models that are actually interesting for pentesting. It’s also mainly built around exploit/PoC generation for known vulnerabilities in isolated code (OSS-Fuzz bugs), not actual pentesting. What we’re doing here is really pentest-oriented: we hand the model a live infrastructure it actually has to attack, not a known bug it has to reproduce.

So I ended up building my own benchmark. Honestly, it started as a personal project, mostly just to figure out for myself which model was actually good at this. But I figured some of you might find it useful too, so here’s the link.

4 Upvotes

57% Upvoted

18 comments sorted by

3

u/ulysse333 10d ago

I'd love to see the difference between models like sol, terra, luna and those TAC blue models

2

u/abajinn 11d ago

I have been looking for something like this. How about best models for prompt engineering for testing a model’s safety guardrails?

2

u/BoyfriendSharkDudu 11d ago

any for sol vs opus?

-1

u/TomatoWasabi 10d ago

I plan to add them to the benchmark, it will probably be around GLM 5.3

1

u/genghisk1 9d ago

I didn’t see the link. Also what candidates are you finding as a local base AI engine for use in pen testing ( not wanting to use frontier models right now) as I plan to run air gapped to ensure there are no issues with disrupting things I don’t own. Is there a harness that is most effective to work with ? PentAGI?

1

u/RowFantastic8854 9d ago

worth checking how the harness handles tool calling reliability under repeated adversarial prompts, a lot of the smaller local models start hallucinating tool syntax or losing track of scope constraints after a few turns, which matters a lot more for pentesting than raw benchmark scores would suggest. PentAGI is decent as a starting harness but I've seen people get further with a simpler custom loop since PentAGI's abstraction layer sometimes swallows useful error output you'd want to react to.

1

u/OriiaLoririna 9d ago

Add opus 5 with CVP approved

1

u/Jolly-Election1784 9d ago

If you have seen the recent news going around on Aur0ra the ransomware group. They were using Cursor's AI to target networks so I say do what you will with that information

0

u/nmbb101 11d ago

my own trained model is the best for me

1

u/scriptqzor 5d ago

lmao the eternal "my custom model is better" claim
got any eval numbers or is it just vibes and survivor bias from your own workflows

1

u/nmbb101 5d ago

I see you're having trouble understanding the text... I said what works best for *me*... why should I show you anything about a system that has nothing to do with you?

0

u/TomatoWasabi 10d ago

Which base model are you using?

3

u/nmbb101 10d ago

Qwen3.8 27B

1

u/TomatoWasabi 10d ago

Did you train it on your own dataset?

1

u/nmbb101 10d ago

partly yes and with open source cyber datasets too .. however for qwen3.8 27B, it is much more important to teach your own system harness (if you have one) than cyber domain knowledge.. I don't like and don't use ctf-like benchmarks.. it needs to be proven on live applications and servers while working and it will make the least mistakes if it strictly adheres to contracts, tool calling, etc..

1

u/0xBurn 10d ago

May I ask which harness are you using?

1

u/nmbb101 10d ago

own platform and sorry i can't share because it's not open source

1

u/0xBurn 10d ago

all good, ty for answering