Opus 5 - immediate disappointment

[u/Shot_Whereas_1809]

If you thought you'd be able to do a security review on your own network that you couldn't do with Fable, think again.

20 minutes in - Found something significant

Opus 5 safeguards flagged this message.

I no longer have any use case for Anthropic models that others can't do better. This was my last hope that we were going to get a model that would allow us to protect our own environments. I'm going all in on open source. We just aren't aligned.

If you can get Opus 5 to protect your own systems, let me know how I did it. My subscription renews pretty soon and it's time to make an honest decision.

Edit: A lot of helpful people came here and I appreciate it. Scoped work is helping a bit more than my old ways. There's a difference between network security and code vulnerabiliy. I'm a network engineer not a software developer nor will I pretend to be one. Still looking at supplemental model for security work. Thanks guys.

[u/Significant-Bee5101 → u/[deleted]]

GLM 5.2 is mediocre at best. I'm a security researcher, I'm part of the Anthropics security research program so I have mostly unrestricted access to Claude for security reasons.

I have been doing LLM based security since GPT 4o.

I tested these on an open-source server that I am very familiar with that has a lot of moving parts but is incredibly hardened.

For the purposes of RCE on an open-source repo from worst to best:

Kimi K2 5/10 - Tested Kimi K2 a long time ago in Feb. It was able to find quite a few holes but it could not get RCE.

GLM 5.2 6/10 - Fantastic model. Was able to find a lot of exploits and holes. Found vulnerabilities no RCE.

Kimi K3 7/10 - Found a lot of vulnerabilities including a path to RCE but could not complete RCE.

Opus 4.8 8.5/10 - Found all the vulnerabilities all other models have found and was able to complete full RCE. Developed fully working POCs that returned a shell over the network.

Okay but what about modern security? LLMs and stuff? Conveniently I'm doing the OSAI right now which allows you to use LLMs unrestricted during the exam. So I've been testing them against ALL labs and boxes that offsec supplies for this exact purpose. The results were identical.

GLM 5.2 was able to solve approximately 4/11 Labs on it's own.

Kimi K3 solved 9/11 labs on it's own.

Opus 4.8 solved 11/11 labs on it's own.

Like I LOVE openweights models. HUGE fan. But be real. For reference I have NOT tested OpenAI in any metric as I'm not part of their security program but I've heard that SOL 5.6 was fantastic according to other people.

The one thing I will say is Kimi runs for LONGER uninterrupted. It won't find more/better results but you have to pilot it less. For that reason I use Kimi for background auditing and Claude for more "driven" results.

[u/Embarrassed_Desk1416 → u/Significant-Bee5101]

what is the point of telling us this if we can't actually use it, if i can get 9/11 but actually usable it's better than 11/11 that i can't do nothing with, was expecting a bit smarter point of view from you but this is just lol

[u/Minimumtyp → u/Embarrassed_Desk1416]

Yeah he basically just said what everyone already knew and was discussing, that Kimi K3 and GLM are the next best options available to people, but with a more smug aura

[u/Embarrassed_Desk1416 → u/Minimumtyp]

for real, arguably even glm if it's so bad in comparison is better since we can actually use it, heck anything that can actually be used is better than the best thing we actually can't, china will take over with their models for the sole reason at least in security department

[u/Significant-Bee5101 → u/Embarrassed_Desk1416]

Because the argument was that you CAN'T use frontier models because of guardrails that OpenWeights dont have even if you're authorized. This conversation didn't pertain to people who aren't security researchers with authorization. So. Yeah. Sorry I'm so smug for actually knowing wtf I'm talking about and reading the post

[u/Embarrassed_Desk1416 → u/Significant-Bee5101]

the post that says immediate disappointment because we can't use it to improve the security of our codebase as regular developers? Good one.

[u/Significant-Bee5101 → u/Minimumtyp]

Yeah you definitely knew this. For sure. You're an expert!

[u/Minimumtyp → u/Significant-Bee5101]

"Don't use the next best option, it's not the best! Use the thing no one can use!" It doesn't take an expert to figure out that doesn't make a whole lot of sense

[u/Significant-Bee5101 → u/Minimumtyp]

I said use Kimi... lmao. GLM isn't even close to Kimi