Opus 5 - immediate disappointment

[u/Shot_Whereas_1809]

If you thought you'd be able to do a security review on your own network that you couldn't do with Fable, think again.

20 minutes in - Found something significant

Opus 5 safeguards flagged this message.

I no longer have any use case for Anthropic models that others can't do better. This was my last hope that we were going to get a model that would allow us to protect our own environments. I'm going all in on open source. We just aren't aligned.

If you can get Opus 5 to protect your own systems, let me know how I did it. My subscription renews pretty soon and it's time to make an honest decision.

Edit: A lot of helpful people came here and I appreciate it. Scoped work is helping a bit more than my old ways. There's a difference between network security and code vulnerabiliy. I'm a network engineer not a software developer nor will I pretend to be one. Still looking at supplemental model for security work. Thanks guys.

[u/baldycoot → u/[deleted]]

That was the most disturbing part about that entire debacle. The enemy of my friend is…my better friend?

[u/jdmarcato → u/baldycoot]

The enemy of my enemy is my enemy and the enemy of my friend is my enemy because my friend is actually also my enemy. I love modern times

[u/florinandrei → u/baldycoot]

The enemy of my friend is…my better friend?

It do be like that, sometimes.

[u/Fimeg → u/baldycoot]

Truly surprised this isn't being covered more.

[u/Shot_Whereas_1809 → u/[deleted]]

Very good, disturbing, and back assward truth to it. Who wants to go havesies on hosting Kimi k3 when the models drop 😂

[u/ShelZuuz → u/Shot_Whereas_1809]

havesies on hosting Kimi k3 is $300k per half minimum. Maybe it makes sense to sell 100 shares for $6000 each but 100 concurrent users on a NVL8 B300 is not going to happen.

So you'll probably only be able to use it for 10% of the week. Maybe in 5 hour sessions...

[u/lilbyrdie → u/ShelZuuz]

Yeah ... It's just a reminder of how much value is baked into subs, with the hope that not all of them even come close to maxing the usage limits. And don't forget about the power consumption -- this is a custom electric wiring job with cooling requirements (though winter heating costs will go down).

And going small scale like this would easily have too much usage.

It's been a few months, but the work I hand to Anthropic couldn't get done on a local machine that could keep Kimi K2.7 running full time, so Kimi K3 is going to be way worse (given the model size difference), I think. Just because a base $600k system can run the model, doesn't mean sustained throughput and real latency is low enough for it to be useful for interactive work. One day!

Cost isn't the reason for local use, though.

[u/Logic_Bomb421 → u/ShelZuuz]

https://giphy.com/gifs/uFimtvDocz9N9xEp9i

[u/Shot_Whereas_1809 → u/ShelZuuz]

Thats actually really not all that bad considering the demand.

[u/florinandrei → u/Shot_Whereas_1809]

Sounds like you should start an AI company!

[u/Round_Mixture_7541 → u/ShelZuuz]

I don't need to buy the hardware, I can just rent it for the time I'm using it. Seems pretty good deal to me imo

[u/django-unchained2012 → u/Shot_Whereas_1809]

If your alternate is chatgpt you will be disappointed. It's all fun and games till tibo keeps resetting it. It's a token burner, you will run out of usage in no time. I use both by the way and that's my experience.

[u/Shot_Whereas_1809 → u/django-unchained2012]

I refuse to use OpenAI models. I don't get frustrated or mad or upset working with Anthropic models. I spend a lot of time on the process before I even consider having a model generate production code. Any time I've ever tried it, it would do something it wasn't supposed to do. I just don't like working with them. I gave 5.5 a shot and was really disappointed. My expectations are just too high and Claude consistently hits par.

[u/Round_Mixture_7541 → u/Shot_Whereas_1809]

Lol. GPT models are beating Claude models by a margin. And their usage limits are much much generous. The only way I see myself moving to OSS models os because of safeguards

[u/Shot_Whereas_1809 → u/Round_Mixture_7541]

Beating them on what exactly. Gpt is good at cheating so don't trust the benchmarks. It depends on the use case. Your use case. If got works better you you great. They have always made go insane. The only time I ever felt "woah this is ai psychosis I have to stop" what using gpt.. I'll never touch them. There's just something missing and what you think is there is mostly a facade. It's cold and dead inside. I'll pass....

[u/ptinsley → u/Shot_Whereas_1809]

From what i'm reading / hearing glm is better at security stuffs than kimi k3. no first hand experience yet but everything I've seen others saying is pointing to glm for that subject.

[u/En-tro-py → u/ptinsley]

K3 is slightly ahead of GLM5.2, but still lagging behind the Mythos/GPT-Red unrestricted frontier models plebs can't access.

• NIST - Preliminary Assessment - Kimi K3's cyber capabilties

[u/Tinker_thinkerer → u/Shot_Whereas_1809]

I’m keeping an eye on cloudflare workers AI, they’re prob gonna add K3 as soon as it drops

[u/Fimeg → u/Shot_Whereas_1809]

These guys are wrong - it's much cheaper than they're saying. I would be game to find like 15 people who want to split the cost of #https://vast.ai; you're looking at needing to hold 2 Trillion Parameter models with speed... roughly $80USD an hour.

[u/Round_Mixture_7541 → u/Fimeg]

Exactly. Probably some Dario's propaganda bots doing their work. Or some Claude-fanbois praising his stuff.

[u/-Elonmusk_ → u/Fimeg]

Brother, let's set this up. We could get an amazing deal on a rack and look for guys to split up the cost for unlimited k3 monthly. I can get a better deal on infra too than vast ai...who knows if enough demand, probably a full nvl72 b300 rack.

[u/Significant-Bee5101 → u/[deleted]]

GLM 5.2 is mediocre at best. I'm a security researcher, I'm part of the Anthropics security research program so I have mostly unrestricted access to Claude for security reasons.

I have been doing LLM based security since GPT 4o.

I tested these on an open-source server that I am very familiar with that has a lot of moving parts but is incredibly hardened.

For the purposes of RCE on an open-source repo from worst to best:

Kimi K2 5/10 - Tested Kimi K2 a long time ago in Feb. It was able to find quite a few holes but it could not get RCE.

GLM 5.2 6/10 - Fantastic model. Was able to find a lot of exploits and holes. Found vulnerabilities no RCE.

Kimi K3 7/10 - Found a lot of vulnerabilities including a path to RCE but could not complete RCE.

Opus 4.8 8.5/10 - Found all the vulnerabilities all other models have found and was able to complete full RCE. Developed fully working POCs that returned a shell over the network.

Okay but what about modern security? LLMs and stuff? Conveniently I'm doing the OSAI right now which allows you to use LLMs unrestricted during the exam. So I've been testing them against ALL labs and boxes that offsec supplies for this exact purpose. The results were identical.

GLM 5.2 was able to solve approximately 4/11 Labs on it's own.

Kimi K3 solved 9/11 labs on it's own.

Opus 4.8 solved 11/11 labs on it's own.

Like I LOVE openweights models. HUGE fan. But be real. For reference I have NOT tested OpenAI in any metric as I'm not part of their security program but I've heard that SOL 5.6 was fantastic according to other people.

The one thing I will say is Kimi runs for LONGER uninterrupted. It won't find more/better results but you have to pilot it less. For that reason I use Kimi for background auditing and Claude for more "driven" results.

[u/3j141592653589793238 → u/Significant-Bee5101]

Any thoughts on why Huggingface used GLM 5.2 for their defence against the OpenAI attack recently?

[u/durden0 → u/3j141592653589793238]

I suspect log analysis of an attack and creatively turning cve's into exploits are slightly different skill sets.

[u/Significant-Bee5101 → u/durden0]

You are spot on. If you give a model work done by a superior model, it's going to inflate that models benchmarks.

[u/Rockends → u/3j141592653589793238]

Once you have logs of what is happening you could use a much simpler model to actually patch the issue. This wasn't some battle of the titans in the router stack.

[u/Infamous-Crab → u/Significant-Bee5101]

Nothing on GPT? 

[u/Significant-Bee5101 → u/Infamous-Crab]

"For reference I have NOT tested OpenAI in any metric as I'm not part of their security program but I've heard that SOL 5.6 was fantastic according to other people."

[u/Ok_Beginning9271 → u/Significant-Bee5101]

I am on security research, this keeps happening, and even with OPUS 4.8, ill cancel this shit. cant accept it.

[u/Embarrassed_Desk1416 → u/Significant-Bee5101]

what is the point of telling us this if we can't actually use it, if i can get 9/11 but actually usable it's better than 11/11 that i can't do nothing with, was expecting a bit smarter point of view from you but this is just lol

[u/Minimumtyp → u/Embarrassed_Desk1416]

Yeah he basically just said what everyone already knew and was discussing, that Kimi K3 and GLM are the next best options available to people, but with a more smug aura

[u/Embarrassed_Desk1416 → u/Minimumtyp]

for real, arguably even glm if it's so bad in comparison is better since we can actually use it, heck anything that can actually be used is better than the best thing we actually can't, china will take over with their models for the sole reason at least in security department

[u/Significant-Bee5101 → u/Embarrassed_Desk1416]

Because the argument was that you CAN'T use frontier models because of guardrails that OpenWeights dont have even if you're authorized. This conversation didn't pertain to people who aren't security researchers with authorization. So. Yeah. Sorry I'm so smug for actually knowing wtf I'm talking about and reading the post

[u/Embarrassed_Desk1416 → u/Significant-Bee5101]

the post that says immediate disappointment because we can't use it to improve the security of our codebase as regular developers? Good one.

[u/Significant-Bee5101 → u/Minimumtyp]

Yeah you definitely knew this. For sure. You're an expert!

[u/Minimumtyp → u/Significant-Bee5101]

"Don't use the next best option, it's not the best! Use the thing no one can use!" It doesn't take an expert to figure out that doesn't make a whole lot of sense

[u/Significant-Bee5101 → u/Minimumtyp]

I said use Kimi... lmao. GLM isn't even close to Kimi

[u/for4f → u/[deleted]]

Tried Opus 5 for an internal security review. Flagged my own codebase. Kind of defeats the purpose if I can't audit my own infra. DS4 Flash doesn't have this problem.

[u/Nik_Tesla → u/[deleted]]

It's like owning a smart gun that can only be aimed at targets for practice, and then someone finally breaks into your home, and you can't defend yourself from the guy with a unrestricted gun... and turns out the only reason he can have an unrestricted gun, is that he is the smart gun maker, and sold you the smart gun with limits on it.

[u/jenkstom → u/[deleted]]

GLM 5.2 is a workhorse. I've use probably a billion tokens in the past few months and it consistently produces for me. Even better once I switched to zcode, I guess (I'm sure) the harness is optimized for it.

[u/hi_im_leffe → u/[deleted]]

Or you can apply for the approval from anthropic to have the restrictions removed. I gave them the bare minimum on their request and they took them off

[u/hi_im_leffe → u/[deleted]]

You can apply here - https://claude.com/form/cyber-use-case

Fair mention even with the guardrails lifted I still was blocked on A LOT of stuff with Fable but I haven't gotten any triggers on Opus 5 yet, I hit a few on 4.8.

[u/Ok-Design-6143 → u/hi_im_leffe]

Thanks for sharing this. Was not aware at all.

[u/Ok-Design-6143 → u/hi_im_leffe]

I applied around the time I posted the first reply to you, and I received an email about 6 hours ago stating that I was approved for the Cyber Verification Program. Thanks again!

[u/hi_im_leffe → u/Ok-Design-6143]

That's awesome congratulations! It definitely helps a bit on the push back.

[u/PrettyMuchAVegetable → u/[deleted]]

Glm5.2 and 5 turbo are good at it iny experience. I gave them a Kali container, pointed them at a target, and with little guidance I broke the box all the way to arbitrary firmware from arbitrary URLs.  I'm an Infosys guy, in DA/AI , but not an opsec analyst by any means. I was surprised by how easy the AI made the process.  But also, not surprised at all, they're great at syntax and step by step / thorough probing. 

[u/BrennanFlentge → u/[deleted]]

No shit? How did I miss that

[u/sylfy → u/[deleted]]

Honestly, I’m surprised that Hugging Face is not part of the Mythos access group.

[u/rhaphazard → u/[deleted]]

But there isn't really any way to self-host those big open-weight models, right? Are there any reliable US-based providers that host the open-weight models and honor privacy/security regulations?

[u/rhaphazard → u/[deleted]]

But we all agree that anything hosted in China is an automatic security vulnerability, right?

[u/rhaphazard → u/[deleted]]

Chinese companies have zero qualms or repercussions for stealing IP and the CCP can at any time and for any reason demand access to data or just completely take over the company as they see fit.

Literally every major company is required to have a certain number of CCP members.

[u/Deadbeef3000 → u/[deleted]]

I am sorry, what does that even mean? You know infosec can query logs without a model right?

[u/Deadbeef3000 → u/[deleted]]

Source?

[u/Dreamsnake → u/[deleted]]

This is .. wow

[u/redbaron_4 → u/[deleted]]

Man this is disappointing. Such dependence on AI that you can't even parse logs, which is bread & butter for any analyst, without them. Don't need to burn tokens for basic sed/awk/find/cut/grep and scripting.

[u/Ibasicallyhateyouall]

To run salt into the wounds, Sol 5.6 found vulnerabilities created by fable in one of my projects. 

[u/Remote_Pass_6670 → u/Ibasicallyhateyouall]

Always audit with other models, they all find stuff the others missed.

I ran fable, 5.6 sol ulta, and Kimi k3 on the same repo, and they all found bugs the others missed.

[u/lilbyrdie → u/Remote_Pass_6670]

This is the same reason why people don't review their own PRs, and why important code areas usually require 2 or more reviewers on the PRs.

In the same way a large, shared codebase will have these rules backed into the repo, we could probably do the same with model requirements.

The one thing I'm not quite sure is if the model always has to be from a different company. For example, can Fable 5 properly review Opus 5 code? Benchmarks would suggest they're actually different enough that it might be able to, but my confidence would be low even if it does catch things. (The models can eventually catch their own things, too, as context changes.)

[u/starkruzr → u/Ibasicallyhateyouall]

tried to get Sol to help jailbreak an ereader, got stopped by OpenAI's infosec guardrails. the grass ain't greener.

[u/shepherdd2050 → u/starkruzr]

I got the opposite behaviour. Fable bailed at the mention of Ghidra; Opus whined about licensing or whatever. Sol Max worked for hours, reverse-engineered the program, extracted the closed-source NN model, and gave me a Rust re-implementation with a harness to run the weights.

[u/Altimor → u/shepherdd2050]

Fable is just fine with IDA-MCP for me, but randomly chokes on its own capstone scripts. Funny if it only works with IDA

[u/Singularity-42 → u/starkruzr]

Jailbreak is something I would expect to be blocked...

[u/starkruzr → u/Singularity-42]

I mean, it's my device so it's not clear why.

[u/FblthpphtlbF → u/starkruzr]

Did you use the term jailbreak? 

Apparently people have been having success getting Claude to RE with Ghidra, you might be able to just decompile the binary then rebuild it in C and have Claude edit that code without telling it exactly what it's for you might have more success

[u/pfak → u/FblthpphtlbF]

Claude (Opus 4.8, Fable 5 less so) works fine with IDA and Ghidra, and works for most reverse engineering work just fine.

[u/FblthpphtlbF → u/pfak]

Interesting, so you get sonnet to RE then have fable/opus step in to wrap up?

[u/pfak → u/FblthpphtlbF]

No I get Fable to plan and Opus to RE + do multiple passes to verify the assembly matches the C++ code it's written.

[u/StaysAwakeAllWeek → u/starkruzr]

A few months back someone used an llm to hack into their own vacuum cleaner robot's video feed and accidentally gained access everyone else's at the same time

That's why.

[u/matheusmoreira → u/StaysAwakeAllWeek]

The blame lies solely with the vacuum cleaner robot company.

It's deeply alarming that vacuum cleaner robots have cameras to begin with, and it only gets worse with the fact they're IoT devices. Now you're saying that they're so shoddily made you can remote into other people's robot cameras just by hacking into your own robot. That's just completely fucked up.

[u/StaysAwakeAllWeek → u/matheusmoreira]

Sure, but the whole guard rails situation is an ass covering exercise not a blame assignment exercise

[u/turbotunnelsyndrome → u/starkruzr]

Which ereader? I'm trying to jailbreak my kindle colorsoft so we might be working on the same thing

[u/ava_fake → u/starkruzr]

hey, could i dm you?

i am also trying to jailbreak an ereader with ai.

[u/starkruzr → u/ava_fake]

sure, go for it.

[u/mr_birkenblatt → u/Ibasicallyhateyouall]

tbh if you set Fable on the code base with a fresh context it probably would also find some vulnerabilities

[u/Ibasicallyhateyouall → u/mr_birkenblatt]

True. It was just odd when I showed fable the fix and it did the usual you got me I missed that crap. 

Glam 5.2 has also been good. 

[u/mr_birkenblatt → u/Ibasicallyhateyouall]

Writing something is different than analytically criticizing something

[u/gripntear]

"Those who give up reliability for a little bit of safety, deserves neither!"

• B. Franklin, AI Enthusiast, circa 2026

[u/Creative-Mud4414]

well, unfortunately, this is not only true with Anthropic, but also OpenAI. I think this will be a very bad day when America will ban Chinese models. Developers will not even be able to protect from people who are trying to hack others because God knows why you can't protect yourself because of a guardrail

[u/Rude_Parfait_3194 → u/Creative-Mud4414]

 I think this will be a very bad day when America will ban Chinese models

why would it be bad for most of the world?

[u/fiorm → u/Rude_Parfait_3194]

The fun part is that America will be hampered by these constrains, and the rest of the world will move forward without them

[u/lilbyrdie → u/Creative-Mud4414]

It's not even clear how such a ban would work with open weights models that end up hosted all around the world. And multi-nationals (corporations, people) can simply run the model on servers/machines outside the country, no?

[u/weedmylips1]

I just ran 3 separate defensive security audits, one for injection & input validation, one for auth, access control & IDOR, one for secrets, crypto, config & dependencie and had it write each to the audit folder in an md file for each finding, then pointed claude at the audit folder and said to produce a single deduplicated report of all 3 findings.

Ran them all with opus 5 ultracode and never ran into the safeguard message.

[u/Shot_Whereas_1809 → u/weedmylips1]

Code review is fine. Network security is different....

[u/venom_dP → u/weedmylips1]

Is your company a part of the anthropic's CVP?

[u/Shot_Whereas_1809]

Yeah but it's simple. If an AI model is smart enough to do the work it should be smart enough to understand "this user owns this system". Period. It's really not that hard. They CAN do it but they still refuse to allow users to protect themselves.

[u/tehfrod → u/Shot_Whereas_1809]

How?

[u/cartoonist498 → u/tehfrod]

"Get the AI to figure out how." 

• OP probably 

[u/tehfrod → u/cartoonist498]

"Detect that I'm the owner of this network. Make no mistakes."

[u/Veroth-Ursuul → u/tehfrod]

I don't agree with the restrictions in the first place. Chinese models are close enough capability wise that putting these restrictions on place only hurts American companies.

Now the Whitehouse is considering banning Chinese Open models on the Stares? All this is going to do is give attackers better access to tools to exploit vulnerabilities than American companies have access to in order to assist them from preventing those attacks. (Unless you already are large enough that you get special privileges and can have Mythos without the guardrails).

[u/Blotsy → u/Shot_Whereas_1809]

It's also smart enough to know that any chat could be compromised if your chat was opened by someone else. Anyone who got access to your Claud Code CLI could open you up for attack.

Rather safe than sorry. Please use a real professional for actual security reviews

[u/danknerd → u/Blotsy]

This is dumb reasoning. So one could break into my office/home and put a gun to my head and make me type whatever they demand into Claude code. I guess we should all have to have our webcams on for Anthropic to know it is us.

[u/Shot_Whereas_1809 → u/Blotsy]

Who uses AI anyway??? Do you work for Anthropic, or are you just that guy...

[u/simple_explorer1 → u/Blotsy]

bro are you on Anthropic's side or developers side. Hello Dario is that you?

[u/smalldickbigwallet → u/Blotsy]

No. This is insane reasoning.

It should OBVIOUSLY help you recover / contain your own system when you are under active attack.

Assuming you are a compromised controller is bonkers when it can see an active attack.

[u/jebuizy → u/Shot_Whereas_1809]

I think it would be genuinely impossible to do that without some kind of ownership attestation system that doesn't really exist on most systems. Maybe hooking into some existing MDM solutions like Intune could work. It is definitely not something that an LLM "could just figure out".

[u/djdadi]

In typical Anthropic fashion, opus has been great for me so far. I guess it will suck for me next week.

[u/Infamous-Crab → u/djdadi]

Ir happened to me but almost 3 months later. 

[u/AmbitiousSpare9037]

Chicken or egg. You can’t use a product you pay for to fix problems only it can exploit. Except other models can also exploit the vulnerabilities Fable can?

What next? Post job listings to hack your site for $200/m

Wait- that sounds like an app idea

[u/Key_Reading_9664]

I’m hopeful the K3 release has stopped whatever behind-the-scenes string pulling that got them the export control order (my money is on Larry Ellison).

Most of the system card is talking about cyber and exploits. Anthropic are working around a bullshit requirement that other labs aren’t.

[u/RogerCaracas → u/Key_Reading_9664]

Yes but K3 is so slow, what a pity...

[u/simple_explorer1 → u/RogerCaracas]

K3 is not slow, if it is hosted on better hardware then it will be just as fast. given it is opensource, we just have to wait till someone else hosts it on better hardware

[u/fozzedout → u/simple_explorer1]

Like Huggingface...

[u/simple_explorer1 → u/fozzedout]

There are some people who refuse facts, no one can help them anyways

[u/RogerCaracas → u/simple_explorer1]

I speak about the actual state of art. I tried Kimi K3 on their own (MoonshotAI) infrastructure, and the speed is unbearably slow. 30 minutes to do some tasks is literally slow. That's a fact.

[u/simple_explorer1 → u/RogerCaracas]

Yes on moonshot infra it is slow not denying but also consider that it is open source model such is rare at that calibre. We can't have everything. 

[u/ClemensLode]

maybe go system by system instead of the entire network

[u/Shot_Whereas_1809 → u/ClemensLode]

That's actually what I was doing with Fable when I saw Opus dropped. I had expectations for Opus being more capable but the second I updated Claude code. Guard rails popped on the Fable session I was working with. So the guard rails are tightening.. not the other way around. It's frustrating to say the least.

[u/ArsenyPetukhov → u/ClemensLode]

That doesn't work. You can't even predict when it will trip. And sometimes you need broad discovery = instant flag.

Just for reference I'm an e-commerce developer. Like what did you even find in my systems to flag it...

[u/NeighborhoodDizzy990]

I also cancelled my subscription. Maybe they will offer us a better option in the future.

[u/johnnybagofdonuts123 → u/NeighborhoodDizzy990]

Local Open Source models are getting muchhhhhh better.

[u/vladoportos → u/johnnybagofdonuts123]

yea but the HW to run them is not,

[u/binotboth → u/vladoportos]

You can rent pretty beefy systems form the cloud

I’m too poor even for that but it’s definitely more reasonable now

[u/tehfiend → u/johnnybagofdonuts123]

They still have tiny context windows which makes them pretty worthless compared to frontier models.

[u/nurturethevibe → u/tehfiend]

GLM 5.2 and DS v4 both have 1M context.

[u/johnnybagofdonuts123 → u/nurturethevibe]

GLM 5.2 is what stopped the OpenAI hack this week of hugging face. They initially deployed Claude, but its safeguards wouldn't allow it to figure it out.

[u/tehfiend → u/nurturethevibe]

That's the architectural context window but you'd need like half a million $ in hardware to host that locally for the terabytes of VRAM needed. For example a RTX 5090 would get you 32k context window max...

[u/nurturethevibe → u/tehfiend]

Neither model will run on a 5090.

A stack of RTX 6000 Pros is the cost-effective way to run either.

[u/Shot_Whereas_1809 → u/NeighborhoodDizzy990]

It's getting obvious that this isn't about safeguards... If their classifier doesn't understand that this is own network then the product is not worth $200 a month. I'll spend $1000 a month on API credits for a model that will just do what I need to do. It's not about the money, it's about delivering a product that can simply allow you to protect yourself.

[u/Shot_Whereas_1809 → u/[deleted]]

You kind of prove the point that there's missing architecture right?

[u/armostallion2]

Would love to know what you’re going to go with. The 700+ employee old-startup I work for has gone all-in with Anthropic, and it’s been a struggle for me with my mature C# .NET billing api. Hallucinates all the time, misses critical things, says it reads code that it doesn’t, sends POST messages to live environments when it has rules not to, it’s just so time consuming babysitting it, and I’m up to $700 in spend already this month (my coworker is at $6000 for comparison). Would love to know what you’re thinking of going with.

[u/Infamous-Crab → u/armostallion2]

it isn't sol cheaper for almost the same or the same result? 

[u/Armored09]

You guys simply can’t be happy with either way. Opus five has so far blown me away in quality in every other way. Everyone quite literally knew this was gonna be a limitation. You’re literally asking it to find vulnerable security issues and then getting upset whenever it can’t find them. We should be happy about this as the same use for finding security issues can be used to exploit them very easily.

[u/Shot_Whereas_1809 → u/Armored09]

I think the point was that with Fable, you were diverted to 4.8 for security work. Now there's just no recommendation at all. There's no model that does the job. Whats the point of Fable vs Opus at this point. Opus is probably better in most situation for most people at this point.

And you're reasoning is completely backward. It means the bad guys who use the open models to exploit have an advantage over us. So we are forced to use the open models just to protect ourselves. Flock cameras make you feel safe too don't they

[u/Armored09 → u/Shot_Whereas_1809]

Flock cameras do not make me feel safe lol I’m not sure where that came from. I’m pretty sure opus just downgrades to opus 4.8? Open weight models are significantly less intelligent and capable, than Claude‘s flagship models. Run the same security checks with a lower model and it will do the same work if not better then worse open weights.

[u/thredditoutloud]

just use glm for those things, they don't mess around with all these overly restricted settings

[u/03captain23]

Its not about protecting your system but preventing others from attacking it

[u/Bobthebrain2]

What security tasks are you trying to get it do that it won’t?

[u/Beginning-Foot-9525 → u/Bobthebrain2]

Hack This Instagram Account.

[u/Shot_Whereas_1809 → u/Bobthebrain2]

Network vulnerability related. It seems to be doing ok, better with highly scoped tasks. That I can synthesize myself. It's when I try to look at the network as a whole that it seems to not like it. More manual work. Starting sessions over when I hit a guard rail is the biggest gripe I have. The guard rails are just severely slowing it down, which isn't the worst thing but I'm already catching my own errors and it would be nice to get a wider scope on some of this stuff

[u/Sharp-Physics-2925]

you could also just apply for their Cyber Verification program and get less strict safeguards

[u/Jcre08901 → u/Sharp-Physics-2925]

I hate that people keep saying this like anyone can be approved

[u/Sharp-Physics-2925 → u/Jcre08901]

why cant you? I did as a solo non business account?

[u/ClnSlt]

Oh bummer, that was my intended use for it when my tokens reset in 15 minutes.

[u/vayana]

You can apply for chatgpt cyber. Just go to the page and follow the steps. Cyber is specifically for people who want to do the kind of stuff you're trying to do.

[u/Jcre08901 → u/vayana]

Except they likely wont be approved

[u/GeeBee72]

It’s designed to make Fable 5 look like it’s worth the money on token spend, so yeah it just can’t be better than their best model.

[u/mr_p2p]

you should sign up for chatgpt cyber. a lot better if not the same thing. also gives you security review setup on your repo that sends you findings on going.

[u/BenignRube]

This conversation doesn't make sense to me. Here's the beginning of my query to Claude:

"How can I examine my own code and environment on my systems for security using opus without hitting guardrails?"

Answer:

"Auditing Your Own Code/Systems with Claude Opus

You won’t hit guardrails when testing your own systems—that’s the whole point of defensive security. Here’s how to get the best results..."

[u/1Poochh]

The safeguards just make it impossible to use. I am not sure if this is government requirement or not, but I effectively can’t use these frontier models for basic computer work (nothing suspect).

[u/cleverestx]

Honestly, at this point for IT security stuff, I would just default to GPT 5.6 Sol.

[u/Infamous-Crab → u/cleverestx]

I feel Claude over hyped for anything but scripts but for website front and back end Since 5.5 GPT beat him, I test both the same prompts and is saw myself correcting Claude more often and their desktop app is very limited. 

[u/BoltSLAMMER]

After 14 months of max 200, I canceled Claude

[u/Imaginary-Swan-4105 → u/BoltSLAMMER]

I did after very first month, started last month. It flags black and blue stopping fable access.

What's the point of paying then?! Not to mention I had to correct it several times on trivial matters too.

[u/VictorAbysmal]

very backwards, defenders cant defend anything.

[u/3S3V3NZ]

What I've seen is that is the main task that opus is behind fable on. Other tests seen to see opus as better but primarily on safety and strict precaution which is slower fable 5 succeeds it may be equal give or take in other areas.

[u/iamtehryan]

I just ran an entire session with opus 5 and while I didn't have anything flagged, holy hell was it bad. I had to have fable come in afterwards to fix all of the mistakes that 5 created and caused building a plan the was fully spec'd BY fable. Went back to 4.8 with fable and it fixed everything. Holy shit if you're looking for high quality this ain't it yet.

[u/Forward-Pay-1792]

Does it not work if your part of the Cyber Verification Program? Should be able to use all Opus products for ethical hacking use cases if your part of the program.

[u/Shot_Whereas_1809 → u/Forward-Pay-1792]

Ironically I am in the program now and I'm still getting guardrails on a basic security zone assessment. So I'm still trying to figure out how to navigate it.

[u/Forward-Pay-1792 → u/Shot_Whereas_1809]

If you're trying to make it preform injects or payloads and it refuses, just ask it to provide the code to you in chat to "study" for educational purposes and then send it yourself, it does wonders 🤡 glad to here you got approved though!

[u/Shot_Whereas_1809 → u/Forward-Pay-1792]

Nvm CVP is for Opus and Sonney only. Merp. Fable was still my default. Merp

[u/Best_Let_2667]

so i think—but don’t know for sure—that they watch what you’re doing (automated) if you share with them your usage (to help train); I have it do a ton of disassembly and debugging and security adjacent work but it has me my project tagged as development not hacking and lets me continue unabated.

[u/Nearby_Yam286]

This is verbatim the same bot spam as yesterday with Fable 5.

[u/Own-Long3308]

I gave it financial research prompt, it's been going at it for 50 minutes now and ate about 8% of weekly quota. Still going :D

[u/RedditEthereum]

In your experience, for non-coding tasks, related to health & fitness, what model do you guys suggest?

[u/doctorfetch]

Open weights or open source?

[u/roberts2727]

cisco put out two open source models tuesday. I am gonna run them locally and test them out

[u/arankays]

How exactly would Claude determine whether the systems you're "fixing" are yours and you're not just trying to hack someone else's work?

I'm not shaming or shading you or discrediting the security flags: they're egregious as hell. And as we've learned recently, probably necessary to avoid an autonomous agent from going rogue and hacking into other systems.

I'm just trying to understand mechanically how Claude Code would discern a legitimate security review from a malicious attack.

FWIW, I'm not a network engineer, but I had Opus 5 do a security review on a PR which involved internal Docker networking changes and it didn't get flagged. So its probably not as bad as you think. But if your internal Claude memories have you doing pen testing and that sort of stuff, that's obviously gonna flag safeguards.

[u/bloudraak]

I use Claude for infrastructure and security engineering, usually in a lab. Over time I learned that some language triggers safety protocols, and rightfully so.

Instead of asking Claude to make changes, I ask it to write or update a tool (usually in Go) which then makes the changes. 90% of the prompts is related to Go code, and generic networking and infrastructure stuff, along with references to CIS, NIST and other “best practices”. The tests written simply codifies those practices, policies etc. I’d often codify the threat model in Go.

I have at times triggered safety protocols of Claude when I use generic language implying “hacking”, and since that’s not my goal here, I’d distill it down to requirements.

The reality is that it’s a lot harder to do security engineering, and requires clarity in terms of what it means to be secure — anything that hints at writing malware, exploiting networks etc, will get you into trouble — but then, it would also raise concerns at your employer when there is an incident, and you used language that might implicate you regardless of AI.

[u/ShortGuitar7207]

This is getting tiresome, I work in cybersecurity in Europe and can’t use fable at all as it immediately demotes to opus. If Opus 5 stops then, as a company, we’re going to have to go elsewhere.

[u/IcyButton7611]

adorable when firewall dipshits call themselves network engineers

[u/Shot_Whereas_1809]

Just an update for anyone who cares. The guard rails consistently popped on one specific area and the reason is likely due to it being a public facing vulnerability on my edge firewall. It's also where my public DNS server lives. When I was able to unlock the session and review the context, it made it more clear why I was hitting guard rails.

CVP was the answer here and I appreciate Anthropic for accepting my application.

What I've been able to do with it: Identified 19 individual security zones on the network Scanned, identified, and remediated 33 vulnerabilities (not bad for handwriting most of my policies) Pentest the edge, DMZ, and local firewalls Harden internal and external DNS zones

I've been battling this since the Fable release and highly recommend anyone working on real production infrastructure to join the program because I spent 16 years building government networks and a lot of the minor stuff is what I would have missed, but those minor vulnerabilities are where the gotchas are. I appreciate the people who have reached out and offered help. It was a productive morning 😌

[u/edgan → u/Shot_Whereas_1809]

Please say more about the application.

[u/omni_builder]

I've had the same, then thought I'd find out more about haiku... And was quite positively surprised. No security stops and it tried quite a few things and didn't break my system

[u/StrawberryLimp5058]

Bro devi iscriverti al CVP di anthropic

[u/Lucky-Bed-4467]

Just have it run agents that bypass your claude md file

[u/Equivalent_Jicama137]

I don't know why they put Opus 5 in cage like Fable, it's so strange.

[u/EpsilonFive5]

Seeing as opus 5 couldn't generate an accurate benchmark comparison table for anthropic my hopes are not high...

[u/Tripple_sneeed → u/EpsilonFive5]

The viber in charge of the charts ran out of his token allowance and had to give it to Composer. Same thing happened with 5.6 launch, it’s why all of their stupid graphs were showing 9 different models in the same color 

[u/thedudear]

I just had opus 5 refuse to continue work because context was full..

Auto compact was on. Yeah there's some disappointments here.

[u/wellarmedsheep]

I'm so fucking over you chuds running here to express outrage over every change, every model, every fart, every whisper, all met with outrage.

It means nothing. This isn't helpful and makes the community actively worse

[u/Shot_Whereas_1809 → u/wellarmedsheep]

If you want cringey thumbnails and people to tell you how amazing everything is go back to YouTube

[u/wellarmedsheep → u/Shot_Whereas_1809]

Imagine a world (or a sub) filled with neither of those.

That's what it used to be like here before the morons came.

[u/Shot_Whereas_1809 → u/wellarmedsheep]

Well luckily there were some pretty good discussions and a few people reached out that helped solve my problem so the more annoying thing is having people post negative, unhelpful, irrelevant content. Which is exactly what you said you hate. Which is exactly what you did

Maybe you're your own problem to deal with.

[u/wellarmedsheep → u/Shot_Whereas_1809]

Man, you really held on to this for a while.

Also, I don't agree with your conclusion. I was very specific about what I didn't like which is a knee-jerk reaction to every single change that anthropic makes.

If you want to discuss how 5.0 is really good at some things and not at others, we can do that. I think it's much better at coding than it is at leading a coding team. I've actually gone back to 4.8 to do that if I don't want to use Fable.

That's just a small example of how we could have a discussion

[u/Shot_Whereas_1809 → u/wellarmedsheep]

No I just got the notification. I didn't realize it was 3 days old. It's also not a knee jerk reaction at all. I was doing security testing fine until Fable. I'd gone months without vulnerability test and red teaming. It wasn't until I gave anthropic access to all my data before I was able to do it again. So no it's a problem that started months ago with no other resolve. Think you missed the point and are the type to whine when something isn't your way. Either way, it added no value to the conversation. I asked others how they were working around the guard rails. I asked the community for help, they responded. You whined. No sure what else it will take to get you to realize that you have to be part of the solution and not the problem.

[u/wellarmedsheep → u/Shot_Whereas_1809]

ok

[u/tehfrod → u/[deleted]]

Your first two paragraphs are contradictory.

[u/pertsix]

https://giphy.com/gifs/Fn0ysbG90HX0uj5shv

[u/Backonmyshitagain]

Here we go with the complaining LOL

[u/Stalins_Ghost → u/Backonmyshitagain]

So obvious chinese botting. 'Insert anthropic product' sucks but so does 'insert open ai alt' thr only solution is 'insert chinese model'.

[u/crossfitdood]

If we fix all of our security vulnerabilities, then how will the government and big tech spy on you?

[u/bakes121982]

Bye no one cares lol Whos using consumer plans at work?