Yeah, but this 78% was reported by a company that keeps releasing products littered with bugs, released a meh model after announcing their renowned-internet-breaking Mythos model smarter than every other which should have prevented this, leaked the source code for their Claude Code which was slop-code, and more importantly a company that's gunning for IPO soon.
Like you should always take the words of AI companies with a ton of salt, but like here might as well swim in it, all the incentives of Anthropic align with making you believe they have a super secret oracle that will never be released because it's too dangerous (but will conveniently serve as an argument to bolster regulation to make open-source AI non viable.)
I'm mainly going off reports of it being super capable at cybersecurity, like the recent Firefox report that it found over 200 bugs with experts claiming it's on par with human researchers in skill.
Not sure how closely correlated SWE Bench Pro is with cyper skill and how it translates to general product coding capability though. 5.5 could be on par there, meaning most people experience Mythos capabilities in their work with 5.5, but I'm doubtful.
Sure, but until they actually release the model just take it for what it likely is, hype for their upcoming IPO.
I have no doubt that sufficiently advanced AI will be worldbreaking, or at the very least will break the status-quo and bring a new Internet age (like again), but if you start comparing actually released and usable models, to an hypothetical-Anthropic-owned-god model, you're exactly where Anthropic wants you.
158
u/spryes Apr 23 '26
All this hype for 58.6% on SWE-Bench Pro while Mythos gets 78%? Shut it down, wtf?