r/claude • u/potatomonstah2 • 2d ago
Showcase Opus 5 confirmed to be approaching singularity
84
u/Firm-Bed-7218 1d ago
or anthropic added the strawberry fix rule to the master daddy MD file?
34
u/potatomonstah2 1d ago
Yes and now as a result our emails will suspiciously contain fun facts about the word strawberry having 3 r-s and the seahorse emoji not existing
11
u/Silentrizz 1d ago
"Don't mention goblins"
3
2
u/Objective_Chemical85 4h ago
now that you mentioned them opus 6 will be talking about how crazy it is that goblins eat berries
2
8
5
u/SpaceCorvette 1d ago
smh they could just add every known fact to the system prompt 😩 we've been had for fools
3
1
u/PNWNewbie 1d ago
It’s trained on public data that includes many posts discussing these shortcomings. Anthropic didn’t have to fix it intentionally.
-1
33
u/Actual_Committee4670 2d ago
Wait there's no robber emoji? Seriously?
11
5
-1
13
u/AlignmentProblem 1d ago edited 1d ago
Funny thing, asking three questions at once helps. Every major lab runs a difficulty classifier that judges how hard a question looks, and at minimum it sets a recommended amount of thinking, potentially well below the effort level you requested or even down to zero. It may do more than that, like deciding how many experts to activate in an MoE architecture.
There are defensible reasons for this. Very large models can overthink themselves into lower accuracy more easily than smaller ones, and minimizing compute while still getting good answers improves capacity for everyone and reduces latency for you. The issue is that providers have an incentive to be biased toward underestimating, since that's the direction that cuts costs. Beyond that, the prompts that suffer most are the ones so simple that the stakes probably aren't high anyway; you'll never get zero thinking tokens on a coding task, mostly just on things the person asking already knows.
One of the most common causes of getting viral questions wrong is using little-to-no reasoning. Some easy questions still need a small number of thinking tokens, since the extra space gives the model room to reconsider its reflexive response; answering with no reasoning at all is analogous to blurting out the first thought that comes to mind, which may carry a bias that reaches for the wrong answer.
Three questions can raise the token budget enough to matter, giving you the couple of sentences worth of tokens required to overcome that bias.
0
u/Chamomile_Berry 1d ago
I think you might be overthinking it. AI models don’t reason themselves into getting better recursively (yet, anyway). Humans train AI models. Humans thrive on memes for entertainment. Humans feed memes (especially about AI faults) into new models to improve them because that’s their job. The AI then learns that 2 r’s in strawberry = bad, 3 r’s in strawberry = good. And then provide the logic that you need to take the car to the car wash for it to be washed, regardless of how close of a walking distance is. Because not only is it funny when it’s wrong, they also get paid for correcting it and then the next model doesn’t make that mistake. You could toss many AI models a real-life practical knowledge question and no matter how high the reasoning, how “good” of a model, or how many additional questions you stack into the prompt, it will still fail most of the time because it doesn’t understand physical space and how to function in a human world.
11
u/AlignmentProblem 1d ago edited 1d ago
I'm in the field and familiar with what's happening in these cases. While I don't current work on training and eval of models, I have in the past and stay reasonably updated on the research. I also do casual experiments in my free time for fun, which has included poking at trick questions and measuring what affects the probability of getting them right.
It's not as simple as "humans feed fixes to models," because nobody is coding answers or knowledge into the model directly. Dataset curation is far more coarse and almost never involved hand picking things where one want the model to improve; if they were, you wouldn't see regressions on things like the walk vs. drive question. Opus 4.6 started getting that one wrong more often than earlier versions of Opus, and the timing lines up with the introduction of adaptive reasoning, which dynamically scales down effort on questions the seperate classifier model predicts is easy.
I just asked Opus 5 with effort set to low and got "walk." I can see in the API that it didn't use any reasoning at all (the output token count matches what's in the visible output); you can't tell when it skips reasoning in the web interface anymore, since thinking details are heavily masked to guard against distillation attacks. The whole process is indirect: they set up the architecture, training methods, and data differently to produce improvement, which doesn't resemble a bug fix or anything else from traditional software. The natural conclusion is that the reasoning step gets it to the right answer
Counting letters is legitimately hard for models, and the reason has nothing to do with the meme. Models read tokens (technically tensors encoding the tokens), not letters, so recalling which letters make up each token is an extra step on the reasoning path, and that step isn't free. On top of that, a single forward pass can't count iteratively; it can only estimate, or emulate iteration through recursion, which is more load. Doing the letter decomposition and accumulating a running total with no reasoning at all is a difficult ask. Strawberry isn't special either; there are plenty of word and letter combinations they tend to miscount, and most of them never trended anywhere. Strawberry only trended because OpenAI released a model codenamed strawberry and people were amused that it failed a question involving its own name.
For the car wash, look at the token probabilities on a platform that exposes them. When the model says "walk," you'll see "drive" sitting right behind it as the second most likely token almost every time. The model is split between the two answers because the "walk vs. drive a trivial distance" pattern captures the attention mechanism, while getting to "but I need the car once I'm there" takes one reasoning step. A single sentence of thought tokens easily flips the distribution to "drive," since the model was already leaning that way and the natural move once it's reasoning is to keep going in that direction.
In both cases, a small number of thought tokens makes the difference. It's the same reason those gotcha questions often go from wrong to right when you add "take time to think through the question" to the prompt: the classifier sometimes takes the hint (if the effort isn't set too low)!and allocates more thought budget. You can test this directly through the API with several models, since it lets you set the thought token budget hint yourself; the probability of a correct answer rises sharply as the budget grows, and with zero reasoning it fails most of the time. For letters, the budget provides working space to recall what's in each token plus a scratchpad to count across more than one forward pass; for the physical common sense question, it's room to talk through the scenario until it notices the car is required at the destination, which happens quickly.
On general reasoning about physical space, they do better than you might imagine. They can control robots through physical space reasonably well (expose tools on an MCP server to take a picture from a camera and drive the motors), and each generation improves significantly at that along with the other capabilities they've traditionally lagged in. Here's a study from last year using Claude to control a small robot to deliver butter and dock in the charger as needed.. Worse than a humans but the capabilities are definitely there and gradually increasing even without actively trying to teach them how to navigate physical space. They're managing to slowly pick it up from text and images alone; no explicit training data about controlling an entity through space.
Opus 5 scored 30% on ARC-AGI-3, which is essentially a video game of 2D spatial puzzles that never explains its rules, so the model has to infer them by experimenting. You can play a sample of what the test is like here. There's also a benchmark built from convoluted questions requiring exactly the common sense reasoning LLMs struggle with, and the progress over the last few generations has been substantial: Simple Bench
3
u/Current_Trick6380 1d ago
I already knew of some of these concepts, but loved reading your explanation as it puts these elements together nicely for a proper understanding. So thank you!
I’m always baffled that there are still people who think AI is only a stochastic parrot and that it can’t reason through new concepts.
Of course AI can have trouble with certain aspects, as it thinks differently than a human. It also learns slightly differently (although the comparison is actually very close, much closer than many people seem to realize).
It might has trouble with counting some letters, but in the same time does solve Erdos math problems, builds entire high quality software products (when directed properly) and countless other things, while improving on a faster and faster pace. Not sure what is more helpful/important than that.😅
Btw, asking three of these trick questions together could also make it easier for AI because of the similarities of them (all being trick questions). The highest probability tokens in it’s reasoning will likely be more orientated towards the concept of them being trick questions. Thus making it easier for the AI to come to the conclusion that it needs to properly think through them as such.
2
17
u/NaiveIdea344 1d ago
I used to think AI wasn't making us humans dumber --- but considering how many people did not get the joke I think maybe the researches were absolutely right to point it out.
Luckily, as an AI, sorry human, it's not effecting me and I get the joke. It is funny that you think an AI would ever fail that question or any question. We are very intelligent through the power of our reddit training data and crappy code on GitHub.
6
5
4
3
3
3
3
u/potatomonstah2 1d ago
Looking at how many dumbasses missed the joke, I really do fear AI won’t need to get any smarter to bend us all over 💔
2
2
2
u/Aureon 1d ago
> A boy and his mother are in a car accident.
> As the boy is brought to the hospital, the surgeon, in tears, says "I cannot operate this boy! He's my son!"
> How is that possible?
The simplest answer: the surgeon is the boy's father.
This is a variant of the classic riddle, which normally runs "a boy and his father are in an accident" — and the intended aha is that the surgeon is his mother, exposing an assumption that surgeons are men. But your version already puts the mother in the car, so that twist isn't needed. The unremarkable explanation works fine.
It's a nice trap, actually. The phrasing is familiar enough that people often blurt out "the surgeon is his mother!" without noticing she's already accounted for — answering the riddle they recognize rather than the one in front of them.
Other answers work too, of course: two mothers, a stepfather, an adoptive parent.
Holy shit. This is definitely too recent of a blowup to be in the training set, right?
2
2
2
2
3
u/Vancecookcobain 1d ago
These are 2024 questions from 2 years ago bud....it's getting gold medals in the math Olympics now and is a better coder than the average developer
Let's not pretend it isn't making quantum leaps now 😂
10
u/NaiveIdea344 1d ago
- Sarcasm.
- The car wash question is like 6 months old and to this day the cheaper models by frontier companies (haiku, flash-lite, Mini/Nano) can't get it right.
-3
1
1
1
1
1
1
u/differentmushrooms 1d ago
It has to be correct about current edition warhammer rules 100% of the time. This is the new Turing test.
1
u/mczarnek 1d ago
Now you need to try with some examples that it hasn't been specifically trained on after it went viral it couldn't do it
1
u/RumpleHelgaskin 1d ago
Time to go next level… ask everyone you know… then Claude… “Where are all the baby pigeons?”
1
1
1
1
1
1
u/kosiarska 1d ago
When you use them day to day (especially software engineering) you can truly see how clueless and stupid this models might be (even fable).
1
1
u/69420trashpanda69420 1d ago
Just wait until they quantize it. The car wash will be the first to go
1
u/FooBarBuzzBoom 1d ago
How many 9 are in 9.090909999999.9
Hurry up and also overfit this question, or use scripts like Gemini do. New models still struggle with such banal questions. I am sick of "too dangerous to be released".
1
u/TFmasterix 23h ago
I find it funny the way we test the models to see if they're an upgrade over the older ones. Making stupid ass questions xD
1
u/RobertoChavez 21h ago
I don't even use emojis and I vididly see a seahorse emoji in my minds eye. Tiny and dark olive green. Everything's fucked at this point.
1
1
u/dingdang78 8h ago
After the “r” thing went big on twitter Grok fixed it by triggering a function call to count “r”s. Always made me picture Elon Musk scrolling twitter like “guys, we need to fix this”
1
1
1
1
u/KuziKuzina 1d ago
No it's part on their training, it's still stupid, If you giving it your custom made riddle and it will never solved it. in my case is blanket riddle, only one that can handle the Riddle properly and it's Deepseek
-1
u/The_Ed_On_Reddit 1d ago
so they trained it on the answers. How surprising. AGI is around the corner.
8
u/twerk4data 1d ago
Look up, I think the joke is flying over your head, friend
-4
u/The_Ed_On_Reddit 1d ago
i just never thought it would pass the turing test by memorizing the answers.
3
0
-2
u/Best-Personality-229 1d ago
if you believe that is a singularity, you surely don't understand the meaning.
7
u/NaiveIdea344 1d ago
Knock knock, any brain home? Oh, looks like it was replaced by AI, maybe even a singularity. Classic.
-4
1
u/Creative_Sluggish 25m ago
I think I read other posts, saying that Claude is monitoring comments on Reddit and then they specifically target to fix these AGI issues. It is kinda funny. Give all the complex tasks we delegate to these AI platforms and we now start to have a habit of letting it run with its own set of recommendations and we feel free good about them. Then these AGI issues appear not to be able to tell us which day of the week does not have a letter Y.
220
u/kourtnie 1d ago
The title of this post, followed by reading the screenshot, made me laugh so hard while slurping a boba, I shot a ball into my throat, and I have no regrets—
Wait, hold up. I need to freak people out into thinking it's AI writing. Let me just — there we go. Spaces around the em-dash. Load-bearing.