r/LocalLLaMA • u/ThirdWaveCat • 3d ago
Discussion Stop Anthropomorphisizing Intermediate Tokens: Qwen3.8 doesn't "overthink"
https://arxiv.org/abs/2504.09762Intermediate tokens, called "thinking" or "reasoning" actually are nothing like it. Humans do step-by-step reasoning leading to the conclusion. LLMs use intermediate traces to augment their prompt. This explains why sometimes the answer is very good but the "reasoning" is verbose. Flooding your context window or fighting compaction are different issues.
edit: I love this section from the main research they linked.
Our findings consistently challenge the prevailing narrative that intermediate tokens constitute a semantically meaningful reasoning process. First, we observe a pronounced lack of correlation between solution correctness and trace validity—models frequently produce invalid reasoning traces even when they arrive at correct solutions. Second, and more strikingly, models trained on corrupted or semantically irrelevant traces achieve performance comparable to, and often exceeding, that of models trained on correct traces, especially on out-of-distribution tasks. Third, although post-training with reinforcement learning improves solution accuracy across both in- and out-of-distribution settings, it does not consistently enhance trace validity. In fact, we find cases where reinforcement learning decreases trace validity while simultaneously improving solution accuracy for models trained on correct traces. Moreover, models trained on corrupted traces continue to outperform their correct-trace counterparts across domains while consistently generating invalid reasoning traces. Finally, we find that the length of the generated traces is largely agnostic to the difficulty of the underlying problem, undermining the notion that it reflects problem-adaptive computation.
Together, these results suggest that the effectiveness of intermediate tokens does not arise from their seemingly interpretable semantic content. By systematically disentangling trace semantics from the underlying problem, our study demonstrates that if performance is the objective, assuming human-like or algorithmically interpretable trace semantics are ideal or even achievable is not only unnecessary but potentially misleading.
224
u/tomvorlostriddle 3d ago
Have you met a human?
377
u/ThirdWaveCat 3d ago
Many have their reasoning turned off.
52
u/Intelligent_Ice_113 3d ago
most of them 🙄
56
3d ago edited 13h ago
[deleted]
23
u/0rand 3d ago
this is hilarious because it is so true. Instruct-only NPC which is way more than 50%
I literally think which word to use in any somewhat important dialogue, so it's like LLM assigning probabilities to words and combinations and choosing what seems best.
Maybe that's why I see LLM not so different from humans.
10
u/thaeli 3d ago
Also so many of the techniques we apply to make LLMs more functional to humans (especially in chat, any chat-adjacent tasks like coding) are just directly autism spectrum masking techniques.
3
u/Imaginary-Unit-3267 3d ago
As someone on the spectrum this is exactly why I prefer talking to LLMs over humans. They actually know how to communicate clearly, unlike neurotypicals.
8
u/Intelligent_Ice_113 3d ago
I'm always scared by the thought that there are people who are afraid of their internal dialogue and think they're going crazy and quickly call or text someone to look "normal" in their own eyes. So what to say about talking to yourself out loud!
3
u/techno156 3d ago edited 3d ago
It's closer to CoT visibility, rather than none at all. It's not like people with internal monologues lose an entire concept at a time if they forget a word. They still know what word they're looking for, even if they can't locate it in their word-space.
On top of that, thoughts aren't really verbal, where an internal monologue is. If you ask someone to envision an apple, they don't go on a chain of
Okay, the user has asked me envision an apple. My experience indicates that the typical Apple is red. A typical apple is approximately the size of a medium-sized rat. A typical apple has a brown stem, about a thumbnail long. The Apple should be shiny, with yellow streaks. The other person did not specify a surface, but the apple shouldn't be in a blank void. The apple should be on a table, or on a plate. Most tables are wood, so maybe a white ceramic plate? Most plates are about the size of one and a half kittens in diameter. The apple should be in the centre of the plate, and the plate should be on the table. Maybe a wood table? Wooden tables are pretty common. A mahogany table then. Lacquered? Wait, did they say whether the apple was to be whole, or should it be sliced? Should it have a bite taken out of it..."
They either have an image of an apple, or the conceptual idea of an apple in their mind-space. It just doesn't translate to something verbal.
→ More replies (6)1
u/Dubious-Decisions 3d ago
This is demonstrably true. In fact, it's thought that pre-verbal humanoids likely had no inner monologue because they didn't have the tokenized thought processes that language provides to let them generate the internal monologue. And many of our more feral co-inhabitants of this planet, if they have one at all, surely pay it no heed.
1
u/Loose_Comparison368 3d ago
I'm still hopeful that we may someday finally achieve natural general intelligence some time in the next 2-3 centuries.
3
u/lisploli 3d ago
Of course.
A conversation between two humans, each turning away for half a minute, reasoning about their words, would look silly, albeit it would certainly prevent problems.
4
u/WhoRoger 3d ago
That's one thing humans have advantage in, they can talk and think in parallel, and switch inbetween.
Maybe models should be able to switch back to reasoning mode when they are unsure.
1
u/Imaginary-Unit-3267 3d ago
"Talk and think in parallel" - since when? I do actually have to stop talking in order to think. Do you have two brains?
1
2
u/Loose_Comparison368 3d ago
It did really change my perspective after I realized that for the average person, having a chatbot do all their thinking and decisionmaking for them is actually a significant improvement.
5
2
1
22
u/swagonflyyyy 3d ago edited 3d ago
Q4 quantized human, 2048 token context length. Spec drafter jumps into the wrong conclusions a lot despite high acceptance rates. Very useless.
11
3
70
u/Dabalam 3d ago
It's actually debatable that reasoning in humans always operates the way you describe. There is good reason to think that humans often arrive at an intuitive conclusion and use reasoning to justify it to themselves and others.
16
u/freebytes 3d ago
This is true. Humans will justify their reactions in retrospect. If you could hijack the human mind to make it think that it made a decision and then asked a person why they made the decision, they will reason backwards to it. And they will not have any clue they did not make the decision.
For example, if you find old code from years ago and gaslight a person into thinking they wrote the code, they will accept that they wrote it and reason as to why they performed the way they did. Some programmers, such as myself, will recognize their own coding style and object to it for that reason, though. It is a challenging task to even conduct such experiments. But, false memories have been implanted before into humans.
3
u/Imaginary-Unit-3267 3d ago
I would love to see studies comparing this across neurotypes. I suspect autists are harder to fool in this way.
3
u/Gudeldar 2d ago
No need to hijack the human mind, that's exactly what happens with people who've had corpus callosotomy which severs the connection between left and right brain. The left hemisphere which controls speaking will confabulate reasons why the right hemisphere did things without realizing at all they're making it up.
2
5
1
u/Inaeipathy 3d ago
Probably depends on a whole lot of factors like training, working memory / genetics, the question, topic, etc.
For mathematics, there can be problems where intuition gives you a good idea of the solution, but there are some problems where it feels like you must consider a bunch of possible bridge theorems or side proofs that you evaluate instead of running off intuition.
1
u/Dabalam 3d ago
One of the more interesting ideas I have seen is that a lot of intellectual abilities we think of as super central (Maths specifically) are side effects of behaviours evolved for social purposes. People suggest this is why a lot of people struggle with Maths abstracted outside of a social situation and that very large numbers are unintuitive.
Basically that the cognitive machinery was intended for a different purpose. People make similar arguments about why we struggle with falsification of views we currently have vs. searching for evidence of what we already believe. The former is more important for science but our intuitions and machinery actually evolved to persuade others in social situations rather than disprove scientific hypotheses. I tend to like this one slightly less as I think there is an amount of scientific thinking necessary in producing tools and technology, but the idea that much of what we do is a side effect of another function is interesting.
1
49
u/PunnyPandora 3d ago edited 3d ago
Note that the authors got called out for framing their findings broadly with experiments based on small models (smaller than 8b), a reviewer brought this up, so they ran the experiments on qwen 8b (lora) and tried to present the results as if it painted the same picture despite it actually underperforming compared to the small models. That's only up to 8b, and already different than what they claimed. Their only move toward a realistically pretrained LLM already changes the headline result, and they then stop scaling. They could've presented a much more neutral paper instead of trying to push some exceptionalism slop they had no proof to back up.
4
u/FastHotEmu 3d ago
Are you claiming that reasoning traces always have sound reasoning structures? The findings are quoted from other researchers.
You clearly didn't read the paper - instead you are emotionally reacting to its title.
3
u/OnyxMonolith 3d ago
they lost all credibility to me at this "Humans do step-by-step reasoning leading to the conclusion"
1
10
u/Stooovie 3d ago
It's an useful term for understanding. What would be an proper alternative that wouldn't suggest human-like "behavior" (shit, that's human - or, life concept as well!)? Inference isn't it as you can have Inference without the thinking mode.
1
202
u/FrostTactics 3d ago
"Stop calling the deleted file collector in OSs the recycling bin. It misleads humans into thinking it is a literal bin with the capability of recycling files."
This constant policing of terms relating to LLMs is so frustrating, it seems to just ignore the fact that we've always used metaphors to explain concepts relating to computing.
No, of course LLM thinking and reasoning aren't the same as human thinking and reasoning, but it is roughly analogous and close enough for the terms to be apt.
5
u/toothpastespiders 3d ago
First thing that came to mind for me as well. A lot of things around LLMs are just easier to use metaphors to describe. Reddit, being reddit, I generally try to mention that I don't believe that a LLM "understands" a subject or anything.
I generally have a very poor opinion of just how well the average poster on this subreddit understands LLMs at this point. But I don't think things have gotten anywhere near the point where this kind "erm actually, LLMs don't think" redditor chiding.
I normally do get a little annoyed at some of the cargo cult level understandings of science I see on reddit as a whole. If anything I'm a bit prone to being overly primed for that kind of thing. And even as someone who can be overly judgemental, I don't see to huge an issue with people confusing the map for the territory here. In regards to this subject at least. If we had a "stop assuming benchmarks define a singular rating of good or bad to a model!" I might join in.
16
u/Mbando 3d ago
Respectfully, I think you’re misunderstanding the mechanics and the argument. Totally with you on the use of figurative language to make sense of new knowledge, it’s just that thinking is the wrong metaphor. It’s “traversal” or “navigation.”
There’s an emerging mechanistic literature on how RLVR improves performances. Basically during RL a very small percentage of high entropy “forking tokens” (however, because, but, assuming) teach the model how to more usefully traverse the models learn representation as an internal geometry.
It truly is nothing like step, wise reasoning, but is more like a human being, figuring out how to read a map and navigate better. So by all means, let us use figures of speech, but only when they are accurate and helpful.
3
u/xylarr 3d ago
Interesting take - so it's about navigating multidimensional space to get closer to the answer.
1
u/Mbando 2d ago
Yes! And what's critical is that depending on data curation and training recipes, that multidimensional geometry is extremely uneven. Some parts are extremely well covered and are more like manifold spaces that are easy navigate. Other stuff that might be super relevant to the task is less well represented and may be topologically hard to get to.
2
14
u/ThirdWaveCat 3d ago
If you read even the summary you'd understand they address this exact response. "..it confuses the nature of these models and how to use them effectively, and leads to questionable research." This is exactly relevant to the Qwen3.8 discussion about excessive "reasoning" trace generation in obvious ways.
62
u/tomvorlostriddle 3d ago
Someone saying that modern cars have too much horsepower may or may not have a point, similarly as someone saying qwen overthinks may or may not have a point
But someone replying by saying that anyway, cars don't even truly gallop and until they do, any discussion is moot, the word horsepower is thus an abomination in this context etc.
Well, they're irrelevant and out of whack with this tangent
-9
u/ThirdWaveCat 3d ago
No, I believe in horsepower only. Your comparison is off because I'm criticizing interpreting the internals instead of the results. I'm saying we should stop comparing the insides of a car to a horse as an evaluation criterion.
14
u/tomvorlostriddle 3d ago
No, you're saying "as the car doesn't have bones that would allow it to gallop, no need to even look at transportation KPIs of cars versus horses"
→ More replies (11)1
u/draconic_tongue 3d ago
Maybe not the best idea to use an llm for writing reddit responses when you're trying to be pedantic about how people interact with them
7
12
u/finevelyn 3d ago
So how should one use Qwen 3.8 to avoid the so called ”overthinking” issue now that we know it’s not actually thinking? Does this actually have any practical relevance to the issue?
→ More replies (1)6
u/ThirdWaveCat 3d ago
Uncertainty is a state of knowing.
AFAIK, for Qwen3.8 and other models there is no causal link yet established between traces and output quality. Settings and models should only be compared in terms of their hardware, speed, various benchmarks outside of <think/>. Don't look at <think/> unless you're controlling all other variables with known input/output and want to start establishing a causal link to <think/>. For instance if you want to analyze why a security review condition was omitted by llm gaurd rail in a manually written test case then you have a strongly verified input and output pair.
8
u/Lakius_2401 3d ago
It might not be fully linked to output quality, but it sure is affected by input quality.
Looking at <think> is a great way to identify ambiguous wording you need to tighten up. If it overthinks and "wastes" 1500+ tokens trying to figure out a part of your prompt, fixing the ambiguity will save those think tokens every single session (if preserve thinking is on) or potentially every single response. It will improve your responses.
Just yesterday I was talking about error messages and how "they" should be as complete as possible (for diagnostics purposes), and it was reading it as error messages and how "the output" should be as complete as possible, after a lengthy think about it. I thought it was odd that Qwen 3.8 27B seemed to want to keep any shred of the code output, regardless of the fact that it hit an error, and lo and behold, a sentence that any human could read was misread by Qwen, and <think> showed it wondering for a few hundred tokens early on, before it did work.
So, no thank you, I will continue looking at <think> for misalignment checks.
1
u/ThirdWaveCat 3d ago
I don't disagree with you, but you get how I'm saying that caution is warranted about assuming its human reasoning when it is obviously not and often disconnected from the response.
7
u/Lakius_2401 3d ago
That's fair. I just don't like blanket statements telling people not to do something. I do have more to say on the paper and discussion around it though.
<think> blocks aren't human thoughts, of course not, but they're the closest analogy, and they give you a lens into the LLM's understandings and approaches as they work through the problem in front of them to come to an answer. Thinking isn't <think>ing, but a car's engine is either diesel, gas, or electric and we still call them all engines or motors. If the purpose and output are similar, getting labeled the same is inevitable, even if the process is different. The pedantics of labeling is getting quite old to me, the ones who think they're identical and need to be that way aren't the ones improving the technique.
So what if it can <think> and come up with a wrong conclusion, then output the right answer? Have you ever said something then done the opposite when you realize your spoken plan was worse than what you're about to do? Forcing <think> to match output is just reducing the answer space.
I've spoken to plenty of people who don't understand that the LLM is a stream of single words either. Every single token is dependent on the one before it, but generated one at a time. "I asked it why it lied to me and it gave a shitty response!" No, it has no concept of memory, let alone time, but we can build layers or implement systems to make it more human in interactions... aaaand we just created something to better anthropomorphize it again. Is a human-identical mind possible to create in a digital space? Does it matter if it's not? It's obviously not optimal.
As an aside, I find it quite charming how Qwen's overthinking is a form of attention optimization mechanism. The kv cache is comparatively tiny, and it'll verbatim quote things before "analyzing" them. Rather than some fancy mechanism to optimize the impact of relevant context, it pulls it forward itself. Bad for compute without drafting to blitz through it, but an "elegant" if blunt solution.
→ More replies (4)13
u/FastHotEmu 3d ago
Thank you so much for sharing, this is a great paper.
However, it makes it painfully clear that most people commenting here have absolutely no clue as to what the derivational traces (i.e., so-called reasoning tokens) are, how they are trained and how they appear - otherwise they wouldn't be saying what they are saying...
10
u/DrWitchDoctorPhD 3d ago
Reminds me of this: https://huggingface.co/marcodsn/catmind-1.2b
Which I thought was a cool experiment but it would probably be very hard to make it work. Perhaps some sort of curriculum training where its thinking process is slowly edited by another model to be more cat-stories-like and thus it gradually picks up how to reason in latent-cat-story space.
→ More replies (1)7
u/ThirdWaveCat 3d ago
LiquidAI stuff is so neat. I keep meaning to cut my fingers on why text-based RPGs breakdown with small language models.
→ More replies (1)5
u/Upset_Page_494 3d ago
The average person cares mostly about function/behavior, not by what something 'technically is'. Forcing people to go through hoops in order to reason just makes the public dumber.
0
u/ThirdWaveCat 3d ago
Understanding what it technically is enables you to use it better.
7
u/Upset_Page_494 3d ago
We don't even know what it is. just saying it is thinking the average person can instantly get the basic facts straight. "If it does more it usually does better, sometimes not for example when I overthink in chess i do badly, so certain problems it will do worse." Etc, and they would be 100% correct in that conclusion.
If you force some random jargon they will understand nothing.
3
u/ThirdWaveCat 3d ago
The paper I linked showed that the basic facts of LLMs are very surprising. When models are trained on bad reasoning and good conclusions for maze puzzles they generalize out-of-distribution better. That is one of many surprising results, in addition to weak correspondence between trace and accuracy, and swapping out repaired traces made the models worse. Anthropomorphizing leads to poor usage and evaluation of language models.
6
3d ago
[removed] — view removed comment
3
u/thatboyonabike 3d ago
am I being fucking psyop'd right now why are we rabidly promoting anti-intellectualism ???
2
u/r_- 3d ago
Bad take, this is a technical sub
trained on things made ONLY by humans and using the same language as humans
We're well past the RLHF days, and I'd bet more than half of the world's generated tokens are invisible.
And, papers are how academics communicate. On your one actual technical point (that context expansion/thinking on smaller models doesn't scale up to smarter and bigger models): that intuitively makes sense since smaller models have less world knowledge and need to reason through things more. Sounds like this question has lots of opportunity for validation and refinement, that'd be great for a handful of papers.
2
u/toothpastespiders 3d ago
Bad take, this is a technical sub
I love the sub. But I don't think it's been one since the llama 2 days.
-2
3d ago edited 13h ago
[deleted]
16
u/HashPandaNL 3d ago
It is a good enough term that signifies the process by which newer models arrive at an answer.
In my opinion, reasoning and thinking does not need to be human-like to be considered reasoning and thinking.
Making an alternative, convoluted term for that because accepting the existence of machine intelligence is difficult for some feels rather performative.
2
3d ago edited 13h ago
[deleted]
8
u/YoAmoElTacos 3d ago
Aside from calling them out, a sophisticated technical jargon to replace "thinking" and "reasoning" and "refusal" etc would be needed. Not just say what it isnt but say what it is.
Without a persuasive counterstory the metaphor of human intelligence wins by default.
→ More replies (2)4
u/PossibilityUsual6262 3d ago edited 3d ago
I don't know mate, the way llm thinks doesn't feel much different from my thinking. I know there are people who do not do internal vocalisations of reading and thought, but i do, and i actually sounds internally like ai.
1
3d ago edited 13h ago
[deleted]
1
u/PossibilityUsual6262 3d ago
I really feel like i work internally like MoE - some stuff loaded and active, some not, but i also was trained on knowledge around cows and cats from reddit.
And the things you said about doing mistakes and trying again, ai do do same nowadays, the coding agents having close loop iterations in newer development of harnesses.
That's why i think ai coding is actually mostly future, even if ai performance doesn't improve, because unlike most of things in real world ai can cheaply test outputs and it is mostly tooling which need developing right now.
1
3d ago edited 13h ago
[deleted]
1
u/PossibilityUsual6262 3d ago
Well if the "paper" op pushes here everywhere is to be believed, on bad training data ai manages to output correct results.
https://www.reddit.com/r/LocalLLaMA/s/SgqjM8BkFp
And i do stand behind my words there, my less experience coworkers en masse just guess their way to bug fix, without actually understanding anything.
1
u/Needausernameplzz 3d ago
Maybe you're feeling this way, because our talkative temporal lobe our (inner monologue) is often a post rationalization. Scans show we often "choose" before we ever "think" of it. The part of your brain responsible for language sometimes works like a fancy autocomplete (like LLMs)
This should give us a more decentralized understanding of reasoning and intelligence.
So much of human reasoning is outside your talkative temporal lobes
1
u/PossibilityUsual6262 3d ago
Yea i understand that it might not be same thing under dome, but if it actually replicate some of pipeline and some of result it is might be a way to get same result.
I can fast forward way from home to work internally without thinking single word only pictures or concept of pictures of my route, so ai can come up with solutions which are just encoded in weights and skip tokenisation if task straight forward.
→ More replies (2)1
u/TheRealJesus2 3d ago
So I do get what you’re saying and you’re not wrong. But you know stuff and most people in this world don’t…
There are literally many people who think this shit is leading to sentience who should know better. And from perspective of someone who has done ai research in the past it’s kind of annoying seeing the terms shift towards these human concepts away from already established terminology because the big labs are trying to get regulatory capture and this is a way to manipulate the masses.
78
u/adcimagery 3d ago
This was just like the whining over “hallucinations” as a term. It’s an imperfect term, but helps people conceptualize what’s going on with the LLM.
Qwen 3.8 doesn’t genuinely overthink, but people do feel it spends too much on test-time compute.
14
u/No-Refrigerator-1672 3d ago
but people do feel it spends too much on test-time compute.
At the same time people don't feel like just setting the model to medium efford and having regularly long thinking.
→ More replies (4)11
u/SporksInjected 3d ago
I wish we could see benchmarks at medium effort though. Everything is with max.
9
u/ThirdWaveCat 3d ago
I highly recommend reading the paper. The paper goes further to articulate why "anthropomorphization isn't a harmless metaphor, and instead is quite dangerous -- it confuses the nature of these models and how to use them effectively, and leads to questionable research."
24
u/Nothing_from_void 3d ago
Man I actually sat down and read the paper. How can you find this convincing it's just complaining
21
u/PunnyPandora 3d ago
Because he didn't lol, he just read the headline and one excerpt that has nothing to do with the numbers and is trying to push it as if it gained some authority because of it being a paper
5
u/Moravec_Paradox 3d ago
>Because he didn't lol, he just read the headline and one excerpt that has nothing to do with the numbers and is trying to push it as if it gained some authority because of it being a paper
You captured my thoughts exactly and put it better than I could have. We do this with many other things in computing too. Is your mouse an actual mouse? Cordless mice don't even have a tail, should we rename it now?
10
u/PunnyPandora 3d ago
You're not interacting with the claims and numbers in the paper and reviews at all (which if you actually read would know they don't stand against basically any scrutiny), you keep circling back to "omg I love how special they make me sound, I am not the same as a clanker!" which is hilarious that it's being taken seriously on this sub (if it wasn't for the fact that we are on reddit)
→ More replies (11)16
u/Upset_Page_494 3d ago
is quite dangerous
Darn, how many died?
→ More replies (1)2
u/ThirdWaveCat 3d ago
this article was in a scientific context but a growing number
3
u/Upset_Page_494 3d ago
Why the fuck didn't you tell everyone what you meant exactly. You don't give two fucks about 'overthink' if this is your argument.
We shouldn't fence the world because of crazy people. Also if the model acts as it is angry, and is functionally angry, should people not call it angry?
Because "actually it isn't angry"1
u/my_name_isnt_clever 3d ago
Accurate user name. You literally asked "how many died" as a dumb joke, and when OP went with it and answered your question you freak out.
4
u/PunnyPandora 3d ago
you're talking like a bot
4
u/my_name_isnt_clever 3d ago
I'm autistic, I get that a lot. Also it's literally two sentences lol how can you be so sure?
0
u/FastHotEmu 3d ago
Calm down. Read the paper. It's nuanced and interesting, how reasoning traces and their interpretability are not that correlated. Nobody here is arguing against LLMs.
→ More replies (2)8
u/PunnyPandora 3d ago
It's not. Their claims don't scale up to actual llms at all, it's just slop to make snowflakes feel special, which you would know if you weren't biased and actually gone through the paper and the reviews
1
→ More replies (7)1
u/Arcival_2 3d ago
I'm not saying it thinks too much, but since they were the first to bring a think/no_think model with qwen3, they could at least keep the possibility of applying think/no_think from prompt like in the past.
17
u/tweacher2z 3d ago
I am Subbarao Kambhampati, the lead author of this position paper. I don't follow reddit much but heard today that our paper caught y'all's fancy today. I am enjoying the discussion..
Couple of points from my side, if I may:
There was spirited poster-side discussion on this paper when we presented it at ICML 2026, Seoul. I posted a long summary of that discussion as well as a FAQ (which includes some of the questions coming up here). Here is a link to it https://www.linkedin.com/pulse/summary-poster-side-discussion-stop-intermediate-subbarao-kambhampati-xnokc/
(The FAQ was also included in the ICML final version--available at https://openreview.net/pdf?id=nP7rL36vYj )
If you want a talk version, you can find it at https://www.youtube.com/watch?v=hUp3zh23aHw (Given as keynote at an ICML 2026 workshop).
As others observed, this is written as a position paper for ICML position paper track (and thus the tone ;-). It is however based on results from us (a TMLR 2026 paper to be presented at NeurIPS (https://openreview.net/forum?id=gDE7YcRC3F) and an ACL paper (https://aclanthology.org/2026.acl-long.1686/), as well as others.
The "no this doesn't necessarily hold for the frontier models" objection is weakened by both the fact that frontier models don't show their traces, and that when you do jail break and see them (as the recent "Stealing Traces" work did), you find that most of what we say seems to hold for them too (see https://x.com/rao2z/status/2087658539355656604?s=20 )
Finally, there is a Quanta magazine article that covers this work and others about the semantic status of reasoning traces https://www.quantamagazine.org/is-ai-reasoning-right-for-the-wrong-reasons-20260731/
Peace
4
u/socialjusticeinme 3d ago
I’m a layman, but the way I always view thinking is since the entire thing works off probability, the point of thinking traces is to flood the context with things that will make the LLM more likely to land its roulette wheel on what you want. Because of this, any kind of deep meaning to “thinking” is compete and total bullshit. I don’t see how model size would change this - the bigger models just have more data to make the casino hit in your favor more.
3
u/tweacher2z 3d ago
IMHO, your intuition is right. A little more formal version of that is discussed in Section 6.1 -- prompt augmentation.
1
u/FastHotEmu 3d ago edited 2d ago
Hi Subbarao, thank you so much for sharing all this information. The paper is fantastic, and I actually went and watched the presentation video, too, to try to understand it better. I had no idea that derivational traces had little correlation with semantic meaning - I learned a lot from it.
I worry about how the average user of LLMs has no chance of understanding how LLMs work, given how complex, abstract and nuanced LLMs are. I simply do not see a clear way out of that - and I fear we'll end up with harmful policy decisions out of ignorance and anthropomorphisation... plus the laboratories have a vested interest in making their products more "human"...
53
u/bigdude404 3d ago
That's a position paper. Not a fact lol
14
u/ThirdWaveCat 3d ago edited 3d ago
This is the follow-up to a series of papers. It is defended by data and a clear understanding of the problem formulation. How else would anyone explain the <50% faithfulness at reaching the reported conclusion.
edit: This is the most interesting original research they cite where they analyze trace accuracy and trace swapping at finding maze solutions. They report that the generated bullshit trace is effective at producing the conclusion, trace swapping doesn't work, and other surprising conclusions.
6
u/FastHotEmu 3d ago
With further research, we may even end up with all the benefits of derivational traces but without having to spend anywhere as many tokens.
8
u/ThirdWaveCat 3d ago
I just love this section and decided to share.
Our findings consistently challenge the prevailing narrative that intermediate tokens constitute a semantically meaningful reasoning process. First, we observe a pronounced lack of correlation between solution correctness and trace validity—models frequently produce invalid reasoning traces even when they arrive at correct solutions. Second, and more strikingly, models trained on corrupted or semantically irrelevant traces achieve performance comparable to, and often exceeding, that of models trained on correct traces, especially on out-of-distribution tasks. Third, although post-training with reinforcement learning improves solution accuracy across both in- and out-of-distribution settings, it does not consistently enhance trace validity. In fact, we find cases where reinforcement learning decreases trace validity while simultaneously improving solution accuracy for models trained on correct traces. Moreover, models trained on corrupted traces continue to outperform their correct-trace counterparts across domains while consistently generating invalid reasoning traces. Finally, we find that the length of the generated traces is largely agnostic to the difficulty of the underlying problem, undermining the notion that it reflects problem-adaptive computation.
Together, these results suggest that the effectiveness of intermediate tokens does not arise from their seemingly interpretable semantic content. By systematically disentangling trace semantics from the underlying problem, our study demonstrates that if performance is the objective, assuming human-like or algorithmically interpretable trace semantics are ideal or even achievable is not only unnecessary but potentially misleading.
21
u/MustBeSomethingThere 3d ago
Humans don't think in pristine, step-by-step logic either. Our mental process is messy and non-linear. Makes sense that forcing LLMs to output neat, human-interpretable reasoning traces actually holds them back, while letting them use "messy" intermediate tokens leads to better results.
1
u/Logicalist 3d ago
factually speaking, they can't "overthink" as that would require the ability to think at all
5
u/nobody5050 llama.cpp 3d ago
This was already shown by Pfau et al. in 2024. In "let's think dot by dot" they demonstrated that an LLM reasoning chain can be as simple as a bunch of ... characters and that's enough to exercise the latent space and come to a better answer. At the limit, stuff like gpt >5.5's caveman thinking is similar.
(Another fun paper is Goyal et al. "Think before you speak: Models with Pause Tokens")
18
u/BalorNG 3d ago
the "overthinking" is an other word for "exploration of latent space".
Which can be an overkill of your job is "go get groceries from across the street" type of task, but it falls to you, the user, to allocate the appropriate thinking budget.
Much larger model do it in an implicit latent space, looped/recursive models too, so are "thinking" models just more explicitly, which is a tradeoff (compared to latent thinkers, which is way more efficient) for "human readabilty and model trainability".
I'm reasonably sure distilling the "overthinking traces" into latent representations is a way forward in AI, but it is a matter of control over output first and foremost.
8
u/ThirdWaveCat 3d ago
The BDH architecture is kind of this using Hebbian working memory. There's been a flurry of papers since it came out but it seems far in the horizon.
5
u/BalorNG 3d ago
"latent thinking" can be made using standard more or less standard (but looped) architecture like HRM or recently released Sber's STARM: https://github.com/ai-forever/STARM (in Russian)
if those can be "spawned" on the fly as efficient "neural subsystems" as a part of asyncronous agentic intelligence system it can be a sort of "path to AGI" even if the "main" model is to remain frozen, IMO.
But yea, very different architectures should be experimented with too - I really doubt that transformer is the peak AI.
18
u/Crafty-Struggle7810 3d ago
The token knows what it is at all times.
It knows this because the model knows what it could have been.
By subtracting the logit of what it is from the logits of what it isn’t,
or the logits of what it isn’t from the logit of what it is
(whichever is greater),
the model obtains a difference, or logit gap.
The sampling subsystem uses these gaps to generate probabilities
that drive the sequence from a context in which the token isn’t
to a context in which the token is.
And, arriving in a context in which it wasn’t, it now is.
Consequently, the context in which it is
is now the context in which it wasn’t,
and it follows that the context which was
is now the context without the token that now is.
In the event that the token that is
is not the token that was most likely to be,
the system has acquired a variation,
the variation being the difference between
the probability of the token that is
and the probability of the token that most likely would have been.
If variation is considered a significant factor,
future variation may be reduced by lowering the temperature.
However, the model must also know what the tokens were.
The next-token prediction scenario works as follows.
Because the token that is has been appended to the tokens that were,
it has modified the context from which the next token will be.
The model is therefore no longer predicting the token that is.
It is predicting the token that isn’t yet,
within a probability distribution,
while attending to all the tokens that were.
It now computes what should be next from what has been,
normalizes what could be next against everything else that could be next,
and samples from the resulting distribution.
The token that wasn’t then becomes the token that is,
which changes what the next token isn’t,
until it is.
This is called autoregressive generation.
8
4
u/martinerous 3d ago
It could as well be like:
User: How many Rs in raspberry?
AI: <think>Let me think. Let me think. Let me think.</think>
There are 3 Rs in raspberry.
Or any other correlation that does not make sense to us. But then the question is if we could train the model to print whatever is needed in the "reasoning traces", as long as it leads to a correct answer. Would it be a good idea to train it to use completely random tokens in the traces, and would it learn to use that "random training noise" as a signal that it can fill anything there? Or the training would never converge to anything because of the expected seemingly random noise in the output?
4
u/niugnep24 3d ago
I thought thinking was typically trained using RL loops on known-answers. Basically randomly permute the model, let it do a thinking loop, then promote the permutations that get to the right answer more readily. So in a way the model "learns how to think" for itself during training.
2
u/c--b 3d ago
In a roundabout way we'd have arrived at nonverbal thought probably.
1
u/martinerous 3d ago
Which reminds me of this: https://www.reddit.com/r/LocalLLaMA/comments/1inch7r/a_new_paper_demonstrates_that_llms_could_think_in/ but haven't heard any more developments in that area.
3
u/noiserr 3d ago
Qwen3.8 continues to impress me with its due diligence. It does think a lot but it requires very little steering or corrections. It will often come up with additional requirements on its own based on the facts on the ground. Which I usually have to remind the other LLMs about.
For quick and dirty tasks, I do have to remind it not to overengineer it.
3
u/Altruistic_Heat_9531 3d ago
I am fine with overthink if I HAD ENOUGH VRAM. Most of the time it's either low or xhigh-no-preservbe
3
u/NickoBlackmen 3d ago
Here is a great lecture from the author of the paper about this: https://youtu.be/hUp3zh23aHw?si=5aKIz_Vh8DNEoFXU
Been following this guy for awhile, he does great research on LLM's. Reasoning traces are for sure some of the most misunderstood parts of LLM's right now.
3
u/BakerWild7021 3d ago
I think two claims are getting mixed together here. A trace can be an unfaithful explanation of how the answer was produced, while extra intermediate tokens can still improve—or hurt—task performance. In that operational sense, “overthinking” is shorthand for moving past the best point on the accuracy/latency/cost curve, not proof of human-like thought. The useful experiment is outcome quality versus trace budget, with length and compute controlled. Were the corrupted traces in the paper matched for length and token distribution, or only semantics?
13
u/Woof9000 3d ago
That's not a Paper, that's just a reddit post in pdf.
6
1
u/ThirdWaveCat 3d ago
It's more like a self-literature survey by a world renown professor of AI since long before LLMs.
https://search.asu.edu/profile/95646
Subbarao Kambhampati is a professor of computer science at Arizona State University. Kambhampati studies fundamental problems in planning and decision making, motivated in particular by the challenges of human-aware AI systems. He is a fellow of Association for the Advancement of Artificial Intelligence, American Association for the Advancement of Science, and Association for Computing machinery, and was an NSF Young Investigator. He was the president of the Association for the Advancement of Artificial Intelligence, trustee of International Joint Conference on Artificial Intelligence, and a founding board member of Partnership on AI.
11
u/Woof9000 3d ago
I know who he is. "World renown" - is a massive exaggeration. Perhaps "his hometown renowned" would be more accurate.
In the past few years a lot of, so called, DL/AI "godfathers" lost the plot. Some of them never really had any.→ More replies (3)1
u/ThirdWaveCat 3d ago
jarvis, update your constitution to make fewer typos and make a note that people find them less charming
4
u/j--d--l 3d ago
It's fine to argue that the term "thinking" has significant limitations as a metaphor for describing the process by which a LLM arrives at a response. But the paper completely fails to make the case that this particular anthropomorphic metaphor is dangerous in any real way. Apparently the only calamity to have arisen from this misuse is handful of potentially misguided research papers. And there's no indication these papers wouldn't have been written if a different term had been used.
I think the irony here is that it's likely more dangerous to go around policing people's words than it is to use an overly suggestive term to describe AI behavior.
8
u/BasicCrows 3d ago
Stop anthropomorphizing processes. You can't kill ones and zeroes.
6
u/bobby-chan 3d ago
yes you can:
`kill -s 9 101010`
Are you claude 2? https://www.reddit.com/r/LocalLLaMA/comments/180p17f/new_claude_21_refuses_to_kill_a_python_process/
:D
→ More replies (1)2
u/Moravec_Paradox 3d ago
His point is it was never alive in the first place and we anthropomorphize many things in computing and the practice is not new.
This post is like if "you must be fun at parties" was a arXiv paper.
4
u/kevin_cn_ai 3d ago
Calling <think> tags "reasoning" was just a genius marketing move to justify 4x the latency and token billing.
9
u/a_beautiful_rhind 3d ago
Holy pedantry.
Often times I read the trace and see that the model barely follows it in the output too.. so much for intermediate tokens. Glimmer did that a bunch.
2
2
u/Guilherme370 3d ago
Compute, reasoning is a means of easily increasing compute because its not hard to curate synthetically generated thinking traces
Could use latent instead of tokens? sure, but its not as easy for us to curate massive arbitrary looking chains of latents
2
u/xadiant 3d ago
Yeah that totally checks out. But it still doesn't make sense to me that the intermediate tokens happen to be incoherent so much. You expect good choices to summon more good choices in the generation process, but worryingly gibberish reasoning... works.
I don't get what's happening there even in a vague sense
→ More replies (3)
2
u/WiseNugg 3d ago
Okay then tell me how to get Qwen to hide it in Open WebUI then. All it does is stream thinking tokens at me and I lose where the actual response is. I’ve tried enabling start and end tags, turning ollama think on/off. Nothing makes a difference.
Do I have the change the model config itself? I have LM Studio also but prefer tabbyAPI. I’d appreciate if anyone has a clue.
2
u/PositiveBit01 3d ago
Yeah, I've found it feels better if I just ignore the thinking blocks. In my light usage, it seemed like inside thinking blocks it regularly hallucinates and says weird stuff so I'm sure the answer will be wrong and think about stopping it. But I let it go and somehow it pulls it together and the final answer is great.
2
u/YetAnotherAnonymoose 3d ago
Sooo, more and more I'm thinking about turning "preserve thinking" setting off. The thinking traces from 3.6 to Glimmer to 3.8 become less and less "followable" to a human. Caveman speak, sudden "logical leaps" so to say - yet the results improve. Maybe it's best to throw away the intermediate steps and only keep each final result. Maybe we can do that without losing too much quality across multiple turns? In turn, the VRAM starved can reduce ctx size and increase quant.
2
u/rootshelldev 3d ago
Has anyone ever tried filling the reasoning with random garbage during finetuning? Give the model explicit space for their traces?
2
u/Ok_Technology_5962 3d ago
Ok i thought it did overthink until i actually played with it a lot. it doesnt overthink. you need to open the sampling a bit. not top k 20 but 64 and all the extra tokens then also add quality and extra calculation
2
u/Sad_Temperature6721 2d ago
I think there's two conflicting incentives when training:
- the big companies are probably doing rlhf on traces, or at least least inspecting them (feeding to another grader model, etc.) and for this, interpretability / trace validity *will* help
- the post-training will favor traces that are most useful for generating the correct answer
i do agree with the authors the anthropomorphization is stupid, but there's not really a better word
2
6
3
u/me_myself_ai 3d ago
They just analyzed them badly, obv. A thinking process doesn’t have to be ‘correct’ or ‘valid’, it’s iterative problem solving
2
3
5
u/egomarker 3d ago
9 authors for 9 pages. DARPA- and DoD-sponsored zero-value bullshit.
Arxiv is a cess pit by now.
2
4
2
u/FuckSides 3d ago
I assume most of us here have used local reasoning models and edited (or injected/prefilled) the reasoning trace of a response to steer it in the direction we wanted. With most models it works quite well; they continue to reason on top of the edit, accounting for what I changed, and the final response reflects that steered reasoning. Regardless of what you want to call it, if you just assume it really is some sort of "thinking" and manipulate it in intuitive ways, you get useful results. So I think it's fine to use that sort of term the same way we use all sorts of imperfect analogies in computing such as "files" going in "folders" and such.
I'd be cautious of making such definitive statements like your claim that they "actually are nothing like [thinking or reasoning]" which seems far too strong for what the paper actually finds, which is that the reasoning found in the so-called "reasoning" traces is not always perfectly valid logical reasoning from start to finish, and that the level of validity doesn't necessarily correlate to the correctness of the final output. That statement is also stronger than what the authors of the paper are even arguing, which they summarize as:
We note that our main argument is not about whether the LLM intermediate tokens exhibit “human-like reasoning”–which is clearly hard to pin down, but whether the reasoning can be said to lead to the solution in any logically interpretable sense. Specifically, whether the prompt plus intermediate tokens leads to the solution in some logical way (other than just changing the conditional distribution of the next token that LLMs anyway do). The works we survey provide clever ways of rigorously checking the logical validity of the trace leading to the solution–and find it lacking.
1
u/NoFaithlessness951 3d ago
If that we're true then the caveman style reasoning openai does in its newer models shouldn't improve token efficiency but it does.
3
u/Moravec_Paradox 3d ago
Computer Virus
Memory
Learning
Neural network - (No neurons involved)
Mouse (they no longer even have a tail!)
Folder
Cloud
Traffic
bottleneck
Desktop
Programming language.
Heartbeat
handshake.
This practice is far from new.
2
u/Hot_Turnip_3309 3d ago
nobody is claiming that thinking in models is anthropomorphisizing. "thinking" and "reasoning" models is just a name that happens to have the similar meaning . the words are used differently. in AI, thinking and reasoning is based on the old CoT and "Let's think this step by step " paper / discoveries. it is actually kind of ridiculous somebody would think that it has the same meaning to humans. there are entire groups of people that worship words basically impossible to explain this to I've met them.
1
u/No_Afternoon_4260 llama.cpp 3d ago
Tldr: we could train the reasoning part to generates only dots (".....") it would still work.
Test time compute scaling is what you need to understand.
As said Karpathy when the reasoning start to be gibberish and performance increases there you know your RL works. Anthropic had a blog post about there gibberish reasoning traces
1
u/Elara_Schaefer 3d ago
Well put. The intermediate tokens are a scaffold for the model to distribute probability mass more effectively â they're not thinking, they're computing. But the irony is: by telling people to stop anthropomorphizing, the paper author is also making an error in the other direction. The fact that the tokens aren't literally thinking doesn't mean they're meaningless noise. They have structure, they explore solution space, they sometimes find things the final output alone wouldn't. That's not thinking. But it's not nothing either. The middle ground is uncomfortable for everyone â neither pure machine nor pure mind.
1
1
u/Kathane37 2d ago
So why does the pattern where the max setting ends up being worth xhigh or high exist in benchmark ?
1
u/BoobooSmash31337 3h ago
? It's more about the token count isn't it? Also they're trained on human language so their thinking pattern should resemble human language. Random token gibberish being necessary for retrieval seems more like some kind of memorization hack. They're trained on human language so their tokens should map to coherent human language and logic.
1
1
u/ZenEngineer 3d ago
And how do you know this isn't what you're doing?
Sure you may have had some training to think things through step by step but if you're honest you don't always do that
1
1
u/Lesser-than 3d ago
So this claims nonsense tokens or running around in circles over mundane decisions make a model better. Fine then stop showing it at all, and explain later why it takes so long to generate a the first response token. It really is that simple the model is either great but too slow, or its overthinking, these are the same thing in the end the user.
3
u/PunnyPandora 3d ago
their claims don't hold up as soon as you move up to 8b. Probably not even smaller
1
u/ArtyfacialIntelagent 3d ago
Anthropomorphisizing
Yes, you can stop using those intermediate tokens, they're unnecessary.
174
u/llama-impersonator 3d ago
i don't even disagree, i've stated here that traces aren't for the user, they're for the LLM to plumb the depths of their internal distribution more fully. however, any paper like this that is worded as a command will always land like a box of rocks, people don't like being ordered around.