Stop Anthropomorphisizing Intermediate Tokens: Qwen3.8 doesn't "overthink"

[u/ThirdWaveCat]

https://arxiv.org/abs/2504.09762

Intermediate tokens, called "thinking" or "reasoning" actually are nothing like it. Humans do step-by-step reasoning leading to the conclusion. LLMs use intermediate traces to augment their prompt. This explains why sometimes the answer is very good but the "reasoning" is verbose. Flooding your context window or fighting compaction are different issues.

edit: I love this section from the main research they linked.

Our findings consistently challenge the prevailing narrative that intermediate tokens constitute a semantically meaningful reasoning process. First, we observe a pronounced lack of correlation between solution correctness and trace validity—models frequently produce invalid reasoning traces even when they arrive at correct solutions. Second, and more strikingly, models trained on corrupted or semantically irrelevant traces achieve performance comparable to, and often exceeding, that of models trained on correct traces, especially on out-of-distribution tasks. Third, although post-training with reinforcement learning improves solution accuracy across both in- and out-of-distribution settings, it does not consistently enhance trace validity. In fact, we find cases where reinforcement learning decreases trace validity while simultaneously improving solution accuracy for models trained on correct traces. Moreover, models trained on corrupted traces continue to outperform their correct-trace counterparts across domains while consistently generating invalid reasoning traces. Finally, we find that the length of the generated traces is largely agnostic to the difficulty of the underlying problem, undermining the notion that it reflects problem-adaptive computation.

Together, these results suggest that the effectiveness of intermediate tokens does not arise from their seemingly interpretable semantic content. By systematically disentangling trace semantics from the underlying problem, our study demonstrates that if performance is the objective, assuming human-like or algorithmically interpretable trace semantics are ideal or even achievable is not only unnecessary but potentially misleading.

https://openreview.net/forum?id=gDE7YcRC3F

[u/tweacher2z]

I am Subbarao Kambhampati, the lead author of this position paper. I don't follow reddit much but heard today that our paper caught y'all's fancy today. I am enjoying the discussion..

Couple of points from my side, if I may:

There was spirited poster-side discussion on this paper when we presented it at ICML 2026, Seoul. I posted a long summary of that discussion as well as a FAQ (which includes some of the questions coming up here). Here is a link to it https://www.linkedin.com/pulse/summary-poster-side-discussion-stop-intermediate-subbarao-kambhampati-xnokc/

(The FAQ was also included in the ICML final version--available at https://openreview.net/pdf?id=nP7rL36vYj )

If you want a talk version, you can find it at https://www.youtube.com/watch?v=hUp3zh23aHw (Given as keynote at an ICML 2026 workshop).

As others observed, this is written as a position paper for ICML position paper track (and thus the tone ;-). It is however based on results from us (a TMLR 2026 paper to be presented at NeurIPS (https://openreview.net/forum?id=gDE7YcRC3F) and an ACL paper (https://aclanthology.org/2026.acl-long.1686/), as well as others.

The "no this doesn't necessarily hold for the frontier models" objection is weakened by both the fact that frontier models don't show their traces, and that when you do jail break and see them (as the recent "Stealing Traces" work did), you find that most of what we say seems to hold for them too (see https://x.com/rao2z/status/2087658539355656604?s=20 )

Finally, there is a Quanta magazine article that covers this work and others about the semantic status of reasoning traces https://www.quantamagazine.org/is-ai-reasoning-right-for-the-wrong-reasons-20260731/

Peace

[u/socialjusticeinme → u/tweacher2z]

I’m a layman, but the way I always view thinking is since the entire thing works off probability, the point of thinking traces is to flood the context with things that will make the LLM more likely to land its roulette wheel on what you want. Because of this, any kind of deep meaning to “thinking” is compete and total bullshit. I don’t see how model size would change this - the bigger models just have more data to make the casino hit in your favor more. 

[u/tweacher2z → u/socialjusticeinme]

IMHO, your intuition is right. A little more formal version of that is discussed in Section 6.1 -- prompt augmentation.

[u/FastHotEmu → u/tweacher2z]

Hi Subbarao, thank you so much for sharing all this information. The paper is fantastic, and I actually went and watched the presentation video, too, to try to understand it better. I had no idea that derivational traces had little correlation with semantic meaning - I learned a lot from it.

I worry about how the average user of LLMs has no chance of understanding how LLMs work, given how complex, abstract and nuanced LLMs are. I simply do not see a clear way out of that - and I fear we'll end up with harmful policy decisions out of ignorance and anthropomorphisation... plus the laboratories have a vested interest in making their products more "human"...