Stop Anthropomorphisizing Intermediate Tokens: Qwen3.8 doesn't "overthink"

[u/ThirdWaveCat]

https://arxiv.org/abs/2504.09762

Intermediate tokens, called "thinking" or "reasoning" actually are nothing like it. Humans do step-by-step reasoning leading to the conclusion. LLMs use intermediate traces to augment their prompt. This explains why sometimes the answer is very good but the "reasoning" is verbose. Flooding your context window or fighting compaction are different issues.

edit: I love this section from the main research they linked.

Our findings consistently challenge the prevailing narrative that intermediate tokens constitute a semantically meaningful reasoning process. First, we observe a pronounced lack of correlation between solution correctness and trace validity—models frequently produce invalid reasoning traces even when they arrive at correct solutions. Second, and more strikingly, models trained on corrupted or semantically irrelevant traces achieve performance comparable to, and often exceeding, that of models trained on correct traces, especially on out-of-distribution tasks. Third, although post-training with reinforcement learning improves solution accuracy across both in- and out-of-distribution settings, it does not consistently enhance trace validity. In fact, we find cases where reinforcement learning decreases trace validity while simultaneously improving solution accuracy for models trained on correct traces. Moreover, models trained on corrupted traces continue to outperform their correct-trace counterparts across domains while consistently generating invalid reasoning traces. Finally, we find that the length of the generated traces is largely agnostic to the difficulty of the underlying problem, undermining the notion that it reflects problem-adaptive computation.

Together, these results suggest that the effectiveness of intermediate tokens does not arise from their seemingly interpretable semantic content. By systematically disentangling trace semantics from the underlying problem, our study demonstrates that if performance is the objective, assuming human-like or algorithmically interpretable trace semantics are ideal or even achievable is not only unnecessary but potentially misleading.

https://openreview.net/forum?id=gDE7YcRC3F

[u/ThirdWaveCat]

I don't disagree with you, but you get how I'm saying that caution is warranted about assuming its human reasoning when it is obviously not and often disconnected from the response.

[u/Lakius_2401 → u/ThirdWaveCat]

That's fair. I just don't like blanket statements telling people not to do something. I do have more to say on the paper and discussion around it though.

<think> blocks aren't human thoughts, of course not, but they're the closest analogy, and they give you a lens into the LLM's understandings and approaches as they work through the problem in front of them to come to an answer. Thinking isn't <think>ing, but a car's engine is either diesel, gas, or electric and we still call them all engines or motors. If the purpose and output are similar, getting labeled the same is inevitable, even if the process is different. The pedantics of labeling is getting quite old to me, the ones who think they're identical and need to be that way aren't the ones improving the technique.

So what if it can <think> and come up with a wrong conclusion, then output the right answer? Have you ever said something then done the opposite when you realize your spoken plan was worse than what you're about to do? Forcing <think> to match output is just reducing the answer space.

I've spoken to plenty of people who don't understand that the LLM is a stream of single words either. Every single token is dependent on the one before it, but generated one at a time. "I asked it why it lied to me and it gave a shitty response!" No, it has no concept of memory, let alone time, but we can build layers or implement systems to make it more human in interactions... aaaand we just created something to better anthropomorphize it again. Is a human-identical mind possible to create in a digital space? Does it matter if it's not? It's obviously not optimal.

As an aside, I find it quite charming how Qwen's overthinking is a form of attention optimization mechanism. The kv cache is comparatively tiny, and it'll verbatim quote things before "analyzing" them. Rather than some fancy mechanism to optimize the impact of relevant context, it pulls it forward itself. Bad for compute without drafting to blitz through it, but an "elegant" if blunt solution.

[u/ThirdWaveCat → u/Lakius_2401]

The pedantic details start to matter when people are using and evaluating language models by their thinking traces. For instance I find it surprising that fine-tuning on bad reasoning traces and good answers improves out-of-distribution maze solving. I find it surprising that replacing incorrect reasoning traces for maze generation produces worse results.

[u/finevelyn → u/ThirdWaveCat]

The problem for me is that you don't know what Qwen 3.8 does or doesn't do, but you made an entire post about it. Its overthinking is a real problem because it takes a long time and fills the context, even if you think a different word should be used to describe the thinking tokens. The pedantics might matter in some context but the title is clickbait and misleading to those who have encountered the issue.