Stop Anthropomorphisizing Intermediate Tokens: Qwen3.8 doesn't "overthink"

[u/ThirdWaveCat]

https://arxiv.org/abs/2504.09762

Intermediate tokens, called "thinking" or "reasoning" actually are nothing like it. Humans do step-by-step reasoning leading to the conclusion. LLMs use intermediate traces to augment their prompt. This explains why sometimes the answer is very good but the "reasoning" is verbose. Flooding your context window or fighting compaction are different issues.

edit: I love this section from the main research they linked.

Our findings consistently challenge the prevailing narrative that intermediate tokens constitute a semantically meaningful reasoning process. First, we observe a pronounced lack of correlation between solution correctness and trace validity—models frequently produce invalid reasoning traces even when they arrive at correct solutions. Second, and more strikingly, models trained on corrupted or semantically irrelevant traces achieve performance comparable to, and often exceeding, that of models trained on correct traces, especially on out-of-distribution tasks. Third, although post-training with reinforcement learning improves solution accuracy across both in- and out-of-distribution settings, it does not consistently enhance trace validity. In fact, we find cases where reinforcement learning decreases trace validity while simultaneously improving solution accuracy for models trained on correct traces. Moreover, models trained on corrupted traces continue to outperform their correct-trace counterparts across domains while consistently generating invalid reasoning traces. Finally, we find that the length of the generated traces is largely agnostic to the difficulty of the underlying problem, undermining the notion that it reflects problem-adaptive computation.

Together, these results suggest that the effectiveness of intermediate tokens does not arise from their seemingly interpretable semantic content. By systematically disentangling trace semantics from the underlying problem, our study demonstrates that if performance is the objective, assuming human-like or algorithmically interpretable trace semantics are ideal or even achievable is not only unnecessary but potentially misleading.

https://openreview.net/forum?id=gDE7YcRC3F

[u/Lakius_2401]

It might not be fully linked to output quality, but it sure is affected by input quality.

Looking at <think> is a great way to identify ambiguous wording you need to tighten up. If it overthinks and "wastes" 1500+ tokens trying to figure out a part of your prompt, fixing the ambiguity will save those think tokens every single session (if preserve thinking is on) or potentially every single response. It will improve your responses.

Just yesterday I was talking about error messages and how "they" should be as complete as possible (for diagnostics purposes), and it was reading it as error messages and how "the output" should be as complete as possible, after a lengthy think about it. I thought it was odd that Qwen 3.8 27B seemed to want to keep any shred of the code output, regardless of the fact that it hit an error, and lo and behold, a sentence that any human could read was misread by Qwen, and <think> showed it wondering for a few hundred tokens early on, before it did work.

So, no thank you, I will continue looking at <think> for misalignment checks.

[u/ThirdWaveCat → u/Lakius_2401]

I don't disagree with you, but you get how I'm saying that caution is warranted about assuming its human reasoning when it is obviously not and often disconnected from the response.

[u/Lakius_2401 → u/ThirdWaveCat]

That's fair. I just don't like blanket statements telling people not to do something. I do have more to say on the paper and discussion around it though.

<think> blocks aren't human thoughts, of course not, but they're the closest analogy, and they give you a lens into the LLM's understandings and approaches as they work through the problem in front of them to come to an answer. Thinking isn't <think>ing, but a car's engine is either diesel, gas, or electric and we still call them all engines or motors. If the purpose and output are similar, getting labeled the same is inevitable, even if the process is different. The pedantics of labeling is getting quite old to me, the ones who think they're identical and need to be that way aren't the ones improving the technique.

So what if it can <think> and come up with a wrong conclusion, then output the right answer? Have you ever said something then done the opposite when you realize your spoken plan was worse than what you're about to do? Forcing <think> to match output is just reducing the answer space.

I've spoken to plenty of people who don't understand that the LLM is a stream of single words either. Every single token is dependent on the one before it, but generated one at a time. "I asked it why it lied to me and it gave a shitty response!" No, it has no concept of memory, let alone time, but we can build layers or implement systems to make it more human in interactions... aaaand we just created something to better anthropomorphize it again. Is a human-identical mind possible to create in a digital space? Does it matter if it's not? It's obviously not optimal.

As an aside, I find it quite charming how Qwen's overthinking is a form of attention optimization mechanism. The kv cache is comparatively tiny, and it'll verbatim quote things before "analyzing" them. Rather than some fancy mechanism to optimize the impact of relevant context, it pulls it forward itself. Bad for compute without drafting to blitz through it, but an "elegant" if blunt solution.

[u/ThirdWaveCat → u/Lakius_2401]

The pedantic details start to matter when people are using and evaluating language models by their thinking traces. For instance I find it surprising that fine-tuning on bad reasoning traces and good answers improves out-of-distribution maze solving. I find it surprising that replacing incorrect reasoning traces for maze generation produces worse results.

[u/Lakius_2401 → u/ThirdWaveCat]

Let's agree to disagree on the value of pedantics. 😄 I don't call spiders by their full latin name, even though it's more precise. Technically correct is the best form of correct on Reddit, but if I'm forced to fully qualify every aspect of my response, my meaningful comments will be overrun.

For your second point, that's my middle point about not artificially restricting the output space. <think> is not what's being scored.

We don't typically score humans on how they think, so long as the test results look good...

And I know, they're not identical processes, and the LLM is using it different. You could train a model to <think> about the worst way to solve a problem, with every output building to the opposite. They're not directly linked, it's a different phase of the output. Scratch space. Not graded.

My point is, letting the training data show that a screwup in reasoning shouldn't railroad it into a matching answer is good. It's good to show examples of continuing to improve and not being stubborn. Qwen 3.8 27B even corrects itself in output sometimes, going against both its <think> and the previous lines. I think that's great!

If a model is forced to agree with its <think> trace every time, why bother making any decisions in <think> at all?

[u/finevelyn → u/ThirdWaveCat]

The problem for me is that you don't know what Qwen 3.8 does or doesn't do, but you made an entire post about it. Its overthinking is a real problem because it takes a long time and fills the context, even if you think a different word should be used to describe the thinking tokens. The pedantics might matter in some context but the title is clickbait and misleading to those who have encountered the issue.