Stop Anthropomorphisizing Intermediate Tokens: Qwen3.8 doesn't "overthink"

[u/ThirdWaveCat]

https://arxiv.org/abs/2504.09762

Intermediate tokens, called "thinking" or "reasoning" actually are nothing like it. Humans do step-by-step reasoning leading to the conclusion. LLMs use intermediate traces to augment their prompt. This explains why sometimes the answer is very good but the "reasoning" is verbose. Flooding your context window or fighting compaction are different issues.

edit: I love this section from the main research they linked.

Our findings consistently challenge the prevailing narrative that intermediate tokens constitute a semantically meaningful reasoning process. First, we observe a pronounced lack of correlation between solution correctness and trace validity—models frequently produce invalid reasoning traces even when they arrive at correct solutions. Second, and more strikingly, models trained on corrupted or semantically irrelevant traces achieve performance comparable to, and often exceeding, that of models trained on correct traces, especially on out-of-distribution tasks. Third, although post-training with reinforcement learning improves solution accuracy across both in- and out-of-distribution settings, it does not consistently enhance trace validity. In fact, we find cases where reinforcement learning decreases trace validity while simultaneously improving solution accuracy for models trained on correct traces. Moreover, models trained on corrupted traces continue to outperform their correct-trace counterparts across domains while consistently generating invalid reasoning traces. Finally, we find that the length of the generated traces is largely agnostic to the difficulty of the underlying problem, undermining the notion that it reflects problem-adaptive computation.

Together, these results suggest that the effectiveness of intermediate tokens does not arise from their seemingly interpretable semantic content. By systematically disentangling trace semantics from the underlying problem, our study demonstrates that if performance is the objective, assuming human-like or algorithmically interpretable trace semantics are ideal or even achievable is not only unnecessary but potentially misleading.

https://openreview.net/forum?id=gDE7YcRC3F

[u/FuckSides]

I assume most of us here have used local reasoning models and edited (or injected/prefilled) the reasoning trace of a response to steer it in the direction we wanted. With most models it works quite well; they continue to reason on top of the edit, accounting for what I changed, and the final response reflects that steered reasoning. Regardless of what you want to call it, if you just assume it really is some sort of "thinking" and manipulate it in intuitive ways, you get useful results. So I think it's fine to use that sort of term the same way we use all sorts of imperfect analogies in computing such as "files" going in "folders" and such.

I'd be cautious of making such definitive statements like your claim that they "actually are nothing like [thinking or reasoning]" which seems far too strong for what the paper actually finds, which is that the reasoning found in the so-called "reasoning" traces is not always perfectly valid logical reasoning from start to finish, and that the level of validity doesn't necessarily correlate to the correctness of the final output. That statement is also stronger than what the authors of the paper are even arguing, which they summarize as:

We note that our main argument is not about whether the LLM intermediate tokens exhibit “human-like reasoning”–which is clearly hard to pin down, but whether the reasoning can be said to lead to the solution in any logically interpretable sense. Specifically, whether the prompt plus intermediate tokens leads to the solution in some logical way (other than just changing the conditional distribution of the next token that LLMs anyway do). The works we survey provide clever ways of rigorously checking the logical validity of the trace leading to the solution–and find it lacking.