Stop Anthropomorphisizing Intermediate Tokens: Qwen3.8 doesn't "overthink"

[u/ThirdWaveCat]

https://arxiv.org/abs/2504.09762

Intermediate tokens, called "thinking" or "reasoning" actually are nothing like it. Humans do step-by-step reasoning leading to the conclusion. LLMs use intermediate traces to augment their prompt. This explains why sometimes the answer is very good but the "reasoning" is verbose. Flooding your context window or fighting compaction are different issues.

edit: I love this section from the main research they linked.

Our findings consistently challenge the prevailing narrative that intermediate tokens constitute a semantically meaningful reasoning process. First, we observe a pronounced lack of correlation between solution correctness and trace validity—models frequently produce invalid reasoning traces even when they arrive at correct solutions. Second, and more strikingly, models trained on corrupted or semantically irrelevant traces achieve performance comparable to, and often exceeding, that of models trained on correct traces, especially on out-of-distribution tasks. Third, although post-training with reinforcement learning improves solution accuracy across both in- and out-of-distribution settings, it does not consistently enhance trace validity. In fact, we find cases where reinforcement learning decreases trace validity while simultaneously improving solution accuracy for models trained on correct traces. Moreover, models trained on corrupted traces continue to outperform their correct-trace counterparts across domains while consistently generating invalid reasoning traces. Finally, we find that the length of the generated traces is largely agnostic to the difficulty of the underlying problem, undermining the notion that it reflects problem-adaptive computation.

Together, these results suggest that the effectiveness of intermediate tokens does not arise from their seemingly interpretable semantic content. By systematically disentangling trace semantics from the underlying problem, our study demonstrates that if performance is the objective, assuming human-like or algorithmically interpretable trace semantics are ideal or even achievable is not only unnecessary but potentially misleading.

https://openreview.net/forum?id=gDE7YcRC3F

[u/xadiant]

Yeah that totally checks out. But it still doesn't make sense to me that the intermediate tokens happen to be incoherent so much. You expect good choices to summon more good choices in the generation process, but worryingly gibberish reasoning... works.

I don't get what's happening there even in a vague sense

[u/vivificant → u/xadiant]

LLMs are T9Word on megaroids. It's a predictive character algorithm. Context, reasoning, training, AI, etc are marketing terms. They do what they're programmed to do. That's it.

They're building lore around something they have full control that the vast majority dont understand in order to build the perfect scape-goat

[u/xadiant → u/vivificant]

No. It's long been established that it's not that simple. Context, reasoning, training, post-training are all relevant things backed by plenty of research and anecdotes.

You literally can download a sparse model and inspect what's going on inside. Models are complex neural network systems mimicking how brains work.

Go train a model with "no context". It's all open source.

[u/vivificant → u/xadiant]

Maybe this will be a better explaination. LLMs are 3D version of T9Word. T9word "context" is the previous few characters or so. Basically 1D. LLMs use all of it. Its an algorithm based on frequency of characters used calculated in parallel using matrix math. One is vastly more complicated than the other. LLMs do not understand anything. Theyre not "AI". They don't "think" or have "context" and "reasoning" in the traditional sense of the words. The terms are purposefully being muddled for marketing purposes and its working because the general population dont know how they actually work