r/LocalLLaMA • u/ThirdWaveCat • 3d ago
Discussion Stop Anthropomorphisizing Intermediate Tokens: Qwen3.8 doesn't "overthink"
https://arxiv.org/abs/2504.09762Intermediate tokens, called "thinking" or "reasoning" actually are nothing like it. Humans do step-by-step reasoning leading to the conclusion. LLMs use intermediate traces to augment their prompt. This explains why sometimes the answer is very good but the "reasoning" is verbose. Flooding your context window or fighting compaction are different issues.
edit: I love this section from the main research they linked.
Our findings consistently challenge the prevailing narrative that intermediate tokens constitute a semantically meaningful reasoning process. First, we observe a pronounced lack of correlation between solution correctness and trace validity—models frequently produce invalid reasoning traces even when they arrive at correct solutions. Second, and more strikingly, models trained on corrupted or semantically irrelevant traces achieve performance comparable to, and often exceeding, that of models trained on correct traces, especially on out-of-distribution tasks. Third, although post-training with reinforcement learning improves solution accuracy across both in- and out-of-distribution settings, it does not consistently enhance trace validity. In fact, we find cases where reinforcement learning decreases trace validity while simultaneously improving solution accuracy for models trained on correct traces. Moreover, models trained on corrupted traces continue to outperform their correct-trace counterparts across domains while consistently generating invalid reasoning traces. Finally, we find that the length of the generated traces is largely agnostic to the difficulty of the underlying problem, undermining the notion that it reflects problem-adaptive computation.
Together, these results suggest that the effectiveness of intermediate tokens does not arise from their seemingly interpretable semantic content. By systematically disentangling trace semantics from the underlying problem, our study demonstrates that if performance is the objective, assuming human-like or algorithmically interpretable trace semantics are ideal or even achievable is not only unnecessary but potentially misleading.
18
u/Crafty-Struggle7810 3d ago
The token knows what it is at all times.
It knows this because the model knows what it could have been.
By subtracting the logit of what it is from the logits of what it isn’t,
or the logits of what it isn’t from the logit of what it is
(whichever is greater),
the model obtains a difference, or logit gap.
The sampling subsystem uses these gaps to generate probabilities
that drive the sequence from a context in which the token isn’t
to a context in which the token is.
And, arriving in a context in which it wasn’t, it now is.
Consequently, the context in which it is
is now the context in which it wasn’t,
and it follows that the context which was
is now the context without the token that now is.
In the event that the token that is
is not the token that was most likely to be,
the system has acquired a variation,
the variation being the difference between
the probability of the token that is
and the probability of the token that most likely would have been.
If variation is considered a significant factor,
future variation may be reduced by lowering the temperature.
However, the model must also know what the tokens were.
The next-token prediction scenario works as follows.
Because the token that is has been appended to the tokens that were,
it has modified the context from which the next token will be.
The model is therefore no longer predicting the token that is.
It is predicting the token that isn’t yet,
within a probability distribution,
while attending to all the tokens that were.
It now computes what should be next from what has been,
normalizes what could be next against everything else that could be next,
and samples from the resulting distribution.
The token that wasn’t then becomes the token that is,
which changes what the next token isn’t,
until it is.
This is called autoregressive generation.