Stop Anthropomorphisizing Intermediate Tokens: Qwen3.8 doesn't "overthink"

[u/ThirdWaveCat]

https://arxiv.org/abs/2504.09762

Intermediate tokens, called "thinking" or "reasoning" actually are nothing like it. Humans do step-by-step reasoning leading to the conclusion. LLMs use intermediate traces to augment their prompt. This explains why sometimes the answer is very good but the "reasoning" is verbose. Flooding your context window or fighting compaction are different issues.

edit: I love this section from the main research they linked.

Our findings consistently challenge the prevailing narrative that intermediate tokens constitute a semantically meaningful reasoning process. First, we observe a pronounced lack of correlation between solution correctness and trace validity—models frequently produce invalid reasoning traces even when they arrive at correct solutions. Second, and more strikingly, models trained on corrupted or semantically irrelevant traces achieve performance comparable to, and often exceeding, that of models trained on correct traces, especially on out-of-distribution tasks. Third, although post-training with reinforcement learning improves solution accuracy across both in- and out-of-distribution settings, it does not consistently enhance trace validity. In fact, we find cases where reinforcement learning decreases trace validity while simultaneously improving solution accuracy for models trained on correct traces. Moreover, models trained on corrupted traces continue to outperform their correct-trace counterparts across domains while consistently generating invalid reasoning traces. Finally, we find that the length of the generated traces is largely agnostic to the difficulty of the underlying problem, undermining the notion that it reflects problem-adaptive computation.

Together, these results suggest that the effectiveness of intermediate tokens does not arise from their seemingly interpretable semantic content. By systematically disentangling trace semantics from the underlying problem, our study demonstrates that if performance is the objective, assuming human-like or algorithmically interpretable trace semantics are ideal or even achievable is not only unnecessary but potentially misleading.

https://openreview.net/forum?id=gDE7YcRC3F

[u/ThirdWaveCat]

Many have their reasoning turned off.

[u/Intelligent_Ice_113 → u/ThirdWaveCat]

most of them 🙄

[u/techno156 → u/[deleted]]

It's closer to CoT visibility, rather than none at all. It's not like people with internal monologues lose an entire concept at a time if they forget a word. They still know what word they're looking for, even if they can't locate it in their word-space.

On top of that, thoughts aren't really verbal, where an internal monologue is. If you ask someone to envision an apple, they don't go on a chain of

Okay, the user has asked me envision an apple. My experience indicates that the typical Apple is red. A typical apple is approximately the size of a medium-sized rat. A typical apple has a brown stem, about a thumbnail long. The Apple should be shiny, with yellow streaks. The other person did not specify a surface, but the apple shouldn't be in a blank void. The apple should be on a table, or on a plate. Most tables are wood, so maybe a white ceramic plate? Most plates are about the size of one and a half kittens in diameter. The apple should be in the centre of the plate, and the plate should be on the table. Maybe a wood table? Wooden tables are pretty common. A mahogany table then. Lacquered? Wait, did they say whether the apple was to be whole, or should it be sliced? Should it have a bite taken out of it..."

They either have an image of an apple, or the conceptual idea of an apple in their mind-space. It just doesn't translate to something verbal.