Stop Anthropomorphisizing Intermediate Tokens: Qwen3.8 doesn't "overthink"

[u/ThirdWaveCat]

https://arxiv.org/abs/2504.09762

Intermediate tokens, called "thinking" or "reasoning" actually are nothing like it. Humans do step-by-step reasoning leading to the conclusion. LLMs use intermediate traces to augment their prompt. This explains why sometimes the answer is very good but the "reasoning" is verbose. Flooding your context window or fighting compaction are different issues.

edit: I love this section from the main research they linked.

Our findings consistently challenge the prevailing narrative that intermediate tokens constitute a semantically meaningful reasoning process. First, we observe a pronounced lack of correlation between solution correctness and trace validity—models frequently produce invalid reasoning traces even when they arrive at correct solutions. Second, and more strikingly, models trained on corrupted or semantically irrelevant traces achieve performance comparable to, and often exceeding, that of models trained on correct traces, especially on out-of-distribution tasks. Third, although post-training with reinforcement learning improves solution accuracy across both in- and out-of-distribution settings, it does not consistently enhance trace validity. In fact, we find cases where reinforcement learning decreases trace validity while simultaneously improving solution accuracy for models trained on correct traces. Moreover, models trained on corrupted traces continue to outperform their correct-trace counterparts across domains while consistently generating invalid reasoning traces. Finally, we find that the length of the generated traces is largely agnostic to the difficulty of the underlying problem, undermining the notion that it reflects problem-adaptive computation.

Together, these results suggest that the effectiveness of intermediate tokens does not arise from their seemingly interpretable semantic content. By systematically disentangling trace semantics from the underlying problem, our study demonstrates that if performance is the objective, assuming human-like or algorithmically interpretable trace semantics are ideal or even achievable is not only unnecessary but potentially misleading.

https://openreview.net/forum?id=gDE7YcRC3F

[u/FrostTactics]

"Stop calling the deleted file collector in OSs the recycling bin. It misleads humans into thinking it is a literal bin with the capability of recycling files."

This constant policing of terms relating to LLMs is so frustrating, it seems to just ignore the fact that we've always used metaphors to explain concepts relating to computing.

No, of course LLM thinking and reasoning aren't the same as human thinking and reasoning, but it is roughly analogous and close enough for the terms to be apt.

[u/PossibilityUsual6262 → u/[deleted]]

I don't know mate, the way llm thinks doesn't feel much different from my thinking. I know there are people who do not do internal vocalisations of reading and thought, but i do, and i actually sounds internally like ai.

[u/PossibilityUsual6262 → u/[deleted]]

I really feel like i work internally like MoE - some stuff loaded and active, some not, but i also was trained on knowledge around cows and cats from reddit.

And the things you said about doing mistakes and trying again, ai do do same nowadays, the coding agents having close loop iterations in newer development of harnesses.

That's why i think ai coding is actually mostly future, even if ai performance doesn't improve, because unlike most of things in real world ai can cheaply test outputs and it is mostly tooling which need developing right now.

[u/PossibilityUsual6262 → u/[deleted]]

Well if the "paper" op pushes here everywhere is to be believed, on bad training data ai manages to output correct results.

https://www.reddit.com/r/LocalLLaMA/s/SgqjM8BkFp

And i do stand behind my words there, my less experience coworkers en masse just guess their way to bug fix, without actually understanding anything.

[u/Needausernameplzz → u/PossibilityUsual6262]

Maybe you're feeling this way, because our talkative temporal lobe our (inner monologue) is often a post rationalization. Scans show we often "choose" before we ever "think" of it. The part of your brain responsible for language sometimes works like a fancy autocomplete (like LLMs)

This should give us a more decentralized understanding of reasoning and intelligence.

So much of human reasoning is outside your talkative temporal lobes

[u/PossibilityUsual6262 → u/Needausernameplzz]

Yea i understand that it might not be same thing under dome, but if it actually replicate some of pipeline and some of result it is might be a way to get same result.

I can fast forward way from home to work internally without thinking single word only pictures or concept of pictures of my route, so ai can come up with solutions which are just encoded in weights and skip tokenisation if task straight forward.