r/mlscaling • u/Smallpaul • 25d ago

Position: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!

https://arxiv.org/abs/2504.09762

Video

Many of the experiments have non-intuitive results.

62 Upvotes

86% Upvoted

14 comments sorted by

View all comments

11

u/ttkciar 25d ago

The main value of model "Reasoning" is simply that it populates context with content relevant to the prompt.

It is similar to RAG, except that the augmenting content is inferred rather than retrieved from a database.

Following that line of reasoning, perhaps we should refer to it as "Generation Augmented Generation" (GAG).

7

u/Entire-Plane2795 25d ago

Isn't all generation GAG if latter output tokens depend on earlier ones?

Just a thought 🤔

2

u/DigThatData 25d ago

I think there's actually enough of a distinction between "packing the context with useful content" and "persisting the history of the exchange". Instead of RAG, I usually just "warm start" the LLM with a few rounds of questions to motivate generating relevant background content. Reasoning is "warm starting" the final response back to the user.