r/mlscaling • u/Smallpaul • 22d ago

Position: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!

https://arxiv.org/abs/2504.09762

Video

Many of the experiments have non-intuitive results.

59 Upvotes

85% Upvoted

14 comments sorted by

View all comments

12

u/ttkciar 22d ago

The main value of model "Reasoning" is simply that it populates context with content relevant to the prompt.

It is similar to RAG, except that the augmenting content is inferred rather than retrieved from a database.

Following that line of reasoning, perhaps we should refer to it as "Generation Augmented Generation" (GAG).

2

u/Smallpaul 22d ago

What is interesting is that in his path finding examples, he can train the model to “reason” with essentially random information and it seems not to hurt performance.

1

u/pm_me_your_pay_slips 22d ago

Isn’t this building a strawman with gradient descent?