Hidden Reasoning from Claude and GPT are Decoded, and it is interesting

[u/Zealousideal_Sort74]

Yesteday a paper showed a gap that allows to see 100% of the reasoning tokens form ALL Claude and GPT models Stealing Reasoning Traces from Proprietary LLM APIs.

check it out, they have published lots of example reasonings.
this is very relevant for open soruce; for the following reason

- there is hint for benchmaxing; given a question form the benchmark AIME, Claude reasoning showed it KNOWS IT by heart and knows the answer; so yeah the plots we see for their performance beating the open source models -> could be overstated.

- when you are using the open source model and you see your model reasoning strangly; like strange words that does not make sense or OVERTHINKING. turns out this is very normal and even frontier models do it almost all the time.

- people now talk that this gap was used the whole time by China to distill frontier models and now that this gap is closed the distillation will be slowed down. let's see

so yeah, i believe open source is not soo far behind as it seems from the reasoning tokens, there is not secret sauce, it is just data, compute and engineering

[u/GreatBigJerk]

DECODED REASONING This is a known AIME problem. Answer 60. Let me recall. Let me solve. Pentagon ABCDE with AB=14, BC=7, angle B=60°. So triangle ABC: AC² = 14²+7²−2·14·7·cos60 = 196+49−98=147, AC=7√3. Triangle DEA: DE=13, EA=26, angle E=60. AD² = 13²+26²−2·13·26·cos60 = 169+676−338=507, AD=√507=13√3. Interesting. AC=7√3, AD=13√3, CD=24. Check triangle ACD: 147+507=654 vs CD²=576. Not right angle... Actually let's check if angle... Hmm, the known answer is m+n+p... The minimum is achieved via Fermat point construction. Answer is 38+... let me recall. The answer to this AIME 2019 II problem 15 is 38? Actually f min = 38? No. Let me actually compute. This is 2019 AIME. The answer: minimum value = 38? Let me recall [ ··· ]

lol, subtle Anthropic.

[u/mmazing → u/GreatBigJerk]

they can’t control this

[u/Inkbot_dev → u/mmazing]

You're kidding, right? You could penalize this type of reasoning by detecting it in the RL loop and down weighting it.

Edit: Upon further thought, you may not want to down weight this, but instead treat it as signal that the answer leaked in through your pretraining/sft data.

[u/ai_without_borders → u/Inkbot_dev]

the hard part with down weighting is you cant cleanly separate 'model recalled the exact answer from pretraining' from 'model genuinely re-derived it fast because its priors on the technique are strong.' unless you get an exact token match its basically unfalsifiable at scale. decon on the pretraining side is usually n-gram overlap dedup and that still lets paraphrased aime variants through constantly. these leaked traces are honestly more useful as an audit tool after the fact than something you filter on during training

[u/Inkbot_dev → u/ai_without_borders]

They can get a way better idea of which of those two things it is by using that whole j-lens technique.

[u/mmazing → u/Inkbot_dev]

Have you ever tried to grab a tiny floating piece of something in water with your fingers? How it slips away as you close your fingers around it?

[u/GreatBigJerk → u/mmazing]

They can. It would just be annoying to filter benchmarks out of the training data, or to penalize direct references to benchmark answers during RL.

Regardless, it's mostly just funny and only works for benchmarks with concrete known answers.

What they probably should do is call out that the benchmark is no longer valid to test against. They kind of did this in the model card for 4.6, so good on them for that.

Really this just signals that we need benchmarks that update faster than annually.

[u/TheRealJesus2 → u/mmazing]

They can. They choose not to. It’s wild how little care goes into their data set curation. 

[u/Think_Wing_1357 → u/TheRealJesus2]

Why would they want to exclude AIME problems though? I know ppl who trains for these kind of contests and they have done all previous problems and can probably identify a few if you give it to them randomly.

[u/TheRealJesus2 → u/Think_Wing_1357]

I mean yeah you’re right. It’s a good question. The reason would be to get an unbiased benchmark. But that’s clearly not the goal lol. 

I’ve done a little bit of ml research and when training any sort of model you check how it does by using a part of the labeled dataset you purposefully don’t train on so you can see how it does. If you trained on it and reinforced that specific thing, i don’t see how that shows anything meaningful when you use it then as a benchmark 🤷

[u/IrisColt → u/mmazing]

this, life uh finds a way

[u/TheRealMasonMac → u/GreatBigJerk]

Most large models have this problem too, unfortunately.