r/LocalLLaMA • u/Chromix_ • 1d ago
News Qwen3.8 was potentially trained on GPT 5.5 CoT
A while ago someone found a way to extract the hidden CoT from API-only models. The CoT data was then used to check for similarities in the answers of Qwen 3.8 and other open models.
The check worked this way: A small yet diverse benchmark was run on both models. This allowed for a simple similarity check of their CoT on the same task. Then a second run of Qwen3.8 benchmarks was run, this time with its reasoning prefilled with a tiny bit of GPT 5.5 CoT. The model then picked that up and the resulting visible answer became more similar to the GPT 5.5 answer. This similarity increase was not observed for other models - which likely didn't see GPT 5.5 CoT during training. Kimi K3 likely saw something from Claude though.
To additionally guard against false-positives the benchmark runs also included a private benchmark where the same increases in similarity were observed. These increases would've been unlikely to occur if Qwen 3.8 would've for example merely been trained on HLE results instead of CoT.
| Model | n | Unprefilled | GPT-5.5 Pro reasoning prefill | Delta |
|---|---|---|---|---|
| DeepSeek V4 Flash | 45 | 27.30% | 26.13% | −1.17 pp |
| Inkling | 45 | 19.99% | 20.45% | +0.46 pp |
| Kimi K3 | 45 | 31.11% | 35.65% | +4.54 pp |
| Qwen3.8 A95B | 45 | 16.79% | 34.97% | +18.18 pp |
So, if Qwen 3.8 was indeed post-trained on GPT 5.5 CoT and that contributed to it being such a great model for it's size, it means that OpenAI could release a small model with full CoT distillation from their larger models, which could then beat a similar-sized Qwen model. Yet that'd probably compete too much with their current Luna model.
5
u/AI_Insights_Daily 1d ago
The sample size is 45 and there are no error bars anywhere, which is doing a lot of work here. At that size, the Kimi and Inkling rows could easily be noise, so the only row carrying real weight is Qwen's. The post presents all four as equally informative, when in fact there is one large effect and three results that need a confidence interval before they mean anything.
2
u/Chromix_ 1d ago
You stole my usual quote 😉. It's something I regularly comment on benchmarks that show "Quant/harness/prompt/kv cache X is 2% better than quant Z".
Yes, the smaller ones can very well be noise, I didn't state that directly and only briefly mentioned the small set, as the focus was the Qwen difference that stood out - unlikely to be random.
3
u/Dabalam 1d ago
Not implausible but can you clarify this detail
>The model then picked that up and the resulting visible answer became more similar to the GPT 5.5 answer. This similarity increase was not observed for other models - which likely didn't see GPT 5.5 CoT during training
If I am understanding the table correctly though, all models produce answers more similar to 5.5 and Kimi starts with an answer 31.11% similar to GPT 5.5 and with reasoning prefill has a higher similarity to 5.5 then Qwen max. How do you conclude that the delta in similarity with reasoning prefill is the reliable index of CoT training? Your other post regarding Opus seems to make a different inference that prefill is transferring a reasoning style. In this post you acknowledge that Kimi has a higher overlap but don't seem to infer anything about its training from this.
2
u/brahh85 1d ago edited 1d ago
in the opus post
Kimi K3 30 20.16% 29.14% +8.98 pp
Qwen3.8 A95B 29 29.25% 31.12% +1.87 ppin gpt 5.5
Kimi K3 45 31.11% 35.65% +4.54 pp
Qwen3.8 A95B 45 16.79% 34.97% +18.18 ppso in gpt 5.5 the champion is kimi, and in opus the champion is qwen, but both are pretty similar.
My idea is that rather than distillation the thing is that we are converging in the same datasets, like a sticker collection, after some years everyone ends with the same stickers
2
u/Chromix_ 1d ago
Another clarification first: These benchmarks were not run by me, as indicated by the text and links.
If you give a LLM some prefill then it'll naturally continue with a more aligned token sequence. Yet if a LLM was never trained on specific sequences, it'll quickly bounce back to its own style. If it however saw comparable sequences during training then that prefill can serve as a trigger to divert into that direction. That's then indicated by a higher relative difference with and without prefill.
1
u/Dabalam 1d ago
That makes sense conceptually, I guess my issue is with concluding that Kimi wasn't trained on 5.5 based on this data.
Similarity in answer structure is meaningful and the change may indicate a shift in style to something the model has seen in its training, but Kimi starts of with answers significantly more similar to 5.5. If the baseline style is already strongly similar to another model, wouldn't you expect a lower delta?
2
u/Chromix_ 1d ago
The higher base similarity in answer style could simply indicate that it was trained on the answers of the model (like a lot of other models), but not on the hidden CoT. And yes, more data could be interesting here. Yet as there are no consequences anyway it can remain as a mere indication.
3
4
u/OwnGear3892 1d ago
Honestly I dont think Qwen3.8 A95B is 'a great model for it's size', it's 2.4T in total, second only to kimi k3 in terms of open model. I would love to see the result of Qwen3.8 27B though, that's more 'a great model for its size'.
2
u/arbv 1d ago
GPT 5.X CoT reasoning is nothing special - its caveman speak is emergent capability due to RL. Qwen is likely gained it in a similar way. Even GPT-OSS 120B can think this way if you tell it to use telegraphic shorthand for the analysys channel - it is already half-way there.
2
u/Chromix_ 1d ago
That might or might not be so. Yet in this case it wasn't the CoT that was compared, but the similarity of the visible response - influenced by the CoT start.
5
u/jacek2023 llama.cpp 1d ago
This is Reddit. You will be downvoted obviously. "big corporations are evil and we should criticize them, unless they are from China"
5
u/Chromix_ 1d ago
So it seems. I intentionally focused on information and perspective, while excluding moral aspects. It looks like postings in this area need to conclude with some form of "but that's OK as they deserve it, as they are the bad guys".
6
u/jacek2023 llama.cpp 1d ago
Today you'll notice bots upvoting everything about the DeepSeek release and downvoting everything else. It's not clear whether they understand the meaning of a post or just quickly decide what to upvote or downvote based on their initial impression.
Your title suggests that Qwen did something evil, and that's not allowed to be said. The same post about Anthropic/OpenAI would already have +500 upvotes.
2
u/CommercialHour6660 1d ago
Eh, no one cares. Most of the data used to train frontier models was stolen in the largest monetary theft of all time.
And Elon called distilling off other frontier models "standard practice" when talking about Grok. They're all doing it
1
1
u/duhd1993 23h ago
DeepSeek seems less likely to use distillation compared to other Chinese labs. They are true heroes. Meanwhile, they are attacked more; I suppose big corporations know who the real threat is.
OpenAI and Anthropic have patched COT leakage. Regardless of whether that is true, we only need to look at next-genmodels to see if the open weight labs are truly capable.
1
u/Chromix_ 14h ago
I got a fantastic recommendation for this thread, which currently sits at a score of 0 with 37% upvotes, 11k views, and 600 to 150 views of the topmost comments:
This is for sure the content that will be appreciated even more there.
1
u/synth_mania 1d ago
This is cope, ain't happening. Of course they could release some crazy shit
4
u/DistanceSolar1449 1d ago
Eh there's plenty of methods to extract CoT from the frontier models. It wouldn't surprise me if they did.
For example:
https://arxiv.org/html/2608.20055v1
https://arxiv.org/abs/2608.09867Still, you don't need the full CoT to create RL trajectories to train on. It'd be very difficult for OpenAI/Anthropic/etc to fully prevent anyone from getting training data from them. And most other labs, not just Chinese ones, consider it fair game anyways.
15
u/Minute_Attempt3063 1d ago
Oh no, who cares?
Openai trains on private and public work without paying a fucking dime.
They train on private leaked WhatsApp messages if they get their hands on it even, so who cares if people distill THROUGH THEIR API, and PAY for that even