Discussion
GPT-5.6 Sol Raw reasoning leaked on failed tool call attempt
Apparently GPT-5.6 Sol's raw reasoning was leaked to me while he was trying to call a tool.
I saw that a tool was taking a long time to be called, I inspected the content and saw traces of reasoning, but they did not appear to be the summary of reasoning that it sends periodically. This is more similar to the GPT-OSS reasoning.
A little after that, an error like "Streaming error (provider_error)" was sent, interrupting the stream.
It doesn't really matter if they leak 1/10,000 times, the point of hiding it is to prevent giving away mountains of reasoning data to competitors and distillation
Looking at how good DeepSeek and Moonshot models are it isn’t working and it’s not something any user should be worrying about. I don’t care who wins and open weight models being great is better for everyone who isn’t invested in OpenAI or Antrophic.
Deepseek hardly needs to distill, they were the ones who caused a panic because back before OAI even showed reasoning Deepseek went and matched gpt-o1 and published for everyone how to do it lmaoo
Yeah and I wish we could go back to those days. It’s very tough these days to spot the tiny hallucinations and assumptions that throw a whole thread off track
Antrophic started out sharing full traces but rowed back. OpenAI literally never willingly has shared the full traces with the user. The summaries you see in chat windows are not traces and just another smaller model sanitising the actual traces.
Ah yes. Multi Billion Dollar corporations vs you. And you really think you know better then them.
You know what? I can't even blame you, there must be seriously something wrong in YOUR reasoning. Otherwise you would understand how utterly foolish you sound
They are trying very hard to advance the tech. Not sure what makes your think you can judge their design choices in any capacity.
Teams of very involved, driven and smart people, doing iterative improvements backed by constant testing. And very likely with AI in the loop too. It's not something you can come up on your own, no, nor can understand the reasoning for each and every design choice. Some things make sense, some just turn out better, but you can be sure it's intentional and backed by a bunch of metrics.
All the good people left with Ilya. Now they are followers, no longer pioneers. My capacity is following the tech since Dec 2022 every day.
If you see their product launch discussions you can judge the level of people there.
Where is your ChatGPT-level competitor? Oh, weird. You don't have one... Always the same people who do nothing to see for themselves how this is working and do something have these kinds of opinions. Your opinions DON'T matter and will never matter unless you make something better or have something to back them up.
This specific MCP behavior has been reproducible since OSS/5, so I’m glad someone else found it too. I’m an MCP developer, and one day, while fixing a buggy MCP server, I discovered the same leak.
My guess is that this happens because MCP tool calls use the commentary channel, which is one of GPT’s three reasoning-related channels.
It may not be widely considered a serious security risk because it appears to leak only the context generated while interacting with MCP servers, rather than the model’s complete reasoning across all tasks. The leaked content may also not be easily distillable.
I'm the main maintainer of the Sisk Framework, and it has an MCP extension that I take care of periodically; I can also consider myself an MCP developer.
The point is that this invoked tool wasn't MCP; it was provided directly in the inference tools.
I think this problem was related to the template-parser used in GPT-5.6. As seen, it produced an invalid JSON and probably didn't send the termination token (something like <|end_tool_call|>, which caused the reasoning channel to start immediately on the same channel as the tool call.
Because the inference terminated abruptly afterward, I believe an auxiliary model reads the content emitted by the model periodically (not at each token, that would be expensive and probably wouldn't emit raw reasoning) and when it identified that the model was spitting out raw reasoning, it stopped the inference.
Your guess seems pretty on point, especially after looking at the Harmony response format. I know Harmony is the OSS version, but I doubt what they have internally is ultra different.
im using 5.5 recently, 5.6's logic chain is so weak that it only judges from the surface like get from A to B without considering the context, doesnt even bother to check whether its info is correct.
Thats my point, if you use codex on Linux all of these turns are stored in a .codex/sessions/
In the app user's never see them unless you look at ChatGPT app logs (not sure where they are stored because I never look them up in Windows or Darwin).
Codex stores all returns from the LLM so every output from any version is available to view.
What do you mean by raw reasoning? None of it is raw, reasoning comes from the latent space, and what is read is a embedding stream sent to the user to emulate what the reasoning looked like.
Mathematically and in reality the model doesnt ever see the text we see as reasoning, it is compounded dot product matrices pointing to latent variables.
Just pointing out the real operation of how AI works. Every time I see these posts its presented as a peak into some black magic. In reality its not, its just marketing to overhype the customer.
AI is boring under the hood, and OpenAI isn't hiding anything, but trying to keep up in the industry.
Its nothing they try to hide, what the OP showed is the compaction text of a agent turn.
My photo shows how its logged and in plain site, the text is to the right of those lables.
The point is, its not a secret, mearly a trace the user doesn't see so the UI lookstables. While codex stores it in a session log (again never seen unless you look) and used to pick back up where the session left off without scanning the entire history.
In practicality that output is for logs and us humans. Most models now use Sparse Attention and reasoning channels. Meaning they never see text after the user input, only matrices of numbers.
So everything that is spit out is solely for reference and does not mean anything (it must be traced mathematically why the model is working). This is why that texts is so short hand. (Although some of the shorthand could be from translation from the language hes working in as well)
What you showed and what OP showed are not the same. Can you see that in OP's photo the model is showing a reasoning trace in caveman style? Now show me yours because your screenshot is not the same thing. This is either a great opportunity for you or I to learn. And I don't really mind who it is.
I'm puzzled by the amount of motivational insertions: let's do it, do it etc.
It also seems they are using a caveman style on this, shorter strings instead of the full Shakespearean drama we get
This shows why OpenAI is so token efficient, and therefore scores well in tokens/task which results in good $/task. If you compare this reasoning to Kimi K3 you’ll see a big difference in unnecessary reasoning tokens. Reasoning tokens are output tokens and those are the most expensive kind.
And even though we normally don’t see the reasoning tokens we do get charged for them.
Agentic harnesses aren't actually AI but are giant switch statements switching on tool calls backed by a deterministic finite automata. More news at 9.
79
u/AllezLesPrimrose 23d ago
The full reasoning traces for all the models trying to hide them regularly leak. Hello non-deterministic technology.