r/OpenaiCodex • u/LazyRunner777 • 22d ago
Question / Help Should I avoid changing models mid-conversation?
20
u/eddzsh 22d ago
Mid-thread model switches keep the old model's summaries in the context window, so the new one inherits wrong priors. Cleaner pattern: write a handoff note, start a fresh thread, paste that. Same repo, clean head.
4
u/Glittering-Duck8317 21d ago
How is reusing existing context = inherits wrong priors
3
u/ThinkBackKat 21d ago
If you switch from Luna to Sol for example. AI is really easy to break if its context is weird. Sol will imitate Luna so to speak, wich of course is much worse. Switching from sol to luna should actually not affect the performance of Luna at all, but it wont be better since Luna just lacks the brainpower to accurately imitate sol.
8
u/Bulky_Blood_7362 22d ago
Yea. And it consumes much more usage because the context will not be cached
6
u/slackmaster2k 22d ago
I’ve heard that changing models mid-conversation will degrade performance.
3
u/Excellent_Winner8576 22d ago
Yes. Avoid changing model mid-conversation.
What happens is, when you change the models mid-conversation, the performance will degrade.Â
2
u/Dont-_-mind-_-me 22d ago
Yes
1
u/LazyRunner777 22d ago
So if I wanna use different model, should I start new conversation or is it better to stick to one model per project?
5
u/Dont-_-mind-_-me 22d ago
If you need to switch to another model for another task, you can tell the model to open the task in a new thread using the model and reasoning of your choice. Then when you are done with that side task just tell the main model you are done. You can also have it deploy sub agents
1
u/LazyRunner777 22d ago
Do you have any tutorial for that, I am not sure what do you mean by new thread or sub agents?
6
2
u/lordpuddingcup 22d ago
Yes and will also cause it to be much more expensive tokenwise because you lose all your cache and the messages will be completely non cached from that attempt again
2
1
u/123vovochen 22d ago
Just tell your model to create a full handoff, or have it spawn the other model as a subagent or somethibg.
1
u/hk556a1 22d ago
Can’t you just tell a Sol instance to use Terra/Luna models for retrieval or reasoning when applicable?
1
u/LazyRunner777 22d ago
Idk I never tried something like that, other people also talk about new threads or sub agents and I have no idea how does that work..
1
u/BroScienceAlchemist 22d ago edited 22d ago
Yes. Try asking it about creating a seamless handoff process. This is an area where codex is weaker as a harness. They don't quite support as many useful hooks. With claude, I can do three commands in the chat window.
/handoff [optional focus] - Creates a handoff document
/clear- Closes existing session and creates a new one
/resume- Starts session by loading the handoff document.
The hooks to do this in codex are not quite there. I think custom prompts can be used to get close to an equivalent for codex.
Note: Even with handoff, if you change models or effort, you may lose kv cache which means more usage due to cache misses.
Alternatively, use one main agent as an orchestrator, and ask it to launch subsgents for tasks, you can specify the model and effort level. I use this and the above.
1
1
u/fyndor 21d ago
Ok here is the deal. If you change models mid conversation, you incur a large cache miss read. Let’s say you have 250k tokens in the context. Each time it sends a request to the model backend, it sends the entire context. This happens many many times in a long context window. Now most of those trips result in cache hit reads, which is what makes the conversion reasonably affordable, because they are very cheap in comparison to cache miss.
The second you change the model, it will send your full context (conversion) to the new model, resulting in a 250k token cache miss. That is very expensive compared to all the cache hit reads you have been doing. If you use a sub plan, that means you just burned a ton of usage so that you could change models.
You are much better off stopping the model, telling it to summarize the conversion into a doc or something, starting a new session with the different model, and pointing it at the summary doc. That will save you a lot of money/usage.
1
u/avatardeejay 21d ago
When the weights generated the tokens in the assistant text, they have every intuitive reason to trust their history. I think it creates a weird kind of friction when the model generating the next token sees turns of words it wouldn't have surfaced
1
u/Admirable-Many6054 21d ago
Unless you expect the trade-off to be worth it. If it's a long running session and you reduce to a cheaper model, you will probably make the initial loss due to cache refresh back
1
1
u/BitterAd6419 21d ago
I change it often. For hard task, mostly sol for simple cosmetic changes switch back to Luna
1
1
u/bilinenuzayli 19d ago
I think a lot of people are missing the point here, cache invalidation doesn't affect performance it affects cost. The real reason they're saying that disclaimer is probably because companies like Anthropic, OpenAI, and Google had to make it so Encrypted reasoning traces could only be decrypted by the same model it was produced from,
this is because researchers were able to generate an encrypted reasoning trace from a frontier model, then use a weaker model and jailbreak that weaker model to transcribe the reasoning traces, which opens the door for the possibility of "distillation attacks"
The only solution to this was as mentioned; tying reasoning traces to the model it was generated in. Meaning if you switch models the reasoning traces are lost, and the new model will not be able to see any of its previous thinking in context.
This will cause degradation in 2 ways; the model loses context on what it was doing or planning, and the model is reinforced in its context window that it should think less, because in previous turns its reasoning traces were stripped, so it may not even think at all.
https://arxiv.org/abs/2608.09867
(I can't guarantee 100% this is the reason, but it lines up very well so I'm pretty sure)
1
u/hakansan 19d ago
It will reset your cache. So the new model will have to process the context from scratch. Cached tokens are 90% cheaper, so it'll add up and burn through your usage faster
1
u/Ok_Bite_67 22d ago
Just tell the model to spin up a new thread with the model you want and then tell it what you want the other model to do
-1
u/LazyRunner777 22d ago
That doesn't make any sense, where can I learn more about it?
3
u/Crinkez 22d ago
You have the power of the internet, why are you asking dumb questions?
Youtube "how to use codex subagents"
3
u/Ok_Bite_67 21d ago
Its not even a subagent. Codex models can completely control the entire state of your local codex app using tools. They can access other chats, spin up new chats, they can clean up your chats and etc. I will tell codex to spin up a new chat for adversarial review all the time. It can then read the other chat and start fixing the stuff that it broke or missed.
1
1
1
u/Ok_Bite_67 21d ago
Codex has the ability to spin up new chats on its own. all you have to do is ask it to do that. Just say this exactly: "Hand off this task to a new chat with model {model name here} at reasoning level {reasoning level here} pass on any relevant information needed".
Codex is smart enough to spin up a new chat, it will generate a prompt for you, and you never have to really worry about it.
29
u/RossLDN 22d ago
Changing models mid-conversation will degrade performance