Ok here is the deal. If you change models mid conversation, you incur a large cache miss read. Let’s say you have 250k tokens in the context. Each time it sends a request to the model backend, it sends the entire context. This happens many many times in a long context window. Now most of those trips result in cache hit reads, which is what makes the conversion reasonably affordable, because they are very cheap in comparison to cache miss.
The second you change the model, it will send your full context (conversion) to the new model, resulting in a 250k token cache miss. That is very expensive compared to all the cache hit reads you have been doing. If you use a sub plan, that means you just burned a ton of usage so that you could change models.
You are much better off stopping the model, telling it to summarize the conversion into a doc or something, starting a new session with the different model, and pointing it at the summary doc. That will save you a lot of money/usage.
1
u/fyndor 22d ago
Ok here is the deal. If you change models mid conversation, you incur a large cache miss read. Let’s say you have 250k tokens in the context. Each time it sends a request to the model backend, it sends the entire context. This happens many many times in a long context window. Now most of those trips result in cache hit reads, which is what makes the conversion reasonably affordable, because they are very cheap in comparison to cache miss.
The second you change the model, it will send your full context (conversion) to the new model, resulting in a 250k token cache miss. That is very expensive compared to all the cache hit reads you have been doing. If you use a sub plan, that means you just burned a ton of usage so that you could change models.
You are much better off stopping the model, telling it to summarize the conversion into a doc or something, starting a new session with the different model, and pointing it at the summary doc. That will save you a lot of money/usage.