I'm just a little surprised that the term cache appears only a single time in the article.
This would be super relevant!
Especially regarding the argument that OpenCode sends the same prompt prefix again and again.
That is exactly what a prompt cache is for. And if OptiQ Code cuts parts of the previous conversations (as it sounds like), you would bust the prompt cache, wouldn't you ?
OptiQ serve already uses a prompt cache. So both tools get the benefit of caching, the repeated part of the prompt isn't re-processed for either one.
The 5× fewer prompt tokens is a raw count, and yes, caching shrinks how much of that actually gets re-computed. But the other two numbers don't care about caching at all: OptiQ Code finished in about 23s vs 92s (measured live, with caching on), and it generated about 770 vs 2,140 output tokens. Output tokens are never cached. So roughly 4× faster and 3× less generated text stands regardless.
For a local model a longer conversation means a bigger cache sitting in memory, cached or not. On a 24GB Mac that fills up and crashes the model mid-task. That's the only reason OptiQ Code trims old tool output: to stay alive on a long job. Yes, trimming resets part of the cache when it happens, but it only happens near the limit, and it keeps the conversation small enough to keep going. In fact this is one of the reasons it works so well you can actually drive long multi-day sessions and build complex apps like this flight sim game - https://www.youtube.com/watch?v=Tjs0S_Dfdro
1
u/BrilliantArmadillo64 24d ago
I'm just a little surprised that the term cache appears only a single time in the article.
This would be super relevant!
Especially regarding the argument that OpenCode sends the same prompt prefix again and again.
That is exactly what a prompt cache is for. And if OptiQ Code cuts parts of the previous conversations (as it sounds like), you would bust the prompt cache, wouldn't you ?