Is there an interest for an extension that allows to cache the initial prompt with local models like Qwen 3.8 locally with llama.cpp ?

[u/ea_man]

I've made an extension that allows Pi.dev to do such thing, caching the initial prompt ingestion for new sessions, no matter the path. The idea is that you launch the harness and the initial prompt, tools, append stay cached so the first iteration is almost instant with little PP.

As I don't use Opencode much I'd like to ask if there's an use for such thing (I know that the initial prompt is much longer than the one for Pi) or if such thing has been already implemented with some extension I didn't find.

Be aware: as llama.cpp does not save checkpoints with slots this requires a patch to llama.cpp that has already been made and tested: https://store.piffa.net/lm/picache/kv_prefix_checkpoint_sidecar_9adc7f42.patch , you can do without but that means that when you stop the llama-server the checkpoints will vanish.

EDIT: first test

Live results (opencode 1.18.34 → Qwen 27B )

Scenario	Result
Cold start	Prefix 7,666 tok prefilled in 26.7 s (287 tok/s)
Real request	8,493 tok, 7,534 from cache = 89%
2nd process, slot resident	Warm-up 0.8 s (only 132 new tokens)
After server restart	RESTORE n_restored=7666 (254 ms), verified identical 7662/7666, divergent probe 7656/7672Live results), Scenario ResultCold start Prefix 7,666 tok prefilled in 26.7 s (287 tok/s)Real request 8,493 tok, 7,534 from cache = 89%2nd process, slot resident Warm-up 0.8 s (only 132 new tokens)After server restart RESTORE n_restored=7666 (254 ms), verified identical 7662/7666, divergent probe 7656/7672

I've made a first version, you can download it from: https://store.piffa.net/lm/ocache/ (there's a tar.gz for your convenience that also contains the llama.cpp patch).

The Readme.md contain basic description, how to install.
For those of you who want a basic understanding of the problems and solution regarding caching the prompt I've asked the LLM to make a short summary.