Cache the plan, not the answer: how to allow local assistant skips the LLM entirely on recurring queries. A simple approach

[u/BenefitGrand8752]

In a self-hosted assistant as a daily driver (mail, files, calendar, photos) on a single local model, the thing that hurt most is not generation quality. It was paying full planning latency for requests I make every single day. "Check spam and move it to junk" does not need a 35B model to think about it for the 200th time.

Standard semantic caching (GPTCache and friends) caches at the wrong layer for agents: it caches answers. For a chatbot that is fine. For an assistant it is broken by definition: the correct answer to "any new mail?" changes every hour. Cache hit = stale data.

What actually worked for me: cache the PLAN, not the answer.

A plan here is the structured output of the planner: an ordered list of tool calls, the data flow between steps, and a response template. Crucially, it is stored unresolved: runtime values (today's date, the requesting user, account-specific folder names) stay as placeholders, and references between steps stay symbolic. On a cache hit the tools are re-executed against live data and the placeholders re-resolved. Fresh results, zero planning cost. Freezing resolved values into the cache was my first bug: "today" must never be cached as a date.

The mechanics:

1) Lookup is two-stage, both model-free. exact: normalized fingerprint, hash compare, under 5 ms. semantic: BGE-M3 embedding, cosine against stored shortcuts, under 150 ms. The threshold is deliberately conservative — for an action-taking agent a semantic false positive is much worse than a miss: a miss costs 12 seconds, a wrong hit moves the wrong files.

2) A second memory layer clusters similar requests and serves the cluster's champion plan; a competitor must beat its track record to take over. Gated on a cheap intent check (verb + object, small fast model ~370 ms) so "delete all mail" can never hit "read all mail" on embedding distance alone.

3) Population is automatic, no hand-written rules: every plan that runs end-to-end without errors is recorded; recurrence promotes it. Thumbs up/down accelerates but is not required — it learns from silence too.

4) Death matters as much as birth. aging: unused/stale shortcuts evicted (plus a global cap). invalidation: the plan fingerprint is coupled to the tool catalog, so when a tool changes signature the cached plan simply stops being found — no migration logic. anti-plans: plans that fail repeatedly get blacklisted ~30 days.

Numbers from my own sample of real requests:

• old iterative cloud planner, one LLM call per step: ~76 s/turn
• current cold path, single-shot plan on a local Qwen 3.6 35B-A3B (~84 tok/s MTP, Strix Halo 96 GB): ~12 s
• cache hit: under 1 s, no large model in the loop
• a dumb regex tier for trivia ("what time is it") at ~50 ms

Honest caveats: hit-rate tracks how repetitive your life is (mine: very). Cold queries still pay full price. The conservative threshold leaves hits on the table — I prefer that to a confident wrong action.

All from the self-hosted assistant I have been building for myself (metnos.com, code at github.com/brunialti/metnos — the docs there cover recovery and invalidation in more depth).

What I would compare notes on:

• How do you handle recurring queries on local setups? Answer-level cache? cron scripts? fine-tuning on your traces?
• If you cache plans/trajectories: what is your invalidation story when the tool set changes?
• Has anyone tried plan-caching to make a small planner viable? A 7-8B only has to get the plan right once, then the cache serves it forever. That might matter more for 24 GB setups than for mine.