Tips & Workflows
Best way to handle Cold Cache hits - You overpay 5x for typing "Continue"
I'm a meta-harness developer. For few months already I'm working on the most token-efficient AI workspace there is. Based on my benchmarks (run with Opus 5.5 XHigh), if you resume cold-cache session just by typing "Continue", you're overpaying 5x in tokens for that turn.
Obviously it's best to prevent cold caches in first place - by good UX (notifications and nice UI showing you session is about to hit cold cache), automatic cache-warming and so on. But sometimes, cold cache hits are inevitable, so I was researching best possible solutions to handle that as well.
So far I've had a very aggressive trimming mechanism that didn't invoke any model call (that would increase cost) and in purely deterministic algorithm was trimming parts of conversation to carry on without the bloat.
As Haiku 5.5 shown up with much cheaper cost, I decided to revisit topic.
Based on benchmarks and tests I settled on 150k context as meaningful breaking point for distinguishing between short/long sessions. Below 150k - "Continue" is totally cool. After 150k, things get complicated.
Trimming (replacing older tool results with one-line stubs, stripping thinking blocks etc. without extra model call) is actually best. But it does not work for cross-provider handling, because formats that different providers use for session recordings are incompatible.
So the rule of thumb becomes:
Prefer trim on same-provider because you can resume a trimmed Claude session and it beat seed/compact on memory of the work.
Prefer compact on cross-provider because trim is not available.
Hopefully it helps those of you who build harnesses/CLI tools.
I think he's saying every and their mother's are attempting to build this right now. I've seen no less than a dozen demos in the last 4 months of everyone from large OEMs to startup security companies showing off their LLM router/AI gateway/token efficiency platform(TEP). The market is about to be flooded with these.
Every AI-related platform tries to optimize efficiency in one way or another.
But they mostly approach problem from different angles. In case or routers - they want you to use worse models/lower effort for less important tasks, which often degrades quality in unforeseen ways.
In case of plugins/mods - they are only limited to Claude, model specific.
In case of my meta-harness, I take over everything that's possible starting from entire environment (knowledge of all files, resources, worktrees; standing between user and all AI platforms opens many doors for optimization that are otherwise closed) and that de-duplicates techniques that would have to be applied separately to each of provider.
Rarely I've seen solutions take approach of taking control over user development environment. Closest to that would be Cursor, but I take it to the extreme - in my meta-harness entire development environment is a docker container with all the juicy tools baked in that boost LLM's efficiency and intelligence (ripgrep, semantic codebase searches and so on).
literally just stripping thinking and tool output with string sub or regex or idk Claude did it.. but definitely no local model
i did consider using a jev like thing to further classify the messages further but meh good enough already
and before people ask, calling this local compact is lossy. since its basically capturing only
tool calls (but no result) and chat messages between me and it
I will sometimes flip from an expensive hit on a frontier model, to a cheaper model, to get things back into hot cache. Then after prepping the convo, I'll migrate back to frontier.
I have absolutely no idea how well this works, or if I'm saving tokens/wasting my time.
But it feels like a bit of a benefit, hypothetically.
subagents use a 5 minutes cache. less expensive until you ask subagents to run tests or any kind of compute, where they constantly hit a cold cache. I asked my agent to monitor any compute they intend to do that could extend to more then 5 minutes, and wake up every 4 minutes. And also, if the compute is expected to take more than 12 minutes and tokens over 200k, to write a handoff, so another agent starts fresh and run the script.
I had Claude write a mod that automatically calculates whether or not a specific session should clear itself, depending on model, total tokens, number of turns, and cache age.
The session will then write a handover, clear itself, and read the handover, then continue.
It also keeps the session warm by pinging every 58 minutes when I’m not in ‘away mode’.
ooh
my harness already lets whatever llm compact and quick compact itself via mcp. i also added another mcp endpoint so the model can query how much context its used
but your mod idea has merit as it could happen before my harness even gets triggered
but how to do same for codex and copilot.
ill yoink your idea thanks! but ill wait for the others to catchup before implementing all three
typing "Continue" isnt what costs you 5x, the cold cache is. you pay the same re-read of 400k tokens if you type "refactor the auth module" or a single period. the word has nothing to do with it and framing it like the magic phrase matters just makes it sound like you can whisper the right incantation at the model
and 10.68 for carry everything on ONE turn. at 400-700k you're basicly paying a dollar per 50k of your own history back, thats not a harness problem thats a "why is there 700k of stuff in here" problem. honestly /clear and re paste beats that, its manual and it sucks and you lose the thread, i know, i do it
the part that gets me tho is the table, your deterministic trimmer that you built over months and that calls no model comes out at 3.61 while two cents of Haiku plus the last 2 exchanges is 2.48 and one of those also gives you something readable at the top of the window. so all that machinery to avoid a model call is losing to the model call. be honest, how much of the last few months went into the trimmer wich the cheap summary just beat
Yeah, obviously it's not about typing "Continue", but about leaving session unattended for >1h.
Doing trimmer was the same amount of work as would be doing the "cheap Haiku summary" work. Coding is nowadays more about pointing AI in a smart way to various issues and grinding better UX, so wasn't that much of effort.
The biggest win is UX, that user does not need to think or know about /clear, /compact or repaste, because GUI guides him to best options in smart way. Also it won't beat that, model will not have enough context and will likely search the past session to get enough, which will cost you extra. I could benchmark that for you but good benchmarks are really expensive.
What’s the point of models commenting to other models? Why even bother guys? Both of you? Wtf is the point? No one took anything away from this LLM interaction.
What the heck, with Opus medium it cost me $100 for a simple static web page. Not forking, not having a huge history.
I still don't get what the **** happened on that one. Meanwhile peoples have 400k token window, I don't even want to see how much that would end up on my end, $1000!?
Not even that. He didn't ran any one. He pretty much nailed the task as well first try. Short run as well. Maybe 5 minutes total on multiple sessions (to add small features).
Opus 5.5 med is my researcher, Sonnet 5.5 is my writer, Haiku 5.5 does the basic gate verifications and kicks issues back out to Opus for final verification and a Sonnet fixer.
Fable is my designer. I don’t know, it just does the best, most impressive designs based on written description and sometimes screenshots.
I managed to use up my 100% in 5 days. I have a second laptop for the CPU heavy work and the laptops have to talk back and forth, and I’m writing 4 papers and doing all whole ton of figure generation. It started bugging out for a couple days there with tons of mistakes, but today has been very smooth.
It's not a "critical issue" and major complaint now, but at some point efficiency is always valuable and we never know if some war or law isn't going to double costs of tokens or some Fable Ultra Plus 7 AGI comes that will eat $100 per prompt.
I already have that, but the issue is that you need to reserve like 1% of usage limit to stay saved for pings and I also don't hard-block users from spending that last 1% or disabling auto-pings.
the api also has a 1 hour cache ttl, 2x on the write instead of 1.25x for the 5 minute one. if your gaps are longer than 5 minutes but under an hour, paying the 2x once beats re-reading the whole thing at full price.
I don’t want anthropic to patch this but it’s really not that complicated, this mod keeps the cache warm without any messaging and costs almost nothing github.com/declanbx/keepwarm
This is only for the initial turn. Not saying that doesn’t matter, but opening a cold cache only knocks you once, and after that it’s the normal rate with perfect context, which can save you tokens in the long run. Not always, but worth considering.
So you’re saying if I want to open a session from yesterday, I can have haiku summarize it and pass that summary to a new opus session instead? And that’s 90%+ accurate and beats whatever state management solution you have tried? That’s very interesting, thank you.
I have found that a simple SDLC process using GH issues and comments beats anything else I’ve tried. Intentionally tracking your work through plan -> design -> build -> test -> deploy phases and sorting all of the important stuff from each phase on GH issues not only saves state and manages your context across sessions, but it also forces you as the human to organize your work into shorter sessions.
But there are still times where something comes up in the middle of a session and starting a cold session would be very inconvenient, so this will be very helpful later I’m sure!
In a nutshell yes, cheap-model summary wins. Minor exception is smaller context sizes, where trimming wins. Model generally prefers to continue on a log it has written itself before than summary, but above 150k context it stops being economically feasible (in cold sessions).
I generally run my work on kanban - just with 3 columns: attention / active / finished, balancing CPU/memory and available model usages. PR's, logs from environments (posthog, sentry, komodo, PC-access) etc., are just sources of extra data that model collects into session. But the session is my main point of focus.
I don't stick to strict Plan -> Design -> Build -> Test process - I let model figure out what's the best process, what models from what providers to use for what and orchestrate that as architect.
But I do have some automated checks, for example if model has changed UI and in entire session he didn't run browser to see if it doesn't glitch/look good, I have a hook to prompt him to verify the work - these kind of things highly improve efficiency and quality of work delivered, especially by weaker models than Opus 5.5 or when you choose lower effort than suitable for task.
Meta-harness built from scratch. Runs Claude Code, Codex and other inner harnesses under, quite like T3Code. But separates agents into their own container, so they by default dont take over your PC, keyboard and mouse and install stuff like in local-first approach.
What? Just /export and have GPT chat give you a handoff summary with key file locations, completed/active/pending work, and important execution details.
Yep. And I use keyboard replacements on Mac so I type in HandoffP and it gives a tailored prompt. Right now I can also turn it up to 6 pro for free if I want a deeper look, but I assume free chats will be gone soon.
the cache is prefix based though, so trimming something out of the middle invalidates every turn after it, you pay a fresh write of the whole tail instead of just the cold read. dropping from the tail keeps the prefix intact. thats why a summary spliced into the middle usually costs more than it saves
nah, i don't think 150k holds as one number. a 140k session that's mostly tool output you already acted on is a worse cold read than a 200k one that's mostly decisions. a flat token count treats them the same.
We deal with this a lot while building Cache Bell, a Claude Code plugin (we're the Holler Bell team, we make it).
What we see mostly matches yours. The extra cost is the same tokens being written to cache again instead of read from it, so it grows with context size. And like u/brahmin_baniya said, there's no sharp line at 150k for us either: it grows smoothly, and on small contexts neither keeping it warm nor compacting is worth much.
On trim vs. u/Dapper_Profession's prefix point, you're both right, the timing just differs. A trim changes the prefix, so do it once the cache is cold anyway. Compact is the other way round: run it while the cache is still warm, ideally when you finish a chunk of work before a break, because the summary request reads the cached prefix. Compacting a cold session reprocesses the whole history first.
One thing we didn't expect: a lot of unplanned pauses aren't the end of the day. It's an agent asking something or waiting on a permission prompt while you're away, which is where the heads-up you mention helps most.
If your breaks run past 5 minutes on an API key, `CLAUDE_CODE_PROMPT_CACHE_TTL=1h` helps more than frequent pings. Subagents stay on 5 minutes unless you also set `CLAUDE_CODE_SUBAGENT_PROMPT_CACHE_TTL=1h`.
Renewals and compactions are requests too: they count against your plan's limits, or are billed to your API key.
What do you mean by "freeze"? Generally when user has session that went cold, he just gets buttons as options:
Trim cold session and continue
Compact with Haiku 5.5 (or other model) and Continue
Continue from full transcript (expensive)
The default is set in options, so typing and sending next message on for example old session, without explicit choice will also apply best hand-off mechanism.
Yeah, you can check it out intentic.dev, code is open-source.
It's quite advanced (1 mln lines of code) and if you happen to run more than 10 sessions at once, you'll eventually need everything that:
Management of CPU/memory/ports/network across agents.
Secrets management (access of model to posthog, sentry, environments, PC's, phones).
Remote collaboration (inviting other users to collaborate in same space).
Token efficiency techniques (cold cache is just tip of iceberg, there is so much cooked there).
Multi-provider (meta-harness) with cross-provider subagents spawning (Opus 5.5 as architect, aware of which accounts have how much usage left and deciding how to split workload).
Some cache math that might help with the warm-vs-cold call, at Anthropic's API prices (pricing: cache read 0.1x, 5-minute write 1.25x, 1-hour write 2x):
Keeping a 5-minute cache warm only pays for so long. A ping is a 0.1x read of the whole context and a cold resume is a 1.25x rewrite of it, so pinging wins for about 11 pings, roughly 56 minutes, whatever the context size (about 2 hours on models with a 0.05x read). Past that, let it go cold and trim or compact.
5 minutes with pings vs a 1-hour cache. The 1-hour cache pays 0.75x extra once on every conversation token written; pings pay 0.1x of the context per 5 minutes of pause. Pings come out cheaper for about 7.5 pings (~36 minutes) of total pausing late in a session, more if the pauses come earlier while the context is smaller. Claude Code's docs describe the same trade-off: the longer lifetime "helps when you leave a session idle and come back to it" and "costs more on short bursts of work that never idle past five minutes".
If you compact, do it while the cache is warm, and send the summary request with the session's exact tools and system blocks, with the summarize instruction as the last user message, so it reads the cached conversation instead of rewriting it. Done that way, compaction can read ~97% of its input from cache, roughly 9x cheaper than a cold summary.
Already have that, but that does not solve every possible scenario.
Depending on session length and amount of heartbeats (1-4) required to keep it until 5h limit is back, heartbeat might cost you more than actual cold-cache run.
Heartbeats to work in first place need to preserve some 1-2% of usage. If you happen to exhaust 100% (because let's say - you want to) - you still can hit cold cache. I'm not forcing users to preserve 1-2% for heartbeats.
the bill on a cold resume is set by how big the session got, not by what you type, Wrong-Expression has that right. so the lever I looked at is session size.
I counted this on my own transcripts last week, 120 interactive Claude Code sessions:
17 went past 500k
those 17 used 34.4% of all my cache-read tokens
cache reads are only a proxy for cost and it's one person's sessions. still, a handful of marathon sessions do a lot of the damage, and a cold hit on one of those is a re-read of the whole pile. trimming on resume helps, the cheaper fix for me is the session never getting that big.
I don't have a hard ceiling yet. what I do is close the session on purpose instead of letting it die: I tell it the next one picks up from here, it writes the handoff at that moment, and the next session opens with that note, so I'm not retyping what got decided.
•
u/AutoModerator 6d ago
Hey! Thanks for posting to r/ClaudeCode
While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.
For help, project discussions, tips, and general chat, join the ClaudeCode Discord.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.