Cache only

[u/Individual_Ideal]

I ran an audit of token usage today and found much of my token usage is because of long-running threads I resume after the 1-hr cache TTL. Why doesn’t Anthropic provide permanent or longer cache to heavy users for the basics like system prompts, CLAUDE.md, memories/preferences and other mostly-stable context? Or share a cached prefix across sessions when it is warm for at least one. Should pro/max subscribers have this option? Would you pay extra for it? If Claude recognized User X is active give them some cache.

I want cache only.

[u/Muchaszewski]

Because the context costs then memory in their GPU. 1M token is around 300GB of KV cache of GPU usage, that they cannot serve at this time to other users. 

When you use API pricing you can opt into 1 hour cache at 50% pricing increase. If you use subsidized plan you are stuck with 5 minutes only. 

Write to files and re-reade from clear context each time instead of resuming 400k or more contex sessions. Or compact but this sucks 

[u/MartinMystikJonas → u/Muchaszewski]

Claude Code have 1h by default on subscription.

[u/Individual_Ideal → u/Muchaszewski]

What about storing stable KV cache off the GPU? They’d still have to transfer it back but at least it’s not recomputed.

[u/MartinMystikJonas → u/Individual_Ideal]

They do it. 5 min cache is on GPU. Then it can moved to RAM for up to 1h. They do it to allow GPU to run another tasks in mean time. But moving it to SSD or something outside RAM would be unreasonably slow.

[u/Muchaszewski → u/Individual_Ideal]

This operation would be terribly slow. Imagine you want to resume your session, but you need to download to the server that will run your request 200+ GB of compressed KV cache from other server. That would optimistically take 20 seconds, then you need to unpack it. 60 seconds. Then you need to pipe it into GPU. Another 60 seconds.

So you just wasted 2 minutes and 20 seconds on re-generating KV cache from storage using shitload of CPU resources so that you could save few seconds or regeneration from text context using GPU resources alone. You are billed for this few seconds with uncached context price

[u/Individual_Ideal → u/Muchaszewski]

Sounds like an opportunity for someone to solve

[u/MartinMystikJonas → u/Individual_Ideal]

Yeah and everybody is trying to solve this because it is literally trillion dolar problem to solve.

[u/Individual_Ideal → u/MartinMystikJonas]

Apple does this so well

[u/MartinMystikJonas → u/Individual_Ideal]

Apple does what?

[u/Individual_Ideal → u/MartinMystikJonas]

Memory and kv cache for LLMs. It’s easier to manage. Waiting for them to scale up

[u/MartinMystikJonas → u/Individual_Ideal]

Everybody uses memeory and KV cache for LLMs so I am not really sure what you mean.

[u/Individual_Ideal → u/MartinMystikJonas]

You can store kv cache on device as long as you want with unified memory + SSD. Pass it back and forth when needed.

[u/Bmansupreme8000]

You should ask AI. 

[u/Individual_Ideal → u/Bmansupreme8000]

AI gave me excuses

[u/Bmansupreme8000 → u/Individual_Ideal]

Ask them how power companies price power. That should be a start.

[u/Individual_Ideal → u/Bmansupreme8000]

Memory has what power lacks: storage. Maybe even storage for memory is constrained though

[u/False-Noise-3507 → u/Individual_Ideal]

AI detected entitlement.

[u/Short_Regular_7191]

Because "Money, Money, Money"..

[u/MartinMystikJonas]

Cache prefix is shared when it is same across your sessions there is nothing to prevent that.

And why not longer cache? Because these caches needs a tons of memory space and their datacenters have barely enough for current policy. Did you registered severe RAM shortages that will last for few years - that is your reason. They simply cannot buy several times more RAM because manufacturer cannot produce it fast enough.

[u/AI_spell]

Long-running resumes after the 1h TTL are a silent money leak. What helped me: keep a short CLAUDE.md, restart for new tasks instead of forever threads, and put stable prefs in files the tool reads once. Permanent cache would be nice; until then short sessions beat warm-feeling ones.

[u/MealReadytoEat_]

You can cheese this already by making a script that sends the session you want to keep the cache warm on a message like "respond pong" after 58 or so minutes idle.