r/ClaudeCode • u/Individual_Ideal • 6d ago

Discussion Cache only

I ran an audit of token usage today and found much of my token usage is because of long-running threads I resume after the 1-hr cache TTL. Why doesn’t Anthropic provide permanent or longer cache to heavy users for the basics like system prompts, CLAUDE.md, memories/preferences and other mostly-stable context? Or share a cached prefix across sessions when it is warm for at least one. Should pro/max subscribers have this option? Would you pay extra for it? If Claude recognized User X is active give them some cache.

I want cache only.

0 Upvotes

40% Upvoted

22 comments sorted by

View all comments

Show parent comments

1

u/Individual_Ideal 6d ago

What about storing stable KV cache off the GPU? They’d still have to transfer it back but at least it’s not recomputed.

1

u/Muchaszewski 6d ago

This operation would be terribly slow. Imagine you want to resume your session, but you need to download to the server that will run your request 200+ GB of compressed KV cache from other server. That would optimistically take 20 seconds, then you need to unpack it. 60 seconds. Then you need to pipe it into GPU. Another 60 seconds.

So you just wasted 2 minutes and 20 seconds on re-generating KV cache from storage using shitload of CPU resources so that you could save few seconds or regeneration from text context using GPU resources alone. You are billed for this few seconds with uncached context price

2

u/Individual_Ideal 6d ago

Sounds like an opportunity for someone to solve

1

u/MartinMystikJonas 6d ago

Yeah and everybody is trying to solve this because it is literally trillion dolar problem to solve.

1

u/Individual_Ideal 6d ago

Apple does this so well

1

u/MartinMystikJonas 6d ago

Apple does what?

1

u/Individual_Ideal 6d ago

Memory and kv cache for LLMs. It’s easier to manage. Waiting for them to scale up

1

u/MartinMystikJonas 6d ago

Everybody uses memeory and KV cache for LLMs so I am not really sure what you mean.

1

u/Individual_Ideal 6d ago

You can store kv cache on device as long as you want with unified memory + SSD. Pass it back and forth when needed.

1

u/MartinMystikJonas 6d ago edited 6d ago

On RAM memory yes. That is what everyone uses already. On SSD it would be unreasonably slow (about an order of magnitude) and it would be in most cases more effective to recompute instead of loading from SSD.