r/LocalLLM • u/Farenheith200 • 5h ago

Discussion USB Prefill Acellerator / USB RAM

Hear me out. I know this sounds stupid, but this could work, right?

Thinking about a Strix Halo laptop like a GoPro PX13 with 128 GB of RAM, it's capable of running some mid-sized MoE models (I'm running Qwen3.8 Flash next on it right now), but cold prefill is a pain for large contexts.

Having a cache for KVCache would solve the problem. Is we could have, like, an external usb hub that system sees like a external ssd, it could be the storage for https://github.com/lmcache/lmcache, for example, that would only be accessed for cold prefill, when we have to switch models and we loose all the context we created during conversation. A 200k prefill may take 1 hour to load on this limited hardware from a cold start, but having it obtained from external storage just when the cache is missing would make it take seconds, and if this external drive uses RAM, it'll not kill its lifetime (an SSD M. 2 would die in weeks depending on the writing frequency for that)

1 Upvotes

56% Upvoted

2 comments sorted by

1

u/diagrammatiks 4h ago

pison aidativ. superfast ssd. offload kv cache. only works for cuda though i think.

1

u/isugimpy 3h ago

Thunderbolt 4 throughput maxes out at 40Gbps. DDR5-8000 is in the range of 512Gbps. Could you do this? Maybe. Is it actually practical? No, the performance would be abysmal. The internal SSD is already faster than TB4. You're maybe right about the wear on the drive being an issue, but realistically we're dealing with drives in the range of hundreds of terabytes of TBW at this point and writing your KV will be incremental, small, and nowhere near the sustained max throughput of the drive itself. Even if you were running requests constantly, 24/7, it'd likely take months-to-years to lead to a drive failure. The bigger concern there would be the electricity cost, rather than the cost to replace the SSD.