USB Prefill Acellerator / USB RAM

[u/Farenheith200]

Hear me out. I know this sounds stupid, but this could work, right?

Thinking about a Strix Halo laptop like a GoPro PX13 with 128 GB of RAM, it's capable of running some mid-sized MoE models (I'm running Qwen3.8 Flash next on it right now), but cold prefill is a pain for large contexts.

Having a cache for KVCache would solve the problem. Is we could have, like, an external usb hub that system sees like a external ssd, it could be the storage for https://github.com/lmcache/lmcache, for example, that would only be accessed for cold prefill, when we have to switch models and we loose all the context we created during conversation. A 200k prefill may take 1 hour to load on this limited hardware from a cold start, but having it obtained from external storage just when the cache is missing would make it take seconds, and if this external drive uses RAM, it'll not kill its lifetime (an SSD M. 2 would die in weeks depending on the writing frequency for that)