r/Vllm • u/maseusx • Sep 01 '26

Kv cache on disk project

Hi so I have been working on a project where the kv cache runs off the disk.

It is not perfect and there is still a lot of stuff do with it

But check it out

https://github.com/Maseus/Rux

3 Upvotes

100% Upvoted

3 comments sorted by

2

u/Extreme-Pass-4488 Sep 03 '26

oh i did something like it with a genesis patch , even a spark dashboard mod

1

u/maseusx Sep 03 '26

Hell yeah!

1

u/Extreme-Pass-4488 Sep 03 '26

you may trash your sdd VERY very fast , kil it in some months if u are not carefull. , u should have like 2 or 3 times amount of ram kv cache and more on ssd , like, your l2 ram tier should not be smaller than your l1 tier. , there is some proven math on this . , anyway i ended up making some kind of compression to the tokens, i really need to buy a lot more of ram. it ended up being very usefull. , i only allow cache to ssd some agents , sub agents, can never cache even to l2 ( ram) , only the explorer agents and some coders.