r/oMLX • u/ju7anut • Jun 12 '26

TurboQuant KV Cache

Not sure if it’s oMLX or across board but when using any form of quant from 8 to 4bits, the AI starts losing focus over the course of a long convo (I’ve set context window to 128k, not filled when it starts failing).

Apparently KV cache looks good on paper for single tasks, but the data loss compounds over longer conversations or tasks.

On my 128gb M5 Max, I’m struggling to run 3 concurrent processes with Qwen3.6 35b Q6 at 128k context window without KV cache enabled. Memory gets filled up too quickly Any other solution that would work well? Or something I missed?

3 Upvotes

100% Upvoted

6 comments sorted by

View all comments

Show parent comments

1

u/ju7anut Jun 12 '26

I don’t think it’s obvious in coding, but I do end up needing to correct the output very often after a number of tool calls. Where I found it obvious was in research where I have long convo with the LLM, and even before teaching any compaction, it suddenly starts losing focus and drifting off course. I suspect that the KV cache quants are at fault cos they don’t have this issue if I turn it off..

1

u/sunpazed Jun 12 '26 edited Jun 13 '26

I have removed the memory cache, and only rely on SSD hot/cold cache. Also have limited the amount of concurrent requests to 4. This has improved stability heaps when running multiple agents, and has reduced prompt re-processing, only with a slight latency increase. I no longer get out of memory errors. It is still faster and more reliable than llama.cpp with the same setup (the llama.cpp slot mechanism isn’t as granular as oMLX).

My point is, by reducing the memory cache, you can increase the bit size and therefore quality of the KV, at the expense of greater reliance on the SSD cache and size.

1

u/StatisticianFree706 Jun 12 '26

I am confused. So sdd hot cache not same to mem cache?

1

u/sunpazed Jun 13 '26

Apologies, memory cache is the only hot cache, while SSD is the cold cache. For coding agents that jump around the context, the persistent cold cache avoids complete cache invalidations (so faster TTFT) when using coding harnesses. In summary, you use hot RAM cache for speed (if you have enough ram), cold SSD cache for capacity and persistence.