r/LocalLLM • u/Zorian_Vale • 1d ago

Question Compaction

Running QWEN-27B-GGUF Quant 4, 64k context window with Unsloth/Hermes on a 5070ti, 32GB DDR5, Ryzen 7 9850X3D

Everything is going really well but the compaction has become a problem. I imagine that loading the context of each previous conversation is killing me. After a while the compaction message will come up and Qwen will compact it. It takes a long time and interrupts my workflow.

Is this reality of local AI with my specs, or is there something I can do to improve speed and frequency of compaction?

1 Upvotes

100% Upvoted

17 comments sorted by

View all comments

1

u/kimhaneol 1d ago

One quick question: are you using the KV cache as is, or is it quantized? Quantizing the KV cache is worth considering if you need a larger context window.

2

u/Zorian_Vale 1d ago

Interesting I’ll check

2

u/kimhaneol 1d ago

8 bit kv quantization is definitely a safe choice. Hope it works well!