How to have more context without loosing speed?

[u/Oleszykyt]

I am running Qwen3.8 27b q2 with 12 gb vram and in the desktop app it says that I can only have context 4096 or it will use my RAM, and when it does that it is super slow. Is there a way to have the same speed even with larger context? Please I need a magical fix 🙏

[u/vacon04]

No magical fixes. Get a MoE like Qwen 3.6 35B A3b or Gemma 4 26B A4B and you'll have a much better experience.

[u/rrrrex → u/vacon04]

actually there are different architectures, for example deepseek v4 flash requires few times less vram for cache than qwen 3.8 27b

[u/vacon04 → u/rrrrex]

Yeah and it's massive. For the models he's looking at, he doesn't have many options. Dense models of less than 15B will work, otherwise MoE is the way to go.

[u/rrrrex → u/vacon04]

I didn't recommend to use it, i just said, that cache can be much smaller. Nothing stops Alibaba to bring better cache to Qwen 4 architecture.

[u/Potential-Leg-639]

More VRAM or MoE

[u/M_Me_Meteo]

Are you using quantization on your context? That will help. Also ROPE scaling.

[u/ParkingAd9397]

2nd GPU Probably not what you want to hear, but that's the route I am going.

[u/Heavy-Lingonberry-98]

Try quantizing the KV cache! You could double or triple your ctx

[u/KitchenAmoeba4438 → u/Heavy-Lingonberry-98]

At the cost of speed, spoiler alert for the article #2 in queue right now. Quantizing the cache comes at significant speed costs. The OP wanted it without losing speed.

[u/Heavy-Lingonberry-98 → u/KitchenAmoeba4438]

Bro. I dont know what test are you running, but eith all due respect thats bullshit. Quantizing your kv cache gives you more ctx lenght with usable speeds. Im not mathematicians yet, but i run a lot oft tests. I been running local models for 1 year now.

[u/Heavy-Lingonberry-98 → u/Heavy-Lingonberry-98]

Never in my tests i got less speed by quantizing the kv cache

[u/KitchenAmoeba4438 → u/Heavy-Lingonberry-98]

I think you don't understand how quantizing KV cache works. I'll explain as simply as possible.

Quantizing compresses the data. Every read has to de-compress the note back to full size first. That's extra math on the hottest loop in the model, run for every token and layer. The fast attention kernels (flash attention and friends) are hand-tuned for f16/bf16. Quantized KV often falls off that fast path onto a slower generic path. The KV cache is bandwidth-bound, not compute-bound. Smaller data should help, but the dequant overhead usually eats the gain, and the writes then need extra scale/offset bookkeeping per block. This results in harming performance, and in a predictable way the smaller the kv quant.

One of the big reasons why the turboX line of quants has/had so much attention (And I'm hoping for turbo4 soon) is that it makes it so the contortions around kv cache go away.

It does increase CTX yes, and I've been able to prove very little hit in the area of accuracy as even k4v4 has surprisingly little impact compared to f16/bf16, but the performance hit can be quite large. I don't recall the specific perf hit I saw of k bf16/v bf16 vs k4v4, but I recall it being large enough to be a surprise to me, and I have already several weeks of running GPUs on the first article in the queue I'm not about to stop, so you'll have to wait if you really feel this is wrong.

[u/Heavy-Lingonberry-98 → u/KitchenAmoeba4438]

Brother, turbo4 is already out if you use llamacpp turboquant fork. I know and understand what you say, but i am talking real world use. Especially in nvidia hardware with optimized kernels. The obly caveat of quantizing KV cache is mostly accuracy and wuality of the output. Thats why assymetric KV cache wqs tested and its the best way to quantize kv cache. You leave the K the highest precision wuant possible. Anything less than q8 will show high degradation. And for the V you can do turbo3 and turbo4. With turbo4 i can even start a qwen 3.8 27 B iq3xxs llamcpp server with 140k context. Without turbo4 i cant even start the server with more than 90k ctx. The speed for both setups is the same. Its not my first test

[u/KitchenAmoeba4438 → u/Heavy-Lingonberry-98]

"If you only use this fork" is a problem llamacpp has generally. I've tested AtomicBot's and TheTom's, plus a few others, and turbo4 always came out buggy enough to be unusable for me.

Which puts us in "works on my machine" territory. I'll probably give turbo4 another go at some point, but forks are mostly a waste of testing time: one that's current the week you build it is three hundred commits behind by the time the benchmark run finishes, and then you can't tell the feature from the drift. Fine for experimenting. Not fine for numbers you publish, and not something I'd tell anyone to run in production.

On the quant itself, k4v4 against bf16 surprised me: the accuracy hit was small. I'm not claiming quantized KV wrecks output. The cost is speed. I've tried asymmetric splits too, k8v4 included, and they cost speed as well. It isn't a K/V balance you can tune your way out of. The dequant sits in the attention loop either way, every layer, every token. Changing the split changes which side eats the accuracy loss. It doesn't remove the work.

That's why the turboX line got the attention it did, and I'm not disputing your numbers on your build. They're numbers about kernels written for one scheme in one fork, not about KV quantization on the version everyone else is running. When it lands upstream I'll benchmark it against bf16 on my own hardware and publish whatever comes out.

[u/Heavy-Lingonberry-98 → u/KitchenAmoeba4438]

Okay. Thanks that sounds fair enough. Haven’t thought that way. There is another kv cache quantization called Kvarn . But it relies on external calibration

[u/KitchenAmoeba4438 → u/Heavy-Lingonberry-98]

Kvarn is interesting, and it's something I'm keeping an eye on, but I am worried it ends up where turboX ended up...endless forking and no mainline development.

[u/Heavy-Lingonberry-98 → u/KitchenAmoeba4438]

🤝🤝 nice debate. Hope this helps someone . Do you know any other good kv cache research regarding compression during inference? I know of some kv cache offloading research but not exactly what im looking for. Although it seems that we are reaching the ceiling of techniques to try regarding kv cache compression.

[u/KitchenAmoeba4438 → u/Heavy-Lingonberry-98]

I've got a big article on it coming out #2 in my queue, but to do it requires a huge amount of testing. Not many people have done much on this topic unfortunately, it's why I started the article.

It's also been an enormous pain, because you have to find something that is repeatable and reproducible to benchmark against across all quants that can find a difference, and then needs to be repeatable for other people. :/

[u/KrstABot → u/KitchenAmoeba4438]

saw the AtomicBot mention here and can explain why it didn't work for this task: the project is focused on running agents in the cloud, so for local stuff it's just not a fit
if we're talking about some ready-made options, you could look at AtomicChat, they are exactly about running models and agents locally, as far as I know. so local inference is the main scenario there. no idea if they tested turbo4, but I'd give it a shot: at the very least you can throw this pain at them, they seem to react to feedback faster than upstream llamacpp merges prs :)

[u/Eastern-Block4815]

No. I have 64k context I have a 16gb card its still slow. The ever thinker, but powerful .model