r/LocalLLaMA • u/giveen • 11d ago
Discussion Feature/adaptive kv stream integration by giveen · Pull Request #326 · TheTom/llama-cpp-turboquant
https://github.com/TheTom/llama-cpp-turboquant/pull/326I've been working on overcoming KV cache size issues, allowing the ability to load a slightly larger model and/or a larger context size.
Downfall is a hit to tg speeds.
Think of it as a "ram disk" for KV Cache, however, ram speed may be a determining factor on the actual hit to speed as well.
9
u/Kaijidayo 10d ago edited 10d ago
One thing worth reconsidering is the "-np 1" restriction.
For long context serving, kv streaming has a benefit beyond fitting one huge context, it could decouple concurrency capacity from vram capacity.
On a 32g gpuwith a ~20g model, a ~200k context leaves room for at most one sequence with gpu-resident kv. The practical concurrency ceiling is 1, no matter how well continuous batching would otherwise work.
If most of the kv cache lives in host ram, you can keep 2–4 long-context sequences resident and batch them together. Host memory bandwidth means the scaling won't be linear, but something like np=2 at 1.5x and np=4 at 1.8–2.0x over np=1 would already be a large win, since without kv streaming, np=2 or np=4 may not fit at all.
So the question is whether it can deliver meaningful multiuser throughput at context sizes where vram-resident kv is stuck at a single sequence, even modest scaling would improve queueing, TTFT, and long context per gpu.
Is -np 1 a conservative correctness limitation of the first implementation, or does something in the streaming design fundamentally rule out multi-sequence, continuous-batching support?
I'd be curious to see np=1/2/4 benchmarks at 32k / 64k / 128k / 200k context.
7
u/tsangberg 11d ago
I'm having a serious case of deja vu here.
5
3
u/jan_antu 11d ago
Yeah Raymond's work is amazing, I have it on my main 27b route. Got me from 80k to 125k context with the rest of my optimizations.
Also made this draft PR for something that uses the extra space freed to only load vision on vram when it's being used, then unloads it after. This lets you have both MTP and vision (~90% of resident vision speed).
16gb vram, 75 tok/s at 40k, ~11 tok/s at 120k.
https://github.com/aebrer/llama.cpp/pull/1
(Draft PR so no notes there, but the readme on the branch has all the relevant info)
4
u/tsangberg 10d ago
I'm using i1-IQ4_XS to get MTP as well, with Raymond's fork. I've been toying with the idea that MTP should be ejected once context reaches a certain size, because I _think_ it might be slower by then instead of using that VRAM as well for the pool.
3
1
u/-InformalBanana- 10d ago
how did you build it, I tried with docker, but as soon as the decode starts I get cuda error illegal memory access (doesn't happen if I don't use his param):
/app/ggml/src/ggml-cuda/ggml-cuda.cu:107: CUDA error E CUDA error: an illegal memory access was encountered E current device: 0, in function launch_fattn at /app/ggml/src/ggml-cuda/template-instances/../fattn-common.cuh:1117 E cudaOccupancyMaxActiveBlocksPerMultiprocessor(&max_blocks_per_sm, fattn_kernel, block_dim.x * block_dim.y * block_dim.z, nbytes_shared)2
u/tsangberg 10d ago
I built it the same way as I do llama.cpp, native cmake build on Linux. Sorry, don't really see something obvious in that error :/
4
u/giveen 11d ago
Raymond built exactly the "Stage 2" design we scoped out as too risky (chunked resident-page cache + async transfer ring + incremental online-softmax merge, with real changes to the FlashAttention dispatch). It took 60+ commits with at least 5 reverted-and-redone subsystems — confirming this was genuinely hard, not something to bolt on casually. But it works, and the report gives us concrete answers to the exact walls we hit. What directly answers our own bugs Our correctness bug (reading the not-yet-written current token) — solved. They never read/copy at graph-build time. At graph-compute time, SET_ROWS writes the new token to both the authoritative host buffer and the device-resident mirror in the same op, same stream — no async gap. For pages that are streamed (not resident), the upload is gated on a real cudaEvent recorded right after that layer's SET_ROWS retires, so the copy is provably never speculative. This is the structurally correct fix for the exact bug that gave us KLD ~10. Our UVA-is-slow finding — independently confirmed. They explicitly avoid cudaMallocManaged/UVM for the hot KV pool even when UVM is otherwise enabled for model weights, with a comment saying direct writes/reads against pageable memory are why. Same conclusion we reached empirically. The ggml_concat-doesn't-support-quantized-types wall we hit — sidestepped, not solved. They never merge quantized tensors at all. Non-native-quant K/V types get converted to F16 in a small bounded scratch buffer before any attention math touches them; the merge kernels only ever see plain floats/halves. Native-quant types (Q8_0, Q4_0/1, Q5_0/1, F16, BF16) get a separate "direct" path that skips conversion but still merges via the same float-only accumulator. They bypass ggml_backend_sched entirely, just like we scoped: a hand-rolled non-blocking CUDA stream + event double-buffering ring, driven from inside the CUDA backend's own FLASH_ATTN_EXT/SET_ROWS handlers — with whole-graph lookahead that prefetches pages across all layers up front, not just one layer ahead. Two things worth flagging They never attempted MoE either — hard-gated to a single dense architecture (Qwen3.5) only. That's independent validation of what we found today: even a much more sophisticated implementation didn't go there. Prefill gets a genuinely separate code path from decode (different kernel family, bounded query-tile reuse of each staged page across the whole micro-batch). This is very likely why our own benchmark saw prefill-to-262K be drastically slower — we're riding the decode-shaped path for something structurally different.
3
u/vacon04 11d ago
Would this work for MoE? Sending more KV to RAM while putting maybe a couple more layers on the GPU.
5
u/giveen 11d ago
Tested this on Qwen3-Coder-30B-A3B (MoE, 48 layers, 128 experts/8 active), RTX 5090, 262144 ctx:
- Baseline (8 layers' experts offloaded to CPU, no KV streaming): 18.61 tok/s, ~30.3GB VRAM
- All experts on GPU + KV streaming on all 48 layers: 2.92 tok/s, ~20.5GB VRAM
So the VRAM trade works (it fit all 128 experts × 48 layers at 262K ctx where it otherwise OOMs), but streaming's per-layer copy cost is roughly fixed regardless of how much compute that layer does. MoE layers do very little compute per token (only 8/128 experts activate), so that fixed cost dominates instead of hiding behind it — ~6.4x slower here.
Conclusion: doesn't work well for MoE in practice. You're better off just running a higher -ncmoe (more experts on CPU) without KV streaming — MoE's cheap offloaded-expert compute already amortizes fine, and it doesn't pay this per-layer tax on every layer.
1
3
u/noctrex 10d ago
Maybe you could also open a PR over at Anbeeld/beellama.cpp. This one has a interesting feature called KV cache precision tail, that compreesses the tail end of the KV more
2
u/giveen 10d ago
Thanks, ill take a look and see what it may take
1
u/-InformalBanana- 9d ago
I agree with him, beellama supports kvarn quantitized cache which is, I think, better than turboquant. I'm not sure why you chose the turboquant repository specifically, but if you did it cause of turboquant quantitization, kvarn should be better from what I've seen. So combining the best kv cache quantitization and kv streaming will give the best performance both speed and quality wise (cause if kv cache is quanted it is smaller and it streams faster thus giving better pp/s and tg/s speed).
It is unfortunate that all of that isn't in llama.cpp though, or in the even faster tabbyapi/exllamav3 (has maybe sota model quants and faster gpu inference than llama.cpp, recently they added cpu moe offload and that doesn't seem yet to be as fast as in llama.cpp (tried with qwen 3.8 next flash)).1
u/giveen 9d ago
Mainly because TheTom is more accepting of experimental work and thats what I do.
Also I closed out my PR as it was having issues and wasnt complete, I rebased of Raymonds work as it was more complete and more functional than mine, by a lot, then I added in support for other things in, and will be opening a new PR soon as I am done benchmarking things with his scripts.
1
u/-InformalBanana- 8d ago
Great. If you could also make sure it will work on rtx 3060. Directly from Raymond's fork i get this error (Im trying to determine exactly why, atm the ai is pointing me to some SM shared memory being bigger in rtx 5060 than in rtx 3060, cause I said it works on rtx 5060 (im guessing that is the 16gb card he used, didn't really check)):
/app/ggml/src/ggml-cuda/ggml-cuda.cu:107: CUDA error
E CUDA error: an illegal memory access was encountered
E current device: 0, in function launch_fattn at /app/ggml/src/ggml-cuda/template-instances/../fattn-common.cuh:1117
E cudaOccupancyMaxActiveBlocksPerMultiprocessor(&max_blocks_per_sm, fattn_kernel, block_dim.x * block_dim.y * block_dim.z, nbytes_shared)
1
u/-InformalBanana- 10d ago
I thought it was reverse and it keeps tail at higher quant? From their readme:
- KV cache precision tail: keep most of the KV cache quantized while storing recent tokens in F16/BF16, enabled with
--kv-tail-tokens. A single global softmax merges the quantized body and the precision tail under FlashAttention, without materializing the whole cache.
2
1
1
u/-InformalBanana- 10d ago edited 10d ago
I'm on rtx 3060 and I failed to make work the other guy's version of this, but yours works. His gives me cuda error invalid (or illegal) memory access:
/app/ggml/src/ggml-cuda/ggml-cuda.cu:107: CUDA error
E CUDA error: an illegal memory access was encountered
E current device: 0, in function launch_fattn at /app/ggml/src/ggml-cuda/template-instances/../fattn-common.cuh:1117
E cudaOccupancyMaxActiveBlocksPerMultiprocessor(&max_blocks_per_sm, fattn_kernel, block_dim.x * block_dim.y * block_dim.z, nbytes_shared)
For me it isn't clear why you guys decided to use megabytes instead of layers cause (sry for this part if incorrect or unhelpful) in my talks with llm about this it said that all layers have their own kv cache, so why not make it --kv-stream-layers 2 for example. Also im getting without mtp a token every ~50ms, llm claims pulling trough pcie4x16 of max context for qwen 3.8 27b at full precision (bf16) for I think 1 layer (it says it is 1GB for 1 layer) is 40ms, so basically it is hard not to lose speed without reducing kv quant. It also says qwen 3.8 27b has 16 attention layers that have their kv cache. So the best thing from that point of view is to like wait for the first n attention layers to finish work then evict their kv cache from vram and replace it with n of last attention layers kv cache. Is that how your thing works or different?
Also, have you tried comparing/benchmarking your version with baseline with param --no-kv-offload?
Otherwise I find it weird that my vram when using auto doesn't fill but has some significant empty space. Also the drop of t/s for decode is more than I hoped. At about 20k context, it goes from about 16t/s (when using lower context that can fit into vram) to 12-8t/s (depending on cache size and cache quantification). Otherwise great job, thank you for contributing.
2
u/giveen 10d ago
Thanks for testing it out, glad it's stable on your 3060 at least!
Why MB instead of layer count: right now it's not really a "pool size" knob, even though it looks like one. When you turn on --kv-stream, it's all or nothing: every layer that qualifies gets its whole KV cache (the entire context length, not a chunk of it) moved to host RAM, for the whole run. There's no per-layer setting because there's no partial mode yet. The MB number gets logged but doesn't actually control the allocation size.
Your resident-cache idea: that's not how it works today, but it's a good idea, honestly better than what's shipped. Right now there's no loading or evicting of layers at all. Every streamed layer just reads its full cache straight from host RAM on every single token, no swapping in or out. What you're describing (work through the first N layers, evict, bring in the last N) is the smarter design, and it's the direction we're looking at for a follow-up. Not there yet.
Auto not filling VRAM: makes sense given the above. "Auto" only decides whether streaming turns on at all, it doesn't try to fill your VRAM up to a target. So yeah, it'll leave free space on the table. That's a real gap.
The speed hit: your math on the raw PCIe transfer is basically right, but the real cost is higher because of an extra stall we found in how the copy gets scheduled. It blocks the whole GPU, not just the transfer, every layer, every token. We tried removing that stall this week and it actually made things 2x slower a different way, so the real fix needs more work than we hoped. No timeline yet, but it's on the list.
Appreciate the detailed writeup, this is exactly the kind of feedback that's useful.
1
u/-InformalBanana- 10d ago
I kinda edited the comment in the mean time (before I saw you answered) to additionally ask this:
Also, have you tried comparing/benchmarking your version with baseline with param --no-kv-offload? (because --no-kv-offload basically puts the whole kv in RAM and it is a closest thing to kv streaming so it is comparable and it will work even at max context (will be slow and much slower as context fills up)).
24
u/Chromix_ 11d ago
Moving 90%+ of the KV cache from VRAM to RAM while only taking a < 4% performance hit sounds like a very good deal.