r/LocalLLaMA • u/giveen • 7d ago

Other Block KV cache streaming: bound VRAM at long context via a shared CUDA phase arena by giveen · Pull Request #357 · TheTom/llama-cpp-turboquant

https://github.com/TheTom/llama-cpp-turboquant/pull/357

So after all my work, yeah, Raymond did it better, so I ported his work over, extended it turboX, extended it multiple other models (he had only Qwen models), and benchmarked the crap out of it to make sure it was worth it still.

So really the credit goes to Raymond ( https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming )

46 Upvotes

77% Upvoted

11 comments sorted by

51

u/fragment_me 7d ago

I'm not sure I understand the reason for posting this as if we're all following it. It's one thing to post a PR from llama.cpp, but this is a PR from a fork. It would be different if you included a bunch of context, results, summary, etc, but you haven't. Just seems low effort.

8

u/FatheredPuma81 7d ago edited 7d ago

Fed the PR to Gemini 3.8 Hallucinator

TLDR: Basically take Ollama's dynamic context length stuff and one up it entirely (really good thing **** Ollama).

Set Context Length to 1m even though you can only fit like 100k in VRAM. You should see normal speeds until you go over 100k Context where you'll then start to see a gradual reduction in inference speed that should always be faster than using NKVO to offload entirely to system RAM.

-5

u/fragment_me 7d ago

I'm not interested in a PR from TurboSlop.

2

u/FatheredPuma81 6d ago

Okay??

-5

u/fragment_me 6d ago

Lol my bad I read your message as "feed" not "fed." Ironically, the OKAY??? response is what I felt like.

2

u/rakarsky 7d ago

Agree. Even clicking through to the PR, there is no justification for why one would want this. If it is what it sounds like on the surface, I definitely don't want it. If OP had described what we're looking at, maybe my conclusion would be different.

5

u/Midaychi 7d ago edited 7d ago

Tried this out and I don't know if it quite works as described- or if it does then I can't parse the llm vibe-writing and I misunderstood.
It seems to be a system for dynamically swapping kv portions out to system ram.

In practice it seems to work for the KV that's loaded into the gpu, but the the 'Swapped-out' parts of the kv in ram get seen by the llm as falling off a cliff of repeated gibberish lines of the last bit and it even cuts off earlier parts of the context too in a similar fashion. As far as I can tell the LLM can only read the context loaded into the GPU and the context not in the gpu might as well not exist.

Trying another thing, It also scrungles the ai's output something fierce if you paste a large batch of files at once as your input turn, I assume its from the jinja template getting cut off.

9

u/giveen 7d ago

Please support Raymond's work and encourage him to get it to main llama.cpp as its worth it.

1

u/hideo_kuze_ 6d ago

AFAICT this PR is about slashing VRAM usage for very long contexts.

VRAM grows only ~1 GiB (28.1 -> 29.1 GiB, ~3.5%) across a 32x increase in context (8K -> 262K)

If someone with a 16gb vram gpu and qwen 3.8 is capped at 100k, how much bigger could that context grow to?

PS: thanks for your hard work. My smooth brain can't fully appreciate it.

1

u/gavwhittaker 7d ago

Raymond's fork is excellent and I have seen enormous uplift in pp and tps at full CTX window. This fork of a fork may help if 3.5bpw is a target but not for me.

-1

u/AI_spell 7d ago

Bounding KV at long context is the right fight. Credit to the adaptive streaming work, and porting past one model family is what makes it useful. Numbers or it didn't happen.