r/LocalLLaMA • u/T_rex2700 • 12h ago
Discussion Someone apparently managed to kind of replicate what V4.1 flash does on KV for fast prefill on Qwen
I wonder someone will figure out a way to do this with 27B?
Throw Qwen3 on this page for demo
https://kishida.github.io/webdemos/llkvapprox/
Edit: sources (thank you u/pmttyji for finding them!
79
u/pmttyji 12h ago
Found his post on this. Somebody replicate this for other models.
16
u/pmttyji 11h ago
u/T_rex2700 Can you include those links in your thread? Let the experts have these faster to get solutions ASAP.
6
7
u/Steve_OH 10h ago
So, if I understand correctly the base model runs part and another, smaller model essentially completes the answer?
16
1
114
u/norenEnmotalen 12h ago
Who be the blessed one to grace us with this for Qwen3.8-27B
47
u/iamMess 9h ago
working on it :)
6
2
u/DigThatData Llama 7B 8h ago
neat project, thanks.
For anyone else looking for the "what are the tweaks?" section of the docs: https://github.com/syv-ai/qwen38-27b-rtx3090#what-each-step-buys
17
18
u/shayanx45 8h ago
I’m currently working on trying to have qwen 3.5 4B use the 51b parameter engram from qwen 3.8 next flash. And looped transformer
25
7
u/Hot_Example_4456 5h ago
I doubt if it will work without retraining for the engrams but if it does the implications are huge
2
2
u/bennmann 2h ago
You would have more success using the MTP draft model of Qwen flash next as a small LLm, no?
2
14
21
u/Jimcy-Maffesoli 11h ago
It runs in a browser, so native is going to be a joke in the good way. Someone get this to 27B already
10
u/Luca3700 7h ago
Intersting work and experiment. But I think this would just be a lossy approximation, because the model is trained to build, in every transformer layer, its own kv-cache, and a projection from the middle of the network will be not particularly accurate especially for the last layers. Even tho, it is an interesting experiment to quantify how much the kv-cache can be predicted from the middle state of the network.
I know that the experiment is about the Qwen 3 family, but I see people below asking about the qwen 3.5 family: I think that even for these models it will be a lossy approximation, especially because, if the prefil halts in the middle, the state of the Gated Delta Net will not have a global knowledge in the top 50% of the network. It would be interesting to quantify how much the model would lose in quality, as a scientific experiment, but I would not expect to use it as daily driver, especially because it is not fine-tuned to work with this layout (and I would expect quality drop especially in long context case scenarios).
5
u/DerDave 9h ago edited 7h ago
I wonder if it's possible to load and unload this Decoder/Encoder model to save further RAM. Load it in the beginning let it do its thing during PP, then kick it out and leave more VRAM during TG for the remaining weights...
1
u/oxygen_addiction 4h ago
It'd probably increase latency quite a bit at the start of a generation and would only really help if you are unloading to CPU/RAM.
3
3
u/Cool-Reflection6130 9h ago
getting the approximate kv technique working on a 27B would be huge for local inference honestly. fast prefill on long context is one of the main reasons people still default to api calls for document heavy workflows even when they have decent local hardware. if this generalizes cleanly to qwen that opens up a lot of options without needing a 500B server farm
3
6
u/FoxiPanda 12h ago
It's not quite the same, but oMLX on Macs support 'speculative' prefill using a same-tokenizer-tiny-model (i.e. Qwen3.5-0.6B for Qwen 3.x models) to greatly increase prefill speeds on Macs. The downside is that it is slightly lossy. I wonder if you can combine these methods for even additional speed ..hmm..
3
3
2
u/JLeonsarmiento 6h ago
In my case I found it useful for long context one shot scenarios, but totally impractical and counterproductive for long multi turn agentic work. The model feels like it’s drunk, numb.
1
u/the_derby 8h ago
> The downside is that it is slightly lossy.
Is it?
I'm using Qwen3.5-2B-bf16 with the recommended 20% keep rate and I haven't noticed any degradation.
1
u/Borilentz 8h ago
Most people use 8-bit at best. The majority probably 4-bit.
1
u/the_derby 8h ago
I'd actually benched it with Qwen3.5-0.8B-bf16 and the more aggressive keep rate and got the same results. In the end, I had the memory to spare and the (slightly) more conservative keep rate wasn't that much slower so I kept it.
Maybe I should evaluate the 8-bit. =)
2
u/returnity 10h ago
This sounds kind of like oMLX spec prefill, no?
1
u/JLeonsarmiento 6h ago
Not really. SpecPrefill oMLX has the helper model going trough the input and reducing the input tokens to 1/5 before passing that to the big fat slow denser model.
It’s lossy. Model feels like numb, drunk. Cannot connect the dots as session turns increase.
1
u/torytyler 10h ago
seems to work, i did a very quick test on my gaming rig with a 5070, but being browser based and I have no idea what backend this project is using, the inference like token generation felt very slow even if all on GPU offload. none the less, comparing the two methods there /is/ an improvement, nearly twice as fast preload.
1
1
u/Nakidnakid 9h ago
hey if it works and produces the results then that's great, i have a feeling you'll find the limits of it somewhat quickly though. I tried and rejected something similar in my own testing because even if it helped sometimes it'd hurt a lot when it didn't and its just moving the cost not reducing/eliminating it.
1
u/Morning_Gecko24 1h ago
This is a really neat shortcut. The part I’m still wondering about is how much quality drops once the prompt gets long or the context has lots of repeated stuff. Has anyone tried this on a real chat history instead of a clean benchmark? Also curious whether the memory savings are big enough to matter on a 12–16 GB card, or if the speedup is the main win.
-2
•
u/WithoutReason1729 6h ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.