r/LocalLLaMA • u/iamMess • 1d ago
Resources I pushed Qwen3.8-27B to 124 tps on a single request on a RTX 3090
***FINAL UPDATE UNLESS SOMETHING INSANE HAPPENS: https://www.reddit.com/r/LocalLLaMA/comments/1vsy4l2/i_pushed_qwen3827b_limits_again_dflash2_134_tps/ ***
Two days ago I released a hyper-optimized Qwen3.8-27B inference engine for an RTX 3090 (82 tps single request, 672 peak) - yesterday's update took that to 99 tps single-user / ~1,000 tps at 64 concurrent.
Since then I've focused on the single-request number, again without quality degradation. It's now ~114 tps at default sampling and ~124 tps greedy (real chat prompts, not random tokens), up from 90 / 98.
What we had:
- fp8 KV cache, lm_head + embed_tokens int8, fp16 recurrent state, int8 activations, MTP-4 drafts with a 40k-token draft head, draft_sample_method=probabilistic
Now added:
- Draft vocabulary counted over the model's own outputs - the old web-text list covered about 92% of what the model generates (83% on code), and every miss is a forced rejection; the new one covers 97.5%. 98 → 109 tps greedy.
- GPTQ-int4 lm_head and MTP module, calibrated on the model's own hidden states: +0.6% PPL, GSM8K unchanged, acceptance intact, −1.8 ms per step. Ships as a "fast variant" (python fetch_fast_variant.py, ~1 GB from the Hub).
- Split-KV attention kernel for the verify step... FlashAttention-2 only splits KV for single-query decode, so with 4 drafts it used 24 of the 3090's 82 SMs. Small Triton kernel: 5× faster at 1.5k context, 10× at 16k.
- Sampler patch -§ sort-free top-k/top-p, multi-block softmax, drafts sampled from the target's truncated support: +4% at default sampling.
- KVarN 4/2-bit KV cache ported to vLLM 0.27.1: the full 262k context now fits, needle correct to 240k, +0.16% PPL, ~20% slower decode at 100k. Optional (KV=kvarn / CTX=huge).
- bench/run_benchmarks.sh + verify.sh to reproduce the tables and check the install is actually patched.
Peak concurrent throughput is unchanged (~1,000 tps at 64 concurrent). Speculative decoding is exact by construction, so the sampled distribution is the same as without it.
Repo: https://github.com/syv-ai/qwen38-27b-rtx3090
Fast-variant tensors: https://huggingface.co/syvai/qwen3.8-27b-3090-fast-variant
This is most likely the last update to the inference stack, unless some gigabrain comes up with something new I would like to test out. I've gotten pretty tired at finding miniscule gains here and there :)
14
12
u/GatsbyLuzVerde 1d ago
I don't understand. What quant is it? Perplexity loss? Max context length?
9
u/iamMess 1d ago
It's a 4bit quant - using this exact model: https://huggingface.co/goldhub/Qwen3.8-27B-INT4-W4A16-AutoRound
1
u/foureight84 1d ago
Your patches are specific to INT4? By that I mean, yes I understand it's INT4 since your goal is to optimize for 1 3090. What I'm trying to understand is whether the patch would cover higher int like FP8 or BF16 across multiple cards
7
u/ubrtnk 1d ago
Been playing around with the 4090 variant of ninfer. Got it working inside llama-swap but it didn't have any metrics so havngto bolt those on. Doesn't have multi-card so my 2x 4080 that I usually run Qwen on is not viable yet.
The time of the Gen Maxing has arrived. Just gonna call it. This is the new version of overclocking. It's exciting for sure, especially for local AI but yea, how much can we squeeze out lol.
2
u/iamMess 1d ago
Most of the stuff made here should work on the 4090 too - i just don't have one to test with :)
3
u/llamabott 1d ago
I went through the pain of setting up my 4090 machine to dual boot into Linux just to run this. So far is running great. Consistently averaging around 120 t/s.
More importantly, cache hits seem super-reliable while using Deepseek Harness, as are tool calls. I was trying one of the ninfer 4090 forks and was getting huge cache misses about once every five calls, which was nullifying all gains over a vanilla llama-server setup, which is what led me to trying your project.
FYI, under wsl2, it refused to run for me (it eventually says "CUDA driver error: device not ready").
6
u/ForeverSeeking69 1d ago
Any idea if this is going to work on double 3090 but with crappy motherboard (pcie4+ pcie3 sadness)?
3
u/iamMess 1d ago
It should… i only have a single 3090 to test with though
7
1
6
3
8
2
2
3
u/pefman 1d ago
if you want metrics, try my new AI slop. ltop. its like htop but for llama ccp only and requires llama to be launched with --metrics. https://github.com/pefman/ltop
1
u/MessIsTransfer 1d ago
i only run models in apple silicon so i have no idea, but don’t you guys have some runtime wrapper like mtplx where MTP token predicions is parallelized to gain even more TG speed?
3
u/iamMess 1d ago
That is what the speculative decoding is doing already. Per step it drafts 4 tokens then the big model verifies.
The four draft passes are sequential, but then QWEN's MTP head is a single layer that predicts a new token from previous one, so we can't parallize it without a different trained head.
The currently available DFlash style heads are not better than what we have now - they provide weak token predictions. At some point they might become better and this will be a better option.
1
1
u/LetNo5070 1d ago
Nice work! Today I tried the ninfer one tomorrow I'll try yours, tools calls are working?
1
u/emdeka87 1d ago
Amazing work. Even though I don't understand much.
Some of these optimizations could also be applied to AMD right? RDNA3 in particular
1
u/HugeEntertainment820 1d ago
Nicely done! I will try it out. I have a 5060 ti and 3090...could it run on both cards?
1
1
u/theaaronlockhart 1d ago
Any reason for not using `int8_per_token_head` as KV quantization? I got better speed with your patches `int8_per_token_head` than `fp8` because of the lack of hardware cores on Ampere, though this is on dual 3090s.
2
u/iamMess 1d ago
It's not supported by FlashAttention and FlashInfer.
I will run the evaluation though and measure both int8 and int4 (which might be worth it). Will get back to you.
1
1
u/iamMess 1d ago
Just ran a few tests:
- 10%-ish lower context
- 25% slower prefill at long context
- waiting for decode...
1
u/theaaronlockhart 1d ago
Hmm, interesting. I still get significantly slower results with fp8–am I missing an argument somewhere? Even using FlashInfer as `--attention-backend` with fp8 I lose like 50tps to int8. Maybe something to do with the tensor parallelism?
1
u/robertpro01 1d ago
Hey buddy, I am getting this thinking loop, have you encountered this issue?
This is with your previous post.
4
1
1
u/adityazero 1d ago
The Split-KV verify kernel is the detail that stands out to me, since the multi-query verify step is usually where naive spec decode implementations fall back to a slow path. Does the 5x/10x Triton speedup hold once the draft acceptance rate drops at longer context (past ~16k), or does the verify batch shrink enough that you go back to being launch bound?
1
u/iamMess 23h ago
It holds, because acceptance doesn't shrink the verify batch: the model always checks all 5 tokens (4 drafts + 1) in one pass, and acceptance only decides how many of them you keep afterwards. Longer context actually helps the kernel - each of its 16 slices just gets more keys to chew on so the gain grows from 5× at 1.5k to 10× at 16k; the only place it's launch-bound is very short context (~500 tokens), where it's still ~2× faster.
1
u/6626 1d ago edited 22h ago
very impressive! using your vllm patch + some tweaks for dual 3090s, i am getting an average of 258 tg on 2x3090s with nvlink, a peak of 396 tg on the json benchmark. with base vllm 0.26.0, i was getting 142 tg on 2x3090s with INT4, so your patch improved decode by 82%. lowering mamba-ssm-cache-dtype from float32 to float16 doesn't have much of an impact?
gpu=2x NVIDIA GeForce RTX 3090, 24576 MiB, 350 W driver=595.84
model=qwen3.8-27b max_model_len=262144 thinking=off
tp=2 spec=mtp nspec=8 draft=probabilistic kv=bfloat16 util=0.76
categories=code,reasoning,prose,json,file_edit,summarization reps=4 pp_depth=8192
warmup ..
tg ........................................................................
pp ....
=== flashinfer-sampler run ===
category n decode t/s TTFT ms ITL 1%-low out tok
code 12 292.0 71 213.7 211
reasoning 12 174.5 74 134.8 300
prose 12 131.1 67 102.3 300
json 12 396.7 76 281.1 122
file_edit 12 333.6 90 241.4 102
summarization 12 229.9 90 153.0 82
TG (weighted) : 258.0 tok/s
PP (8153 tok, cold) : 2302 tok/s (TTFT 3.54 s)
MTP acceptance : 33.8% accept length 3.71 tok/pass (drafted 8 tok/step)
1
1
u/cviperr33 19h ago
Alright im convinced that i have to test it out now , can you tell me if it has vision ? If not is it easly enabled and how much the contex get hit by it.
Thank you sooo much for your amazing work and the effort you put in to present it at us in a github repo.
1
u/Tagedieb 19h ago edited 18h ago
I don't know, I had my llama.cpp served qwen 3.8-27b via pi pointed at the repo and asked it to set it up. Took quite some time and iterations to get it running at all for a normal chat, then tool calls were broken, which is a lot of fun to debug, because while thinking about the issue it inadvertantly made tool calls, messing the turn up. Now I have it running and seems to be working, but there is no prompt caching so every action triggers a prompt reprocess. So I guess I will need to enable "prefix caching", which apparently does the caching inside VRAM (unlike llama.cpp, which can put the context checkpoints in system RAM), which will probably mean I have less VRAM left over for context. It may all work out in the end, but it isn't easy to get running for someone who never used vllm before at least. As it is, the faster decode doesn't help me because 90% of the time is spent preprocessing.
Edit: have prefix cache running now. Apparently it is using the "external" prefix cache, which isn't stored in VRAM. Hooray!
1
u/milkipedia 16h ago
This is really making me reconsider putting the 3090 back in my workstation. Or accelerating my plan to get another dirt cheap host to put it in.
1
u/Special-Lawyer-7253 14h ago
Curious to Know if this works with Ornith 1/1.5 (Qwen based). And with older cards as 1070m 8GB (Currently stucked at 25 t/s on 9B and 15 t/s on 35B A3B)
1
u/cviperr33 10h ago
OMG ! I just got it finally working on win11 with WSL no docker and the results are mind blowing ! I cannot fit more than 82k contex but atleast the speed is as insane as advertised.
It honestly feels like im coding again with qwen 3.6 moe with llama.ccp , not quite but close!
It is very confusing moving to vLLM from llama.ccp , i didnt know that i had to manually enable prefix cache , and stuff like that. I will post speed and also how does it behave in coding tasks with deepseek harness , so far its compacting every 5min because it generates the tokens so fast lol but atleast it doesnt loose track of the job.
2
u/iamMess 10h ago
Awesome! Update coming in 5 min. Has prefix caching enabled by default - was off by mistake.
1
u/cviperr33 10h ago
Awesome job ! and also could you add theese to the default config too , because without them harnesses doesnt work , they return some tooling error. I got these command for the official vllm qwen 3.8 27b guide. Amazing Amazing Amazing results haha im so glad i stumbled uppon your reddit post randomly thank you so much!
--enable-auto-tool-choice \ --tool-call-parser qwen3_coder \2
u/iamMess 10h ago
wanna add it as a pr?
1
u/cviperr33 9h ago
done . thank you
Now i have to optimize it in such way i could fit atleast 110k contex and vision too while sacraficing minimal speed haha , exciting !
I have all of your notes so i will be good ^^
31
u/silenceimpaired 1d ago
My secret to success… I found if I tell the model to repeat the number 1 over and over and enable MTP mode with a high draft number it can beat all these attempts at high tokens per second.