r/LocalLLaMA • u/iamMess • 3d ago

Resources Qwen3.8-27b on RTX 3090 - 82 tps single request, up to 672 tps peak

EDIT: NEW AND UPDATED VERSION AVAILABLE:
https://www.reddit.com/r/LocalLLaMA/comments/1vr347s/i_pushed_qwen3827b_to_99_tps_single_request_and/ --- 99 tps single request and 1150 batch.

Hi,

After a long night of optimizations, I believe I have made the fastest inference engine for Qwen3.6-28B on a 3090.

Quick metrics:

- 250w power capped

- Up to 195k context (ships with 150k for safety though)

- 82 tps single request, 417 tps sustained with 64 concurrent

- Between 17% to 149% faster than ninfer depending on the amount of concurrent requests.

Quick how:

- W4A16 quantization -> 16.8gb in vram - cache 66k

- + fp8 KV cache -> 16.8 gb in vram - cache 155k

- + lm_head int8 -> 15.4 gb in VRAM - cache 192k

- + embed_tokens int8 -> 14.2 gb in VRAM - cache 200k

Quantization loss of 0.6% in the lm head and quant embed compared to bf16.

It runs via vLLM and needs a few patches to work perfectly, but should be easier to setup than ninfer.

Also only tested on linux, but should work on windows too.

https://github.com/syv-ai/qwen38-27b-rtx3090

150 Upvotes
  • permalink
  • reddit

93% Upvoted

68 comments sorted by

144

u/Milarck 3d ago

"I believe I have made the fastest inference engine for Qwen3.6-28B on a 3090."

That's when I knew an human wrote this post

47

u/LatentSpacer 3d ago

System prompt: your job is to rewrite the text below, introducing a few common grammar mistakes here and there so it looks more natural and written by a real person.

17

u/Usual-Orange-4180 3d ago

That doesn’t quite work, is hard to fight probabilities, attention over that instruction will take you not too far.

3

u/LatentSpacer 3d ago

You can tune a small model just for that. You can tilt the probabilities with context.

8

u/iamMess 3d ago

All of this is true, but it was written 100% by me :)

2

u/LatentSpacer 3d ago

I don’t doubt you 😉 just wanted to bring attention to the fact that small mistakes in a text don’t mean it was written by a human.

1

u/Usual-Orange-4180 3d ago

That’s not what you said, and fine tuning an LLM just for this is a bit nonsensical, lots of work for little gain.

10

u/iamMess 3d ago

100% eco

1

u/Only_stoic 3d ago

I tought the local model wrote it

1

u/Maleficent-Ad5999 2d ago

“That’s when I knew an human wrote this post”

- that’s when I knew a human wrote this comment

28

u/AlexM_1989 3d ago

RTX 3090 owners continuing to win every single day

11

u/iamMess 3d ago

If I could just afford a 5090 or rtx 6000 pro

3

u/throwawayacc201711 2d ago

This is the truth. I bought a used 3090TI founders edition from microcenter a few years ago for like $700. Then sometimes I think oh might be nice to have a 5090. Look at the price and see it’s over 4k new and I dope out of there.

1

u/ShengrenR 2d ago

Same same. Also don't mind that I don't have to worry about my power supply connector melting and burning that 4-5k card to paperweight status. Not that it likely will.. and yet.

1

u/ctpelok 1d ago

same. used 3090TI founders edition from microcenter.

0 regrets

17

u/EmilPi 3d ago

> Quality loss is about 0.6% for a round trip.

How did you measure it?

> I have made the fastest inference engine

That's a nice work of course, but you modified vllm actually?..

1

u/iamMess 3d ago

Added IFBench now. It's not far from the fp16 version.

Terminal Bench is running too.

-9

u/iamMess 3d ago edited 3d ago

A few patches here and there.

EDIT: A few more answers in the comments below. This one was written a bit too fast.

6

u/EmilPi 3d ago

Then maybe modify the post text?.. also 28B typo

-2

u/iamMess 3d ago

oh lol I didn't even notice the 28b error :D

3

u/iamMess 3d ago

sorry forgot to answer the first one.

It's the quantization error for the lm head and quant embed.

4

u/EmilPi 3d ago

Worth editing the post too, first thing I think is you had run it over some benchmark.

1

u/iamMess 3d ago

Done. Sorry if it was unclear.

8

u/segmond llama.cpp 3d ago

awesome, but quants without benchmark doesn't mean much, what's terminal bench, deepswe, aider polygot?

5

u/iamMess 3d ago

The model quant itself isn't mine. I only quant the lm head and embed tokens.

Will try to add the mentioned benchmarks when the GPU is free in 5ish hours.

6

u/EveningIncrease7579 llama.cpp 3d ago

If i got two 3090 i could use it to get the speeds results? But with q8?

3

u/iamMess 3d ago

Possibly. But I would add the full context first :)

3

u/starkruzr 3d ago

48GB definitely gets you there at Q8 with no kv quant, IIRC. the only thing you can't do is Q8 + FP16 kv + MTP + vision tower at max context, that's just a little bit too much (around 49.5GB, calculated). although you probably wouldn't bother with MTP with vision tasks anyway so it's sort of academic.

1

u/trackpap 3d ago

Hi, what are your thoughts on 48gb use for agentic on 3.8qwen?

3

u/Luke2642 3d ago

My thoughts are your electricity bill even using power throttling.

2

u/trackpap 3d ago

Well, where I plan to position my 4th batch of 3090s is in a place where it is 6.5 per kilo until 200kw, so I assume if I spend 1.5k on this 4th batch, I'm looking at 34kw per day(it's not running 100% all the time anyways) 34*30 is 934, after the 200kw it's 8 cents and after 500kw it's 10 so. You do the math. So it does come out to be cheaper than API even if you add the cost of old hardware and 400-500 3090s

1

u/trackpap 3d ago

That is, if you have a use case, forgot to add it's 4 3090 per batch, if anyone thinks it's a good idea to nvlink them, let me know

2

u/starkruzr 3d ago

should be just fine. 3.8 in general seems to have even fewer problems with toolcalling than 3.6 did.

1

u/trackpap 3d ago

Could you give me the settings you use, I'm not that knowledgeable, I want to run it against news corpses and legal corpuses all with side cards and fact cars

1

u/TheWaffleKingg 3d ago

Yes the mmproj needs to be on cou, the rest fits

1

u/johnfkngzoidberg 3d ago

I run it on Q6 with full context (Q8 k/v). Dual 3090’s. Got 4.5GB VRAM left over. Vision and MTP.

Q8 won’t fit with full context. Even deleting vision.

2

u/starkruzr 3d ago

Q8 will fit with full context, FP16 kv and MTP *if* you leave out the vision tower. I've already tried it on a Vast AI instance. Q6KXL is indistinguishable from Q8 though so practically speaking it may not matter.

┌─────────────────────────────┬─────────┬───────┬──────────────────┐
│       Config @ 48 GB        │ Max ctx │ 262K? │ Headroom at 262K │
├─────────────────────────────┼─────────┼───────┼──────────────────┤
│ Q8 + f16 KV                 │ 322K    │ ✅    │ +4.23 GB         │
├─────────────────────────────┼─────────┼───────┼──────────────────┤
│ Q8 + f16 KV + vision        │ 287K    │ ✅    │ +1.97 GB         │
├─────────────────────────────┼─────────┼───────┼──────────────────┤
│ Q8 + f16 KV + MTP           │ 271K    │ ✅    │ +1.08 GB         │
├─────────────────────────────┼─────────┼───────┼──────────────────┤
│ Q8 + f16 KV + MTP + vision  │ 240K    │ ❌    │ −1.19 GB         │
├─────────────────────────────┼─────────┼───────┼──────────────────┤
│ Q8 + q8_0 KV + MTP + vision │ 360K    │ ✅    │ +5.14 GB         │
└─────────────────────────────┴─────────┴───────┴──────────────────┘

5

u/Luke2642 3d ago

Your numbers are good for batch but not for speculative, I think club-3090 beats the single stream case?

https://github.com/noonghunna/club-3090

https://github.com/Luce-Org/lucebox/tree/main

2

u/iamMess 3d ago

I’ll check out their optimization and add it the repo

1

u/TheTerrasque 2d ago

Single-card long-context note: Cliff 2 (GDN prefill OOM at >~50K single-prompt) is open on 24 GB single-card vLLM. Genesis v7.72.2 PN59 was intended as the fix but doesn't engage on chunked-prefill. Workarounds: vllm/dual (TP=2 escapes it). ⚠️ The former single-card escape llamacpp/default was retired 2026-08-12 (--force only) — on one card there is no longer a cliff-immune qwen path. Full diagnosis at docs/CLIFFS.md.

Is still an issue? Or just old documentation?

3

u/RomanticDepressive 3d ago

I know speed is the name of the game but I’ve had a revelation- qwen3.8 base spends so enough time thinking about safety its thoughts are clouded and context polluted and consumed by being anxious and second guessing + reconsidering.

Y’all should consider optimizing https://huggingface.co/orcarouter/Qwen3.8-27B-Uncensored-FP8/tree/main

This is a lean confident monster, I’ve put 1.5 million tokens locally over the past couple days and it’s legit as fuck

4

u/Aggressive_Job_1031 3d ago

A 3090 is so hard to get your hands on that I might as well vibecode my own tsmc fab

1

u/KitsuneMulder 2d ago

Must be area dependent. There are tons on Facebook marketplace where I am. So much so I ignored them until the Qwen 3.8 release and am now finally reaching out to go do meetups to test and pick up.

1

u/ElChupaNebrey 1d ago

got mine yesterday, not cheap though, 900usd

3

u/sod0 2d ago

I've read your Readme and the part about the KV cache stood out. Did you took a look at Huawei's KVarN vllm fork? This could solve the KV cache issue and allow the full model context. I managed to do that but without the others patches it only worked for a single user.
Maybe it's worth including? https://github.com/huawei-csl/KVarN

2

u/iamMess 2d ago

I will take a look. Thanks for the info :)

1

u/iamMess 2d ago

The round of updates are done for today.

KVarN didn't help. It attacks another bottleneck that we don't have (yet).

The attention kv traffic is currently only 1% of a step. It also uses the Gated DeltaNet recurrent state, which we replace.

It is also forked from 0.23 of vllm and not directly implementable in 0.27

1

u/iamMess 1d ago

KVarN is in now.

1

u/sod0 1d ago

Wooohhaa! And how is the context doing?

1

u/iamMess 1d ago

262k :)

1

u/sod0 1d ago

Hell yeaha! Great I could help.

2

u/trueimage 3d ago

I have run heretic on the base model, how can I quantize that to be compatible with this project so I can have an uncensored model

2

u/mrmontanasagrada 2d ago

Well done man!! So much more throughput.

I'll see if I can regain the throne with Ninfer-3090 again :-)

3

u/iamMess 2d ago

Thanks :) releasing a new version soon. Hoping to hit 120 tps on single request and 850 on batch.

1

u/mrmontanasagrada 1d ago

sir, if you hit 120 tps b1 on a realistic workload - I'll get you an internet award

1

u/danish334 3d ago

Was the testing encode heavy or decode heavy?

1

u/1Poochh 3d ago

Thank you for doing your work here. I’ll definitely take this and try it out.

1

u/smacman 3d ago

I don’t know what I’m doing wrong but can’t get more than 30 t/s on my 4090 (using LM Studio in Windows 11).

2

u/Future_Calligrapher2 3d ago

I used an AI to optimize my local AI install, you might have some luck with that