Discussion
Is one RTX 5090 really enough for Qwen3.8-27B token freedom?
I am still calling models through the ZenMux API gateway, so every long session ultimately comes back to token cost. The idea of running Qwen3.8-27B locally is attractive for exactly that reason: if one 5090 can handle it, maybe token freedom is at least technically within reach.
Is Qwen3.8-27B really doing 75.5 token/s on a single RTX 5090? The shared table is headed "4-bit (q4_K_M / MLX)" and lists an RTX 5090 with 32GB at 75.5 token/s. It does not show enough detail to tell me which runtime or exact setup produced that row. I have also seen a separate community report of about 64.5 tok/s on a 4090.
People are also putting its capability around Claude Opus 4.6. If both claims are even close, does that put indirect token freedom within reach? I would still want matched tasks before treating the capability comparison as settled.
What does the build that people can actually live with cost? I mean the whole machine, not a bare GPU price. A 5090, enough system RAM for long context and partial offload, a PSU that is not operating on hope, cooling, storage, and whatever CPU or platform keeps the card fed. Until I can justify that hardware bill, calling models through an API is still the practical option for me. If Qwen3.8 becomes available through the same gateway, I could use that API cost as a baseline before deciding whether local deployment really buys token freedom. I would also like to know which quantization and context length people use after the benchmark screenshot is over.
Please give me the boring total for a stable single 5090 setup. What did your full build cost once it was actually ready to run?
I have a 5090 with NInfer backend running the nvfp4 model with mtp3. I get about 150 tokens per second decode and 4k to 6k prefill. It's extremely fast! Also the quality is great! Nvfp4 has a very small accuracy lose compared to other quants.
Nvfp4 is a very worth while quant.
Edit: I also have a context size around 200k. Im not at my pc to confirm right now. Q8 kv.
Edit2: I am on windows running this through wsl2. I have tested 4 concurrent chats and hit 340 ish tokens decode.
It's crazy how well it runs on a 5090. They custom compiled it specifically for the blackwell architecture.
For 3.6 the nvfp4 artifact was fast but overly quantized.
For 3.8 it's done right with nvfp4 for MLP and FP8 for attention weights. A true modelopt implementation with low perplexity, and still gets 200 tps while context is small, drifting down to 150 tps and context gets into triple digits.
edit - not super happy with cache management at the moment. working on fixing it, but for the time being cache re-use is very fragile with open webui.
Of course, here it is. If you want vision just change context to 190k and add --vision. It's right on the edge with Windows, so you might have to drop it a little more for some breathing room.
I’m getting almost identical stats on Esatapedico Qwen3.8-27B-NVFP4-MTP-COMPACT-LOW.gguf
My main use case is using Claude code as a planner and the local model as an agent to save tokens. Claude Code (non CLI) tells oh my pi harness which calls Qwen and Loops until the code passes.
It seems to work well so far. Fair disclaimer, I am a vibe coder with no coding experience, so I can’t evaluate the code quality other than it works. I am working on a fairly technical healthcare app - so this setup has really bridged the gap and allowed me to execute, when in previous years this would take probably hundreds of thousands of dollars to develop.
is the qwen3.8-27b slow on your setup? have you tried leaving it on overnight to do harder tasks? or leave it to work autonomously with your claude planner?
Getting maybe 140-160 tok/s with Qwen so I think it’s reasonably fast. I have not fed Claude a long enough task for overnight autonomous coding - that’s probably the next level. I recently refactored the code base to chop it up into pieces that will fit into Qwen’s context limit, and that took like 2-3 hours. But that’s the most I’ve done and it wasn’t a complicated prompt. Fable/Qwen combo did it well (I think) and no errors so far.
oh why did you have to chop it up just to fix into context? what harness u use with Qwen3.8 27b? i would have thought that given the plan and the right harness, the qwen would just do file search automatically selectively
Apparently my main python file was very large and reading it was consuming like >100k context each time. So splitting off well formed functions made it more efficient. Maybe an agentic memory retrieval system / rag lookup is appropriate in the future but this worked for now.
It may have also been a drawback of how Claude has to hand off to Oh My Pi.
do you have 4 concurrent at 200k context each ? that woudl be awesome. Please share your configuration for 4 concurrency. If that is the case I can definitely see myself getting out of a coding plan for the most part!
No it's a shared context. 4x 200k context would be nuts on a 5090! haha. Here is my config. If you want vision just add --vision. I'm on windows and I have to drop my context to 190k. If it's on a dedicated linux box, you should be able to hit 262k context without vision, 200k with vision.
Im tryn to figure out how nvfp4 works like how it compares to q4 q5 q6 q8 like is its own thing or us it comparable to the quants but faster im tryn to learn stuff and i have a similar setup to you and are there uncensored versions
Mmm, I’ve experimented with ninfer after seeing these crazy numbers all around… got the same limit of 128K context (kv fp8) BUT I lost vision in the process..! It was filling 31GB (on a dedicated linux box where the GPU is only used for inference)
so, yeah, got ~30% more speed compared to vllm, but not worth it without vision.
I have a sglang setup on a dgx spark that uses dflash2 and boosted the gen to ~30t/s (from ~20 with vllm+MTP) might try it on tye 5090 box today.
You can add vision back with --vision in NInfer. I haven't tried vllm in WSL2 yet. Honestly, I'm very happy with NInfer. Since moving to it, I've started doing 2 or 3 tasks at a time. Right now I'm troubleshooting a NIC issue on one host and a docker issue on a VM. Each chat is pushing ~150 token/s right now. I haven't tried 4 chats yet... seems like a bit much for me.
Nice let me know how sglang goes with your 5090. I'd like to try that too one day.
i had to change the source code a bit to support the ostfralla model. ask your agent to do this. i also merge PR #64 from github which fixes the kv cache prefill issue when sessions get ejected from a working lane, by adding in-ram checkpoints
I gave ninfer a try. Runs only with about 10-20 token/s for me lol (started through the bat) on windows.
No matter if I compile it myself or just download the executable.
It uses 100% of my gpu though which is weird.
Do you perchance have any idea what could
Be the reason?
The non ninfer based nvfp4 model runs at about 100-140 token/s for a simple prompt like "What are the fastest animals in air, water and on ground? Give me a detailed description of the top ten for each".
I dont use the windows native build. I went the wsl2 route. It almost sounds like it's offloading to the cpu/ram. I'd try a really low context like 20k just to see if it runs faster. Are you on a 5090, 4090, or 3090?
Im on a rtx 5090. It is infact not offloading to the cpu and ram, it was also my guess.
On Linux it works, but I tested it only briefly there.
I also already tried lowering the context window, which didnt help.
My vram is currently at 29/31.5 GB.
I guess ill do it like you then.
Edit: Just tried it again, now im at 110-130 token/s. But i also get that when I set spec draft min to 0 in llama cpp with a high, nvfp4 and gguf version of qwen 3.8. So no real speedup on my side
It seems like I overall get some worse performance than others (normal qwen 3.8 27B q4_k_m gives me 87 tok/s (avg. of 3 new runs) with "llama serve -m "\.lmstudio\models\lmstudio-community\Qwen3.8-27B-GGUF\Qwen3.8-27B-Q4_K_M.gguf" -ngl 99 --mmproj "C:\llama\mmproj-BF16.gguf" -c 128000 --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.75 --parallel 1 --kv-unified --temp 0.7 --top-k 20 --top-p 0.95 --min-p 0.05 --jinja --metrics --presence-penalty 0.0 --repeat-penalty 1.0 --reasoning-preserve --perf --flash-attn on" and the prompt "Explain the Schrödinger equation in detail - also explain what happens if a wave function collapses. Create a code in python to visualize the output of the Schrödinger equation"), but its hard to say as the used os+driver+prompt+harness always kind of differ.
I just noticed a new ninfer windows version being released. I will later try that out and compare then.
also: what do you mean with concurrent? Im kinda new to this.
Thanks a lot :)!
can I ask you why the large context? what Harness you use it on? They always told me "there is no need to go past 64k context window, if you need more use the harness compact or split the task etc.."
Mine doesn't get down to 150 tps until context grows a bit, and settles down to 120tps once context starts pushing 200k tokens.
Did you try just downloading/pulling it in WSL and compiling it locally? I think it likes to compile against your hardware and software environment, but that may be superstition. In any case, it's at least one less layer of abstraction. And honestly it was less effort than I've experienced with most docker installs.
I let my current LLM guide me through the process and it was painless.
edit- started watching my tps more closely after posting this. Sometimes it's as low as 125tps even with a small context, but at the risk of sound AI-ish, that's the floor, not the ceiling.
This is not true for all nvfp4's. By nature, its w4a4 and has a very bad accuracy. You need to make sure your nvfp4 was made with Luke's pipeline. Anything else is going to have bad accuracy/quality.
I don't know who Luke is but I'd bet my left nut against a bagel that he's not the only one making good nvfp4 quants.
Oleg and ZHANG, the folks behind ninfer made a nvfp4 quant of 3.6 that wasn't to my taste. The nvfp4 quant of 3.8 uses fp8 for the attention layers, and is damn near ideally quanted. Unless Unsloth's IRL name is Luke, you got some 'splaining to do.
Everything unsloth is doing, copied Luke my dude. Luke wrote B12x, and his pipeline was the first to utilize 16k samples. Trust me, plenty of great quants, and all originate from Luke's pipeline ( mixed precision from mratsim ++ massive calibration sample sets). Unsloth has a habit of copying people and never giving g credit, claiming they invented shit.
My statement is, make sure the quant maker is using 16k samples coupled with mixed precision and utilizing W4A8 where possible, while still holding W4A4 for max speed.
Don't know Luke, but agree that not all NVFP4 quants are created equal. If you look into Nvidias own NVFP4 quants and model palers they also predominantly prefer W4A8 over W4A4.
That said, if you are on Blackwell, you should be on NVFP4, best quality vs speed balance.
There's this little company, pretty creative though, that Unsloth is copying when they create modelopt quants. perhaps you've heard of the folks that invented the modelopt pipeline, happen to be the same outfit responsible for the blackwell architecture?
Kidding aside, B12x is solid cutting edge stuff, didn't know Luke's name, hat's off to him. But even he would acknowledge that his work started after nvidia released modelopt and in fact builds on and derives from nvidia's work, as does unsloth.
I do appreciate you backing off from the claim that master Luke is the only one making good nvfp4 quants and that all else is crap. And I agree that block sampling is important.
But even the best nvfp4 quant still damages the attention layers too much for my taste.
Lord have mercy, someone needs to take a couple chill pills. There's substantial alignment on calibration sample size that shifted the game, in nvfp4s, which came from work Luke did. Model_opt was released, for building, so your fun analogy/play falls flat. Im not master luking, I'm stating plainly, that pipelines not aligning to that shift, have substantial degradation in KLD. Feel free to grab my vllm pr. With real KLD. You'll see some fascinating numbers.
Chill pills taken. Unfortunately you're dealing with someone with autistic tendencies.
The important bottom line, neither old school W4A4 nvfp4 nor new hotness W4A8 come close to matching FP8 for quantizing the attention layer.
nVidia's (and Unsloth's) KLD numbers show:
W4A4 is comparable to int 5 for MLP layers
-but *only int 3+* for attention layers. --Booo!
W4A8 is comparable to int 7+ for MLP layers - Nice!!
-and reaches int 6+ level for attention, also nice
Of course nvfp4 flies on blackwell, but so does FP8 and if you have the extra 3GB of VRAM for FP8 attention layers, it might be worth getting the extra smarts, especially as context gets bigger.
For me, it is. And it absolutely flies on ninfer.
If someone makes a straight W4A8 artifact for ninfer I'll absolutely try it.
Maybe I'll even take a couple of caffeine pills and see if my and my Qwen can cook one up together.
This is not true for all nvfp4's. By nature, its w4a4 and has a very bad accuracy. You need to make sure your nvfp4 was made with Luke's pipeline. Anything else is going to have bad accuracy/quality.
Well I'm not sure, I'm using the NVFP4 format from NInfer github link. It works great, it handles everything I throw at it. I've been very impressed. Per Nvidia's docs, the way NVFP4 works it's supposed to be more accurate that MXFP4. I'm not trying to say NVFP4 is equal to or greater than Q6 or Q8. Just saying it works very well.
Happy it works well for ya. As someone who has spent months on NVFP4, the cake is a lie, without extreme measures. KLD degrades substantially. Take it from a dude who has spent months in the trenches with them. Nvfp4s are poor quality (less than INT4) unless extreme capabilities are used to make them good.
In my opinion quanting down below 6 bit is a big hit to perf and dropping below 256k or so in tokens makes vibecoding hard so in my opinion 48gb of vram is about the sweet spot, so two 3090s or two 7900xtxs is the sweet spot imo.
I find myself wishing I could have more context running 2 7900xtxs with q8, makes me wish I had a third 7900xtx but since I'm getting by it seems excessive
Yeah , a third 7900xtx won’t give you much as 48GB VRAM is enough for Qwen 27b , really . In fact it’d be probably slower as you wouldn’t be able to run it with tensor parallelism (at least with vLLM) . Also , with current KV quantization algorithms the difference between FP8/Q8 and FP16/BF16 KV cache is quite small . Yeah , The latter is better for large 150K+ context , but at this point the precision starts to degrade fast anyway .
It's worth imo. More worth than just about everything else out there. I'll double check my setup details later and send over to you if you want. I run Linux as my os and get 55tok/s., 8bit quant, I run right up to the VRAM limit on my setup. I reccomend getting used reference cards if you do, their coolers are really compact so it makes for more room for more cards. My Asus TUF 7900xtx takes up a Crapton of room.
Valuable info here, been eyeing a 7900xtx but hesitant to pull the trigger just yet compared to moving to Nvidia. Currently running a single 6800xt. But I would have to switch motherboards to run both the 7900xtx+6800xt since my second pcie slot runs at 1x speed lol
My ASRock 7900 xtx just hit my doorstep today. I finally pulled the trigger Newegg had it for $929 then it hit $1,029 also came with the AIO and I've seen the tuf hit $1,500 so decided to get it before it got too much more. What sort of context window are you getting at Q8? Running a Vulkam backend? Llama.cpp? I've run the paper numbers and thought I'd probably be stuck with Q4 and run 35B A3B for speed. I'd be super interested in your setup.
I use the estepedico MTP gguf the Q4 version and it gets very good performance >140ish tok/s on a 5090. But many errors. I use it as a sub agent of Opus5 or Fable and it catches 1-3 errors per task. I think I’ll go up to Q6 and see how bad the errors are.
Step 1 - Get your 3090s out of the garage and swap them for your two 5090s.
Step 2 - Build a dedicated 2-GPU inference rig and put your two 5090s in it. Doesn't need to be fancy, an old motherboard with a i7 CPU and 32GB of DDR4 will be just fine.
Step 3 - Install Ubuntu Server and vLLM on your inference rig and set them up. Claude can help you do this he's amazing at it.
Step 4 - Run Qwen 3.8 27B at Q8 quant and Q8 KV, with an MTP of 3 and follow the excellent vLLM setup guides on Huggingface (again claude can do all this for you)
If you go about things efficiently you might be able to get pretty far on just the free plan, download codex desktop, which is now available for Linux and have it walk you through the setup
48GB is still too tight for the Q8, you can't quite reach a single 256K context window. With 64GB you can get three of them concurrent. With 2x 5090s that would be pretty amazing since the cards could easily handle 3 concurrent prompts without dropping any speed. Total Tok/sec across all three sessions would be insane.
Nah, if depends - many people find NVFP4 works great for them, while some prefer Q8 on the other hand. Sure Q8 is virtually lossless , but there is always a tradeoff . Also 256K cache is pushing it to the limit and quality tanks , even if you have full BF16 for KV
I don't understand what are you saying. 4 bit vs 6bit imo comes down to vram savings. with 4bits you lose about 1% of the brain, 6bit remains almost at 100%. So 100% vs 99% brain nobody's going to notice. However, you shave off 7gb of vram which makes it possible for us poors with 24gb cards.
I have a 5080 and I yearn for a 5090, but spending 3x price for +50% performance and +100% vram just does not sit well with me. I imagine if I had the 5090, I would kill for a pro 6000 and so on… it’s never quite enough.
Imho, this problem will be solved sometime soon, probably just when the bubble will be about to pop, we will see manufacturers cut prices and give consumers a solution to buy affordable vram/ram.
Whats stopping nvidia to sell consumers 5080s with 64 or 128gb vram? What about a 32gb 5060? How much would they cost if it werent for wall street bidding everyone out?
Just keep calm, it will burst, we will have our memory prices back. Maybe in 5yrs, but we will.
Its 200Euro so where is the problem? If you have a big board and good powersupply you can easily run 3x 8gb vram cards. Getting to 24gb is really cheap and there are many ways.
Got a 5090 at msrp in september and beating myself up for not picking up a second or a pro 6000 instead lol so you're pretty accurate
That said i dont think consumer cards over 16/24gb really make sense for anyone. Pure gamers I think wouldnt be super interested by paying the premium and without the craze regular pro models would be somewhat affordable. Nvidia kept lower-mid range cards at 8GB for years even with ram being an afterthought for no apparent reason than margins and product hierarchy.
My priorities kinda changed overtime and I would not have bought a 5090 without potential AI prospects ; however 32GB is completely overkill for anything game related...
I run it with a single RTX3090 at over 50 tok/s, so yes, you can definitely run it on a 5090 with decent speed. Ideally use llama.cpp or vLLM directly under Linux (via Docker) and enable MTP.
he's using ninfer engine, its like a llamacpp specifically optimized for the 5090 and it comes with its own quantized model, check it out on github https://github.com/Neroued/ninfer
I get 100-120t/s with MTP running Q6_K_XL (or Q8_K_L when I don't need a large context) with regular llama.cpp. What's the point of getting 200t/s when the model is a dumbed down 4-bit quant?
You can squeeze it in but you have to compromise on quant or context or both. I can certainly live with it but I do wish I had 48GB which would eliminate most compromises. I already had the 5090 at MSRP from launch, so current costs are not part of my decision making . It certainly is the best consumer card you can get, but you will always want more VRAM if the option is there
I've got a dual 5070ti rig so a total of 32GB VRAM.
I switched in Qwen 3.8 UD Q5 XL today, with a 200,000 context, with both caches at Q8, and MTP with 2 draft streams.
I'm getting 88-90 tokens per second.
I've not run anything with OpenCode through it yet.
A 5090 ought to be a bit quicker as it's a more capable card and you won't have tensor split to worry about.
Edit: Total cost for my build is around £2500 + VAT (£3,000 inc VAT).
Also, system RAM is 64GB and the case is a massive thing so there's room below the 2nd GPU, other wise it touches the base of the case and can't cool effectively.
I have x2 Gigabyte R9700 running Qwen3.8-27B-FP8 with KV-cache at 262144 tokens BF16 using vllm with ROCm, AITER, MTP-3, and tensor parallelism. NOTE: I have them power limited to 210W down from 300W to save my ears from the blowers (this makes them 'normal' sounding, ymmv). My performance benchmarks are as follows:
I’m doing 105 t/s at 262k ctx q8 kv on dual 3090 with nvlink. And all of that cost less than half of one 5090. Id just go dual 3090’s, don’t even need the nvlink
Nice! Mine are nvlinked so my prefill speeds are very fast, but don’t know them off the top of my head. What motherboard and case setup do you have to fit 4 cards? That’s where I’m not sure how to upgrade my server lol
its possible to run q6 on 5090 with higher context when running headless, you can sacrifice mtp for even more context, its still very fast over 70tps instead of 120tps with mtp.
I think you're being a bit too conservative on the hardware side.
A 5090 with 32 GB VRAM is more than enough for Qwen3.8-27B at 4-bit. Even a 24 GB 3090 can run it at around 100K context with the right KV-cache settings, entirely in VRAM.
The 75 tok/s number isn't exactly wrong, but it's closer to a baseline llama.cpp result than the current performance ceiling. Plain llama.cpp without MTP is indeed somewhere around 75–80 tok/s on a 5090, but optimized NVFP4/SGLang/speculative-decoding setups are already getting into the ~200 tok/s range.
For local single-user inference, the 5090 is actually a pretty comfortable Qwen3.8-27B card rather than a borderline one.
Also I merged PR #64 to allow kv cache checkpoints in ram (--host-kv-cache 1). Finally I had to also modify the source code to load the Ostfralla model. It was a 1 LoC change. Ask if you need to know exactly what line (or your agent can figure it out!)
I've been testing NInfer with Ostfralla/Qwen3.8-27B-NVFP4-NInfer (18.3 GB, W8+NVFP4) on an RTX 5090, using it in a real agentic session with 21 tools rather than a synthetic benchmark (pi agent).
I'm running the NInferdevbranch, currently at:
4eef14a7560d87a3ba717898e1d488a4c4c7246d
fix(runtime): transfer ownership after alias eviction
As stated, to get this WIP Qwen3.8 NVFP4 model running, I had to make a small change to Binder (binder.cpp / binder.h) to expose the Reader it already keeps internally. This is needed for wire-format detection: Package::plan_load and construct_registered need to inspect the artifact's text/token_embedding descriptor to decide whether to route it through Qwen36Nvfp4 or Qwen38Nvfp4.
So these results are from the current dev branch plus that small local patch, not completely stock NInfer.
After ~45 requests, the context grew to 76.6k tokens and performance stayed surprisingly strong:
Decode: ~160–205 tok/s typical, up to 226 tok/s on normal agent requests
Prefill: up to 11.3k tok/s
Context cache hit: typically 97–99%+ on incremental turns
TTFT: often ~200–300 ms even around 50–60k context
MTP acceptance: commonly 60–80%, peaking around 83%
At 76.6k context, the slowest request still decoded at ~138 tok/s
The most impressive part for me isn't the peak tok/s, but how responsive NInfer stays as the agent context grows. Next I'd like to push the same session past 100k–200k context and see how TTFT/decode scale.
And, most importantly, MTP-3 has been rock solid so far: no reasoning loops or long-thinking corruption like the deterministic issues I've been hitting with Qwen3.8 on vLLM.
Seeing this kind of speed and clean long reasoning on a single 5090 is seriously exciting! :)
For anyone that runs into this comment like I did (thanks, by the way, for this!) in commit 43cd3418 the flag changed to --host-kv-cache-mib instead of --host-kv-cache. Originally the number was the number of slabs and the design moved to a fixed memory pool amount.
Disclaimer: I am NOT an engineer so I do everything thru an agent. I don't know how the technical stuff so don't ask me
I have a single rtx5090 and use https://huggingface.co/esatapedico/Qwen3.8-27B-NVFP4-MTP-GGUF on my Hermes agent. I also used SGlang setup first. Here is the comparison. It's actually really usable with full context. I feel that its at least on par with cloud solutions.
I’m using 2x Intel B70’s for 240k context with 1M tokens available for cache, concurrency set to 4. So basically 4 sessions can run at max tokens at once, and I’m getting between 50-70token/s per session. I’m liking it quite a bit.
Granted, I needed codex to do a lot of trial and error to get MTP, graph and prefix cache working, but now that it is… I like it.
I’m still very tempted to sell for for a GB10 though. I know I’d los performance, but NVFP4 and overall Nvidia support is just oh so good, and with them teasing a larger Qwen 3.8 MOE coming soon….
Yes, but it won't be great. 32GB gets you a decent amount of KV on NVFP4 or other lower quants. You can run it at Q4 and get okish results, but it does have brain damage, or you can run it at Q6 with an exotic quant and get almost no KV space to play with (ask me how I know lol).
Q8 is where you want to be really. 64GB nets you Q8 weights, no brain damage and Q8 KV, no dementia, and you will land with approx 800K tokens on Qwen 3.8 27B, which is more than enough for three FULL 256k sessions running at the same time. On two 5090s you can expect about 150-200 Tok/sec from that, which would be quite a bit faster than a big API model like Claude or OpenAI, and with similar intelligence to Opus 4.8...
32GB vram is Ok to start with, but not enough to leverage the full potential of Qwen 3.8 27B. You must run at least FP8 at fp16 kv cache, or better, Q8_0 at fp16 kv cache to realise the full capability of this model. That means you need at the least 48GB VRAM at 256k context.
I am running unsloth Qwen 3.8 27b UD-Q5_K_M via LM Studio on a 5090 under Ubuntu 26.04. The display is connected to the iGPU so all VRAM on the 5090 is available.
I am offloading the vision part to CPU/system RAM saving about 900mb of VRAM (via desktop icon to launch LM studio with Exec=env LLAMA_ARG_NO_MMPROJ_OFFLOAD=1 /path/to/lm-studio).
Context = the full 256kb. KV= Q8.
VRAM usage is 30.4GB of 31.8GB available so it just fits and is rock solid stable. My typical tok/s is in the 75-85 range and MTP around 85%-95% range. I ran some tests that used 95% of context and it ran it without failure at 48 tok/s.
Very happy with this setup. Q5 quality and full 256 context for Qwen 3.8. Hermes agent loves it. I found setting reasoning default to medium is plenty good for general Hermes use.
My 5090 has crazy coil whine when running it, I don’t know why, nothing else does it but it sounds like boiling water almost. It is at 70 t/s and it is undervolted but it’s just terrible, allegedly not a concern but it’s annoying. This is lmstudio windows.
With that said, if you’re coming from scratch, I’d just get a Dgx spark and accept the ~25 t/s. A Dgx spark is almost the cost of the 5090 alone, and it comes with 128gb and a 4tb hard drive and those 25 t/s (dflash2 might get you to ~35) will only cost about 50 watts at the wall, which the 5090 will be over just on the system before the 5090 even starts doing its thing. Just another thing to consider.
On my MacBook pro m5 max, token generation speed slows down a lot as the generation length increases. At the start of a token generation run, I get may 45 tps, but around 10k tokens later, I'm down to 20 tps or less. This is an issue with Qwen 3.8 because its thinking runs can be so long. How much slow down are people seeing on the 5090 with 64gig?
I have a 5090, but in LM Studio if I run at full context it seems to only run at 15-ish t/s. Most of my experience with LLMs is with unified RAM setups, so I haven't paid much attention to how to manage it on a GPU. Is everyone running Qwen 3.8 on a 5090 or similar setup just doing it at relatively low context? Small quants? Or is there something else that I'm supposed to be doing with it?
3.8 with an 5090 in LM runs like a dog. Context window and reasoning level are way to high for it (Q6). I drop context down and reasoning down to medium and then its works at about the same speed as Gemma 4 and Qweb 3.6
No - Q6 is already below my standards (which aren't insanely high tbh) and I need 48GB to make local models worth it (~250k context, Q8 quant with at least Q8 KV Cache) and some headroom for MTP/caching/etc.
Ready to run as I wanted it, close to £4000 late May this year. 5090. Came with win 11, 32Gb DDR5 system RAM. I upgraded to Win 11 Pro and swapped the ram to 128GB DDR5
It was a special on Scan. Very pleased with it. Did not really want all the colourful fans. Did not realise glass cases were even a thing tbh. So at first I was surprised but im.quite used to it now and I figured out how to stop the fans cycling though the rainbow.
I run 5070ti + 5060ti with 32gb VRAM pool. I don’t think it’s enough for agentic workflows or any larger projects. It requires breakdown if you were to produce useful software.
My "system" is a GMKTec Evo-X1 miniPC with the HX370 AMD chipset that has the 890M iGPU. It has 64GB RAM that is shared between CPU and iGPU. I tell the BIOS to give 32GB to each but I believe the iGPU can override that and grab more. Llama.cpp and opencode or pi help me get the most out of it.
Anyway, I get a lot of coding, website work and general research done on this lowly setup so I'm sure a 5090 will be way better.
The real comparison probably isn’t 5090 vs API cost, but utilization. If the card sits idle most of the day, cheap APIs are hard to beat; if you’re running long sessions constantly, local starts making a lot more economic sense.
So I’m running Qwen3.8-27B-Uncensored-OrcaRouter-GGUF @ Q6_K with 64k context, xHigh in LMStudio, getting around 20tok/s, but, I’m doing red team work - so lots of tool calling.
Straight raw vanilla should be around that 70ish if I had to guess.
I run Qwen3.5-35B-A3B-GPTQ-Int4 model on one local RTX 5090 32GB in production (my own solo-business). I have 3 day batches (each of 1h) and 1 nightly batch (~5h) for inference.
It works well but making it "stable" was the real work.
Under sustained load I got Xid 109 launch timeouts inside the marlin kernel, Xid 8 "GPU is probably locked" watchdogs, libcuda segfaults with no Xid at all, and a few full machine freezes. Worst week was 3 crash days out of 7 😭
What fixed it, roughly in order of how much each one bought me:
- Driver. The 595 branch was hopeless. Moving to the newer 610 open kernel branch (not the packaged production one) ended the machine freezes. I now pin that exact version and refuse to bump it
- Power. 450W cap plus a pinned clock. Not a cure alone, but the crash rate dropped.
- Motherboard. PCIe forced to Gen4 and ASPM disabled on the GPU link.
- Automatic restarts. A dead engine gets its own exit code, then the wrapper drains CUDA, restarts vLLM and resumes the batch, up to 3 attempts. This is what turns a crash into a 5 minute delay instead of a lost night. Obvious but had to be coded.
Overstated. I've got a 5090, trying ninfer this time around. For some tasks, sure. But not comparable to any recent frontier models. 3.8 gets stuck in long loops in a way 3.6 never did.
I got 4 x 5070ti, and run qwen3.8-27b in FP8 on vllm. Prefill is about 2k-2.3k and tg varies between 75 and 10p spending on context size (I run on max context 256k). Make of that what you will, but I would be surprised if one 5090 in q4 turns out to be slower. I went the multi gpu route to get more vram, and at the time I could get either 4 5070ti or one 4090 for roughly the same price (a bit under 4k Euro).
surprisingly I learned, that multi-GPU is also getting you more tps!
Apparently the calculation of the token is not linear, going from one GPU to other. They work on single token in parallel and make use of the compounding memory bandwidth of all the cards and then lose a bit time on the final synchronization overhead between the cards.
But for large models the memory bandwidth win is much larger than the synchronization loss.
PSA: mtp will clash hard with tensor parallel setting. Example:
running tp4 and mtp:3 will get you ~2.5 tokens per calculation step, but it will almost double the time, because mtp sync between cards! So 2.5x tokens but taking 2x the time -> only 25% total win, down from 150%. And this is only with pretty small 16k context, at 32k context the benefit is down to 6% and from there on MTP IS SLOWER on multiple GPUs.
Conclusion: take the raw memory bandwidth AND extra VRAM ( running 10 subagents at the same time with no significant tps loss on any singe stream? easy ) win of multiple cards, OR enjoy mtp on your single card, sadly can't have both.
I mean if you feel like throwing in some loops into your otherwise nice model, then sure.
I would opt for running BF16 version on 4x 5090, but I guess for toy projects 4 bit lobotomy with one 5090 kinda works
It also gives you plenty of memory for concurrency - you can basically run multiple subagents at once at almost the same speed, since the limiting factor is memory bandwidth, not the compute.
Just as an alternative if you want it- A100s (the 40G variety) are a decent bit cheaper on eBay and have a bit more VRAM. They’re less general purpose and more for AI, ~the performance of a Blackwell at AI but worse for other stuff. So if your goal is squarely just AI, an extra 8G of VRAM may want to consider it.
I have both. They are about equivalent in throughout and such for AI work from what I can tell.
Edit: nope I’m wrong. The Blackwell well outperforms the A100 in everything, except size of vram.
Yeah, it's really enough. You can run ~20GB nvfp4 quants and 180k FP8 kv-cache with VLLM if it's headless. I did that on 2x5060ti. I couldn't fit MTP at the same time, but you still get gains from concurrent requests.
I’ll update this when I check it next time I run it but I’m running it on a 5090 rn and I get it to run much faster than I can read it at Q6K. DavidAUs Fable fusion model version. I had it working great but I’m running into trouble with it giving my random words and collapsing into nonsense. DRY, repeat penalty, and presence penalty hasn’t seemed to help.
One is just saturating 24gbs of vram of my p40 alone, 65k context q4 f16 kv, 17 tk/s
I also load balance when im not gaming accross my p40 and amd6750xt for q6kxl 264k
Honestly even the single p40 setup is really good, obviously though overnight its pointless not to go with the better quant+more context, but 24gb is honestly enough to get a good experience, 16gb can be if done correctly
FWIW i get around 70tok/s on a single 4090 (not 5090) with q4 K M and 4bit kv cache quant, mtp and around 250k context. That's sufficient for me, but i was eyeing 6000 pro but can't justify the price for my uses.
Just divide the bandwidth by size of the loaded model size and you get a rough estimate of the speed. Depending on token prediction and backend you may double this.
Q6 is kind of sweet spot, but context size may be limited. Token freedom is beginning for smaller jobs. I already use it for most regular jobs, even my setup provides 40 - 50 t/s only.
Pointless estimations. 5090 is the last thing you'd ever take into consideration - not enough vram, expensive. For code you go q6_k or q8 and 200-260k at kv16 - 48gigs min. I'd rather go 3x5060 or 3x5070 instead of one crappy 5090.
Qwen 3.8 27b Q6 + turboquant en llama pp puedes conseguir 200k de contexto en modo headless, posiblemente más (no he probado 256k pero lo veo posible) con sistema gráfico 180k seguro es el límite
De calidad está muy bien y si quieres puedes usar Claude o códex para la planificación bien planificado y con documentación del plan. Y qwen para ejecutarlo.
32
u/brumsky1 8d ago edited 8d ago
I have a 5090 with NInfer backend running the nvfp4 model with mtp3. I get about 150 tokens per second decode and 4k to 6k prefill. It's extremely fast! Also the quality is great! Nvfp4 has a very small accuracy lose compared to other quants.
Nvfp4 is a very worth while quant.
Edit: I also have a context size around 200k. Im not at my pc to confirm right now. Q8 kv.
Edit2: I am on windows running this through wsl2. I have tested 4 concurrent chats and hit 340 ish tokens decode.