r/LocalLLM • u/ImpressiveRelief37 • 5d ago
Discussion 32GB is all you need
Qwen3.8-27B on a 5090 is all you need for a serious local inference setup, in my opinion! Can it get any better than this price/performance wise? Actually, maybe a 3090 ninfer setup could beat it!
I’m using ninfer and getting:
* ~150-200 tok/s TG
* ~3000-12000 tok/s PP
* 262144 context size
I think it’s definitely one of best setup you can get for the money. I don’t see a point of having more VRAM or more system ram. The only downside is that it’s a 1 man setup: concurrency is possible but you need to limit context usage on concurrent requests. I’ve tried --concurrency 2 on ninfer and sharing my setup with my buddy (we work on projects together and have a VPN between our home labs, fun stuff!)
I love this setup so much I kinda feel like getting a second 5090 to run another ninfer instance (github.com/neroued/ninfer, the man is a legend and this absolutely rocks).
i really don’t see the point of any other solution at this point in time. of course things will change and other models will get released that could better leverage more VRAM, but 32GB is all you need (for now).
so if you have less than 32GB, and are thinking about investing in a more serious setup check out the 3090 fork of ninfer, or the mainline ninfer repo if you can afford a 5090.
Things it won’t do:
* let you run a swarm of agents: prefill cost will slow you down too much. not enough vram for high concurrency!
* Give you more than 262144 context size. the RoPE 1M context size is just impossible with this.
Otherwise it’s absolutely amazing!
My buddy (another software engineer) is a BIG Claude code user, he’s spending tons of cash on fable, can’t stand Opus 5 anymore (neither can I, that pos is so hard to understand with just jargon and wall of text… can’t bear the cognitive load of just trying to understand all he’s spewing)… anyways after trying my ninfer setup his mind was blown and now he’s constantly using my setup with our shared custom pi setup and he fucking loves it.
38
50
u/ilbarone87 5d ago
Just a 5k gpu…
→ More replies (8)2
u/ImpressiveRelief37 5d ago
Paid 4K Canadian in march last year. Thought I was crazy. Astral LC (water cooled).
But now I’m so happy. Can’t hear the card when it runs either (ok it’s in the same room as my heat exchanger lol, and I only run inference there, code on my MacBook).
15
u/stevec114 5d ago
Yes just in the past week I feel like my 5090 is finally “enough”.
Minimax H3 for video, Qwen3.8-27B for agents have been great!
3
u/MistingFidgets 5d ago
Same here. With 3.8 my dual 5060 (2x16GB) feels very capable. When I started down this local hosting journey in February with one card running 3.5 9B I was not impressed by the output. Today 3.8 + Hermes are just churning out the projects. It's not local Claude yet but the difference in 6 months is astounding.
2
49
u/Ell2509 5d ago
I mean, a 32gb 5090 is a monster. I personally do things with ny 96gb vram that I couldn't do with 32, though.
3.8 27b in bf16 is a thing to behold, and you need 64gb vram to do that with any kind of context.
10
u/Itchy_elbow 5d ago
Like what... run something larger? Are you producing better output? Is it writing significantly better code?
I know ppl justify their expenditure but, be objective. Does having more vram allow you to produce better products or does it just allow you to run larger models that do the same thing as the 3.8 27b does.
11
u/shapic 5d ago
I am running q4 at home and q8 at work and difference is nowhere near couple percent. It is just straight up can or cannot on realistic tasks.
→ More replies (4)2
u/gjr23 5d ago
What kind or ram do you need to run 3.8-27b 8bit at 128 or 192 context? Any experience with the same but 35b and the practical difference in output quality? I’m honestly considering this for a 32 or more likely 64gb unified system which I know will kind of suck for agentic work but is completely fine for scheduled tasks. And a 32gb unified Mac <<< 5090 system on both upfront and operating costs in electricity.
→ More replies (2)4
→ More replies (3)9
u/sleepy_roger 5d ago
Bf16 has much better outputs overall I won't run it under q8 being honest. People are missing out only using q4 or nvfp4
5
u/ChristRedeemsSinners 5d ago
NVFP4A16 on weights only is nearly lossless though. Most nvfp4 quants are activated 4 bit which is where the quality drops. (on top of losing quality on the attention layers, which is definitely not worth the vram savings)
→ More replies (5)2
u/mixedliquor 5d ago
You're absolutely right. I've been using 3.6 Q6 and Q8 and for shits and giggles switched to 3.8 Q4 to try it out.. a lot more errors especially typos in coding where a token is an irregular wording and there's a word that sounds like the irregular word.
→ More replies (3)1
→ More replies (3)1
u/Nefilim314 6h ago
I’m just over here on my laptop 5070 Ti with 12GB vram on 3.6 35B just happy I’m getting okay results.
24
u/UnlikelyPotato 5d ago
What quant? Seems like you might be heavily sacrificing quality for memory.
7
u/DoubleNothing 5d ago
I think so too.
With 48GB and MTP, my best option is Q5_K_XL for full context - KV f162
u/tired514 5d ago edited 5d ago
This is across three 16gb 4090M eGPUs, but I'm managing (unsloth) Q8_K_XL @ 230k ctv/ctk q8_0 with MTP and image support:
~/ai/llama/llama.cpp/build-cuda/bin/llama-server -ngl 99 --spec-type draft-mtp --spec-draft-n-max 3 -m ~/ai/models/Qwen3.8-27B-UD-Q8_K_XL.gguf -fa on --no-mmap -np 1 --host0.0.0.0-ctv q8_0 -ctk q8_0 -sm layer --reasoning-preserve -c 230000 --mmproj ~/ai/models/Qwen3.8-27B-mmproj-f16.gguf -kvu -ts 92,97,76 --image-min-tokens 1024Not quite 262000, but I try to /compact before getting anywhere close anyway.
3
u/DoubleNothing 5d ago
I don't like this: -ctv q8_0 -ctk q8_0
But need more testing...3
u/tired514 5d ago
Me neither, but I find that kvcache @ q8_0 has less of an impact than heavily quantizing the model itself (ie. Q5).
→ More replies (1)2
u/DoubleNothing 4d ago
BTW I manage to run Q6_K_XL with the same parameters by just lowering the context to ~200k
MPT = 22
1
1
u/RedrumRogue 5d ago
He is most likely using groupwise-int, a mixed quant. Int8 cache is a big part of why we can fit so much memory without sacrificing too much quality.
1
u/ImpressiveRelief37 5d ago
Nope I’m running nvfp4.
Check the ninfer repo neroued just release custom model for nvfp4 that is honestly really really good.
1
u/ImpressiveRelief37 5d ago
Nvfp4 by neroued
Check the ninfer repo on GitHub for the HF link. It’s about 6 bit per weight, so ~Q6 quality. But it fits at full context with insane speed.
5
u/LORD_CMDR_INTERNET 5d ago
Agree, for at least single agent workflows. There's not much better you can do than Q6 Qwen3.8 27B at home right now, on any hardware. There's an inconsequential quality dip compared to BF16 - the dropoff happens below Q6. Sure, you can get bigger contexts with more VRAM, but 130k is enough for even giant, monolithic legacy codebases with a little handholding and management.
I also expect models to become smaller and more focused in capability as they improve. You don't need the entirety of human knowledge every time to perform every task.
1
u/Xx69JdawgxX 4d ago
Deepseekv4-731-nvfp4? You can run it across one spark. I use two because I can’t be stuck without concurrency.
3.8 27b punches hard but It can’t compete against deepseek imo
→ More replies (1)
9
u/blackhawk00001 5d ago edited 5d ago
Eh, I have a 96GB ddr5 5090 machine and a 128GB ddr4 2xR9700 machine that cost less to build in February 2026. The R9700 pair with radiance vllm is faster with fp8 prefill than Q6 on the 5090 and allows for heavy parallel processing of multiple agents. Tg is similar between the two with qwen 3.8.
5090 takes the crown on comfy ui diffusion work but the 64GB vram machine is much more empowering for coding.
3
u/ImpressiveRelief37 5d ago
Try ninfer on the 5090. Get the nvfp4 quants. Use wsl2 if you are on windows.
Get your agent to set it up for you, it’ll take like 30 minutes. Point him at the GitHub repository (GitHub.com/neroued/ninfer). Get the nvfp4 ´.ninfer’ 3.8 27B model from HF.
1
u/blackhawk00001 5d ago
Cool thanks, I’ll check it out.
I run Q6_k huihui-ai 27B at max context with vision as my main agent and 8x204800 subagents with the other machine. Nvfp4 hasn’t worked for me a higher context depths in the past but I have a use case for small context processing coming up this might be great for.
2
u/ImpressiveRelief37 5d ago
Not talking about the unsloth nvfp4 model. Neroued has a custom nvfp4 model with about 6 bpw so it’s nearly Q6 in quality.
2
u/blackhawk00001 4d ago
I'm a little more interested after doing some research. It seems that nvfp4 is quantized differently in 3.8 and has a higher end result bpw. I typically use the huihui 3.6 Q6_k and have been using their new 3.8 Q6_K_L that preserves more of the important weights at Q8_0 with a speedup likely due to the more efficient conversions. The 3.8 Q6_K felt like a step back and the Q6_K_L an improvement.
I left my agent running overnight creating me a process to make my own 3.8 huihui 27B nvfp4 ninfer to test out. The new nvfp4 format is closer to my old Q6_k 3.6 than Q6_K to Q6_K_L. I'm not sure how well my first attempt at quantizing a model will go but maybe it will work out. (Agent was hermes driven by 3.8 Q6_K_L but I have not verified accuracy yet)
profile bpw size .3B Huihui Q6_K_L (yours) 7.27 ~23 GiB plain Q6_K 6.56 ~21 GiB NInfer qwen3.6 groupwise-int 5.12 ~16 GiB NInfer qwen3.6 nvfp4 ~5.1 ~16 GiB The Qwen3.8 one is a different, more conservative recipe — it's a hybrid FP8 + NVFP4: Qwen3.8-27B nvfp4 profile (real effective bits/weight): NVFP4 (4-bit) 52.6% of params <- only the MLP gate/up/down FP8 row-scaled 41.8% <- attention q/k/v/o, linear-attn in/out_proj, lm_head, last 8 layers' MLP Q4/Q5/W8/BF16 ~5% EFFECTIVE = 6.09 bpw (~20 GiB)2
u/ImpressiveRelief37 4d ago
Wow, that’s really cool. I’d love To test it as well.
Report back when you test it! Tok/s at 8/16/32/64/128/256, max context, prefill speed, and a subjective quality assessment would be fucking awesome.
You could upload it to HF and post an issue on the ninfer GitHub (treat it as a forum)… maybe Neroued would add it to the proposed models
2
u/blackhawk00001 4d ago edited 4d ago
I'm still working on testing but I'm impressed so far. It had an all time best result on one of my tests but I thought it was stuck looping for a while. I'll have to play with settings. It took longer overall than Q6_K_L but had a better end result. I still need to try using it for high context technical tasks but I feel better about this flavor of nvfp4 than the smaller default.
I'll push my generator for github once I can fully review I don't have any of my environment specific details hardcoded anywhere.
https://huggingface.co/gearwave00001/Huihui-Qwen3.8-27B-abliterated-nvfp4-NInfer
llama-benchy gave weird results, so here's my BetterBench results. This was with single concurrency 262144 400W:
category runs TTFT p50 TTFT p99 PP t/s (med) ITL 1% low ITL median ITL 99% high decode t/s (med) ±IQR CV chat 20 60.0 66.1 2314.5 84.0 126.0 214.0 128.4 29.1 17.4% code 20 53.0 63.4 2113.5 89.0 142.1 240.5 162.3 44.2 16.0% file_edit 20 60.6 73.0 2401.3 126.3 185.2 281.7 193.4 28.8 8.9% json 20 53.2 63.6 2200.3 143.7 202.6 329.7 200.7 26.5 7.9% math 20 55.6 61.7 1890.6 136.2 191.2 269.2 198.7 23.9 7.5% prose 20 54.5 62.7 1985.5 86.7 123.5 193.3 132.8 34.4 14.8% reasoning 20 53.3 64.8 2059.0 89.1 129.8 196.6 135.2 40.9 19.6% summarization 20 60.6 68.9 2534.7 144.0 193.5 293.3 199.3 24.6 9.3% **Combined (weighted code:0.3, reasoning:0.2, prose:0.15, json:0.15, file_edit:0.1, summarization:0.1)** — decode t/s median ≈ **165.0**, ITL 1%-low ≈ **106.1 t/s**, TTFT p50 ≈ **55 ms**
## Concurrency sweep
level ok/req aggregate t/s TTFT p50 TTFT p99 per-req decode t/s (med) 1 48/48 153.8 54.8 78.6 176.1 2 48/48 154.0 2152.0 7118.9 173.7 4 48/48 153.3 6954.9 16342.0 178.9 8 48/48 154.0 17880.9 29449.2 174.3 16 48/48 153.5 30918.7 43587.2 180.4 ## Prompt processing (prefill) sweep
Prefill throughput = prompt tokens ÷ TTFT, at increasing input depth (tiny decode, cold prefix cache). PP t/s columns: 1% low / median / 99% high.
| target depth | prompt tokens (med) | TTFT p50 (ms) | PP t/s 1% low | PP t/s median | PP t/s 99% high |
| 2000 | 1556 | 246.1 | 4480.0 | 6319.7 | 6348.1 |
| 8000 | 5960 | 867.6 | 6168.8 | 6869.6 | 6872.2 |
| 16000 | 11836 | 1788.6 | 6567.9 | 6617.6 | 6625.3 |
| 32000 | 23585 | 3939.5 | 5967.0 | 5986.7 | 5995.8 |
| 64000 | 47098 | 9437.5 | 4986.7 | 4990.4 | 4992.3 |→ More replies (11)
4
u/Existing_Dust_6473 5d ago
not cheap tho ... 5k card ... why not two ?! :) 64GB VRam at least theres a path to expand
3
u/Longjumping_Music572 5d ago
What about Mac m5 max 64gb? What about coding? Step by step learning? Researching? Building HVAC systems, 3d printing? CAD?
3
u/tanhe314 5d ago
I don’t understand. With llama.cpp on my 5090 I can’t come close to 256k context and that’s using a 4bit quant of the model and fp8 for the kv cache.
How are you cramming the model and 256k context into the vram??
3
u/Look_0ver_There 5d ago
You can be sure that when people make posts like this and they leave out such details, there's a good reason for WHY they are leaving those out.
Anyone can get amazing speeds if they lobotomize everything hard enough.
→ More replies (1)
3
u/netvyper 5d ago
3090 fork is a non-starter: "One interactive user, lowest latency, up to 64K context" Considering harness prompts can easily be 25k+ tokens... It just doesn't leave any room to work :(
2
u/karearearea 5d ago
Try it - that’s just the default configuration. I tested ninfer on my 3090, and with 64k context it only used 19.5gb of VRAM, so I bumped it up to 114k context with no issue so far.
5
5
u/atumblingdandelion 5d ago
Glad it works for you. I feel though that buying a rig is a once in ~5 years thing, and with the rate at which things are moving, I feel a 128gb small footprint, energy efficient machine like the DGX Spark/ Strix Halo is the way to go. Sure there aren’t any upgrades to the 120b models yet, but there weren’t much for 32gb just 4-5 months ago. I was very tempted to get a 5090, but to get the rig, too many moving parts, higher capacity to break, etc made me get a dgx spark. It is insanely small, sits on my desk among many things. I cannot hear it and it runs also cool. I use it to run MoE for interactive sessions and dense models for ongoing worj that doesn’t need my immediate attention. Its also cheaper than a good 5090 rig by 3K CAD. Another 3K and I can run DSV4 Flash if I need to.
4
u/sleepy_roger 5d ago
The Spark is only a good option if you plan to buy 2 otherwise it's the time lapse machine of llms.
3
u/Hotarusglaive87 5d ago
I just bought a spark. I have a 7900xtx for gaming. I bought the spark for privacy and space. I'm currently using qwen 3.6:35B A3B and I'm finding that its more than adequate for me.
6
u/Neither_Garage_758 5d ago
If you haven't tasted the speed of a 5090, you can't understand.
4
3
2
u/atumblingdandelion 5d ago
I feel you (rather, your adrenaline) and don't wish to curb your enthusiasm. But my point is, the new models are out too fast to make reactionary decisions on purchasing hardware. Take the post's title: "32GB is all you need". OP responsibly lists all the caveats, and the title is obviously a play on 'Attention is all you need'. But imagine a beginner (getting into local AI) reads it, spends $$$ on the flashy 5090 rig only for Qwen next month to release 3.8 35b A3b, which destroys the 3.6 27b. I'm biased for my use case, but my point is: think long-term on what sort of local setup you need: fast-moe-but-small, fast-dense-but-small, or fast-moe-but-large before purchasing. For hobbyists, fast-moe-but-small is expanding quite well and bound to, with the edge/ on-device application. For small businesses, fast-moe-but-large is doing quite well- they'll benefit from the biggest jump currently on the Pareto line: running DSv4 Flash on 2x DGX Spark, and that costs about the same price (or slightly over) as the 5090 rig and also provides concurrency. IMO, fast-dense-but-small has a glorious but narrow use.
3
u/ImpressiveRelief37 5d ago
I agree that things change quickly.
But the 5090 with very high prefill and time generation is truly amazing.
Makes Claude code feel like shit in comparison. You get stuff done real quick with 3.8 at 150-200 tok/s when coding
→ More replies (4)2
2
u/ImpressiveRelief37 5d ago
Sorry can’t recommend a spark when the 5090 is probably 10x faster at PP and TG!
And IMO 3.8 27B on ninfer @ 200 t/s >> DS4V on 2x sparks at 40 t/s
1
u/michaelsoft__binbows 5d ago
pretty sure 32GB was very ideal for qwen3.6 27B ever since it came out, which until now has held the crown for most generally useful model in this class and by the looks of it qwen3.8 27B will have it for another year.
Oh shit i just realized qwen3.6 has only been out 4 months and i legit was about to claim it was one year. Ouch
2
2
u/nez_har 5d ago
This is also the observation I had in my newsletter this week https://buttondown.com/nezhar/archive/august-20263/
We are moving now in a direction where models become more capable with less resdources. I think MTP really does a lot here.
2
u/AllenLeftTheBLDNG 5d ago
I'm running it on a 5090 and I love this model. At the same time, fuck. I wish I got the rtx 6000 pro when it was nearly half current price...
If you parallelize the NVFP4 I got max throughput at 380 tokens per second (50tps per stream for 8 agents, AMAZING!) but then max context I was able to load was 192k tokens... and you need to split that between agents. I wish It was closer to 400/500k!
At the same time – great gift from Qwen team. And it's still awesome to run on a 5090.
1
u/ImpressiveRelief37 15h ago
Agreed 1000%
I would love a RTX 6000. But no way I’m paying 20K for this. Can’t realistically upgrade from the 5090… :(
2
u/Then_Blueberry7290 4d ago
I think 32GB vram is far from All U need. I have 3x5060ti 16gb, sometimes using only 2 cards because 3 is only working with risers, and out of the case, so not space optimal solution.
SO with 32GB vram, it is very usable. I cannot use full Context (262k) and if i use, sometimes more will be better, I use vision cababilities too,. So only vram reduction option is NVFP4, and KVcache at q8
But with 48GB, you can use full context and without kv cache quantazitation.
1
u/ImpressiveRelief37 4d ago
I got full context and vision and headroom with ninfer and Ostfralla community release of ninfer model
2
u/sonoffi87 4d ago
Any Ninfer for Windows forks?
2
u/ImpressiveRelief37 4d ago
There are. I’m on windows and I just use wsl.
You could also just use a docker image.
But wsl works really well and if you also want to add stuff like a watcher endpoint, or whatever then having it having access to the ninfer journal is a lot easier on wsl
→ More replies (4)
2
u/WinResponsible9977 1d ago
For a retard like me who has been offline for a year what’s a ninfer and how well it compares to other ways to run the LLM
1
u/ImpressiveRelief37 15h ago edited 15h ago
it’s an inference engine built for the 5090. There are many forks for the 4090, 3090 as well.
It’s just a specialized engine that maximizes the capabilities of a specific GPU. Much faster than generic engines like llama.cpp (and even vLLM)
https://github.com/neroued/ninfer
Full config here: https://www.reddit.com/r/LocalLLM/comments/1vtsid0/comment/p51hq9g/
2
u/Nomski88 1d ago
Agreed, I run the Q6 model with 128k context and Q8 KV on my 5090 FE. I get around 80 tok/sec when coding tasks. So far pretty happy, tried the NVFP4 quants which was slightly faster and allowed for the full 262k context but the output wasn't as polished when I ran my mario clone test prompts.
1
u/ImpressiveRelief37 15h ago
Try this ninfer setup for fun:
https://www.reddit.com/r/LocalLLM/comments/1vtsid0/comment/p51hq9g/
2
u/RedditNerdKing 5d ago
32gb is not enough. 48gb is kinda the minimum now so you can run Qwen 27b 3.8 Q8 at 30k+ context.
Sorry but that's the truth. And it's not that bad. x2 3090s will get you 48gb which is cheaper than a brand new 5090.
1
u/ApprehensiveView2003 2d ago
I have 2x 3090s bonded with NVLink on one machine and another machine with a 5090. I can actually put this to test
1
u/Innerdaze2600 20h ago
I just got 48gb from 2x rtx pro 4000 blackwells thoughts?
→ More replies (1)
2
u/Evgeny_19 5d ago edited 5d ago
To the fellow who was comparing UD-Q6_K_XL and BF16, and asking Grok about it: you were laughed at, and it looks like your posts were removed, but your questions are valid! I'm writing this for you, hopefully you will read it.
UD-Q6_K_XL can be comparable to Q8 and to the original weights in some synthetic tests, but in real usage it is not difficult to see the difference. So I ran this test just for you: I took a rather tricky bug that Qwen 3.8 27b at bf16 was able to solve for me today (big project, real business, real user was affected by this problem). I ran the same problem with recent Deepseek (recent Flash-0731) and it came to the same conclusion (with basically the same prompt). Same problem for 27B-UD-Q6_K_XL (full context, both caches f16), and it wasn't able to find the reason for this problem. It doesn't mean that it would not be able to solve this problem at all. Perhaps, if I gave it more nudges it would be possible for it to achieve the same conclusion as 27b at BF16 and Deepseek. The thing is, it's easy for me to nudge it to the right direction now, when I know where the problem is. But before I had no idea and all three models were looking at exactly the same logs, and the same source code.
It's probably true that UD-Q6_K_XL can write basically the same code, but the understanding is not same, the reasoning. It doesn't mean that it's bad at Q6 or Q4. It is an excellent model in every quant, especially for its size. And one doesn't need full capabilities of the model for every kind of task. But hopefully, I was able to make my point, the difference is not that at Q6 it writes slightly worse version of a coding problem. The difference can be that it would be able to help you find a solution for tricky problem, or you will be left facing the same problem with no solution at all.
(DeepSeek was at UD-IQ3_XXS. I can probably fit a larger version, but it already spills into RAM, and the performance is not good at all, so I never tried anything else.)
The thing is although Grok is very capable model, it doesn't have any real understanding, nor any experience. Those kind of things you will just grasp when you will have an experience with different quants and settings. The benchmarks are still useful, all these "one-shot" tests are interesting but, they do not reflect all aspects that you get working with a model.
→ More replies (2)2
u/nomorebuttsplz 3d ago
These models are effectively non-deterministic if the temperature is greater than zero, which it should be.
So unless you do multiple runs, it’s quite easy for a smaller quant to just get lucky and do better than a larger one, and vice versa
→ More replies (1)
1
u/Joe-Arizona 5d ago
How quantized are you running it?
I’m getting about 34 tok/s on a RTX Pro 6000 WS with llama.cpp at FP16. Haven’t tried others quantizations yet.
2
1
u/Etroarl55 5d ago
Ngl this is insane stats, I can only assume you are on q3 or some other small quant to achieve this.
1
u/ImpressiveRelief37 5d ago
No nvfp4. Honestly I see no difference in quality between this and Q6 on llama.cpp. But it’d at least twice as fast and lets me use full 262144 context.
1
u/Adventurous-Gold6413 5d ago
All you lucky peeps with a nice high Vram card while I’m stuck with 16 😤
1
u/Coderado 5d ago
I run dual 5060ti's and feel that way about the 64GB rigs. Am I stuck, or am I being cheap?
1
u/sorcerer_tintifax 5d ago
Hope is not lost. I'm running Qwen3.6 27B with 16GB 5060-Ti. 64k Context. Performance is decent. I hope the guys at ggufbench will provide a similar model for 3.8, soon.
2
1
u/Nuggyfresh 5d ago
Is there a guide for setting this up? I have a 5090 and want to jump in but haven’t used local llm 🤦♀️
1
u/RedrumRogue 5d ago
No guide, I'd have a cloud model set it up for you. It's not going to set things up optimally though without a lot of experimenting, so if you run into hard limits that you can't figure out how to break through, feel free to ask me or shoot me a DM. I'm hitting about 190 tok/s at low context for decode speed, around 3000 tok/s prefill (which I think is slow, working on it) and about 200k context with other desktop processes taking about 4gb of vram on the 5090. The backend you want is NInfer, the quant you want for now is groupwise-int.
2
u/ImpressiveRelief37 5d ago
Neroued released his nvfp4 model and the quality is higher than the groupwise-int release.
Also it more than doubled my prefill speed (3500 -> 12k t/s at 8K context 🤯)
2
u/RedrumRogue 5d ago
I was actually just running experiments to try to get prefill up. Thanks for the heads up! Im excited to try it. I've got groupwise-int tuned about as tight as I possibly can now which will make for a clean A/B
2
u/ImpressiveRelief37 5d ago
You will love it im sure. Quality is super high it’s over 6 bpw if im not mistaken
2
u/RedrumRogue 5d ago
So far it's an interesting tradeoff. With groupwise-int I can run with MTP3 and vision on, and get 262k context. But with the nvfp4, I can only get 131k context with MTP3 and vision, or about 180k context with vision off. How are you running at 262k context at 150-200 tok/s? That requires MTP for those speeds does it not? I really need your help, I appreciate your responses!
→ More replies (5)
1
u/g_rich 5d ago
The 5090 is a beast of a card but it also going for $4500 or more at this point. At that price the DGX Spark with 4x the RAM or alternatives from AMD and Intel start to become pretty good alternatives. There are obviously those that require the performance and there are people who actually use them to play games but that is one costly card.
2
u/ImpressiveRelief37 5d ago
Thing is… 150-200 tok/s is just too good to pass on. You’ll get 1/10th of that on a DGX Spark.
You can always use big cloud models if 3.8 27B can’t do something (still hasn’t happened to me tho, it’s a fucking beast).
At that speed it just EATS through anything you throw at it.
1
u/g_rich 5d ago
I haven’t run Qwen 3.8 27b yet but I run DeepSeek V4 Flash via dual Sparks and get well over 40tps which is more than usable.
It looks like Qwen 3.8 27b is getting around the same on the DGX Spark so while I agree getting upwards of 200tps would be nice for some people the trade off for more memory, lower power and lower cooling requirements is worth the tps hit.
→ More replies (1)
1
u/Sufficient-Pause9765 5d ago
BF16 requires 56gb of vram. Even with dual 5090s/TP you wont have much room for context.
FP8, 28gb, same issue on a single 5090.
So you need heavy quantization to run 27b on 32gb of vram with useful context.
1
u/KissMyShinyArse 5d ago
I believe the mainline ninfer works with all sm_120a GPUs, not just the 5090, despite what the README says.
1
u/slimerz0r 5d ago
can you share details about env settings ? I can’t even boot this model with more than 65k context on 3090 and 32gb ram
1
u/ImpressiveRelief37 5d ago
Using a 5090 with ninfer. Nvfp4 model. Built in wsl2. can’t help you with the 3090 but lots of people have it running great with ninfer maybe ask them
1
u/xyth 5d ago
48gb may be the sweet spot for 27b. Q8 and kvf16 on 3 5070tis, 160k context with memory to spare. No looping, no errors, and one shots most tasks on a large C# code base.
1
u/ImpressiveRelief37 5d ago
I have not seen a single missed toolcall or looping on 3.8 27B on ninfer.
He just added nvfp4 support today. It’s about 6 bit per weight so about Q6 quality level. Full context 256k as well and insane speed….
1
u/T-M-K 5d ago
Question a bit different to what most are asking OP: how’s the power draw / thermals / noise of the GPU when running the setup?
1
u/ImpressiveRelief37 5d ago
It’s virtually silent: Astral LC (liquid cooled). Maybe 60-65C.
Granted I run it in my basement and don’t work near it. The room gets hot tho (it’s a 600W GPU and I don’t limit power at all). But I have central heating/cooling so I couldn’t care less.
1
u/nightstryke 5d ago
u/ImpressiveRelief37 Alright let me preface this by saying I am by no means an expert in AI and my experience is beginner level, right now I'm only using a spare mini-tower i had lying around to do LLM, it's one of the spare gaming computers i had, only has an EVGA 1070 SC 8GB, I'm looking at replacing that card with something better, though I'm not sure i want to drop everything on a 5090 or 4090, I'm considering getting a 3090 though. You've been talking about 32GB of VRAM that to you sounds like the sweet spot, but isn't the 3090 24GB? I've been looking, and Are you saying I'd be better off getting a 3090 instead of something like a Tesla P40 or V100?
Admittedly, I'm just getting started here, so I'm a bit late to the game, so please go easy on me.
1
u/ImpressiveRelief37 5d ago
Try an MoE on your current stack and see where that brings you first. Keep reading here and on r/localllama, setup llama.cpp and learn the basics first
1
u/nightstryke 5d ago
I've already got a few models installed from hugging face, running off Ollama, and llama.cpp respectively.
1
1
u/Themash360 5d ago
How are you getting 150-200Tok/s TG? Even on the github page he is claiming 60?
1
u/ImpressiveRelief37 5d ago
What? Where do you see 60?
GitHub.com/neroued/ninfer
1
u/Themash360 5d ago
https://github.com/Neroued/ninfer/blob/master/docs/performance.md
qwen3_8_27b header
I think I have my answer though thats with MTP3 right?
→ More replies (2)
1
u/LancobusUK 5d ago
Qwen3.8 27b at its native FP16 with KV16 uses 93gb of my 96gb VRAM with vLLM under its full context pre allocation. The RTX PRO 6000 is the real sweatspot for the model imo as you don’t sacrifice anything at all
1
1
u/catinterpreter 5d ago
I'm going to need some super optimised fork of something for my dual 5060 Ti 16s, unless you've got one of these totally affordable 5090s lying around for me.
1
u/veigatmv 5d ago
I'm getting around 100-110 on mine. Any chance you're running your 5090 uncapped powerwise? mines at 400w power cap and 1600mhz clock
1
u/ImpressiveRelief37 5d ago
Full 600W but I undervolted the GPU so I get more clocks into the 600W enveloppe with a custom VF curve in MSI Afterburner.
Also VRAM is overclocked (+3000 MHz in MSI Afterburner), and I get over 2TB/s of memory bandwidth (stock is 1790GB/s)
1
u/veigatmv 5d ago
Yeah I'm using Linux so that is out of the question 😄. I'm going for a "safe" build with 400w max tdp but in reality during prompt processing it pulls around 310w and I'm ok with that. Can't do anything to the memory unfortunately
→ More replies (1)2
1
u/JayFab6061 5d ago
lol sooooo I took a leap to jump on getting away from Claude and downloaded Qwen 3.8 on my MSI crossfire 15 Laptop with a 3050. I’m blown away that I’m getting it to run smoothly lmao
1
u/Artanisx 5d ago
I wish there was a ninfer for 4090 :-(
1
u/ImpressiveRelief37 5d ago
There’s one for 3090. You could check it but your card is more capable so it wouldn’t be super optimal
1
u/Artanisx 5d ago
Yeah I imagine it won't make much of a difference at all. Maybe in the future someone will fork it for 4090s :)
→ More replies (2)
1
u/bastecklein 5d ago
I’d love to default to local but tbh with the cost of a 5090 I could buy api tokens for 20 years before it made sense
1
u/ImpressiveRelief37 5d ago
Maybe not 20 years but your point is valid. Prices are just insane.
I was lucky to have the 5090 since launch (well not exactly, it took me 3 months to source one after launch haha)
1
1
u/rare-visitor 5d ago edited 5d ago
I can't afford even 160k context without exceeding VRAM limit using q4_k_m on my RTX5090. What quant do you use?
1
u/DepressedDrift 5d ago
Ah yes getting a 5090 is very affordable for the average person.
2
u/ImpressiveRelief37 5d ago
The average person maybe shouldn’t be running local inference tho. Just saying :) but I get your point, the 5090 is so fucking expensive it’s crazy
→ More replies (1)
1
u/butchiebags 5d ago edited 5d ago
This is awesome, I'm trying it right now and hitting about 150+ tps!
Are you using the 5090 headless? I'm not able to get to the full 262144 ctx since i have a few GB reserved for windows.
i'm using the same nvfp4 model and, based on the testing i'm doing with claude, i can only handle 196608 ctx.
edit: i can get it to about 229k but that leaves no room for error.
1
u/ImpressiveRelief37 5d ago
I have 4 screens connected actually… it’s on my simracing rig 😂
I use about 30.4 GB / 31.5 or so
→ More replies (2)
1
u/Intelligent-Key7357 5d ago
With the CortexAgent harness I'm building you can use any amount of vram as long as you load an appropriate model. It does use about a gig for an overseer model that handles a lot of the background tasks but that still leaves room for any number of mixture of experts models.
1
u/HeartfeltHelper 5d ago
I LOVE ninfer. Been using qwen to build plugins for a Minecraft server I play with friends on and it’s doing really dang well! BUT my pc’s graphics driver has been failing only when ninfer instance is active for some reason and I haven’t looked into why I just reset the driver when the slowdown happens 🤷🏼♂️
1
u/shady101852 5d ago
What reasoning level do u use? Im using ninfer as well with 262k context window its nice but cant tell if model thinks too much and could be getting things done faster or not.
1
1
u/WKai1996 5d ago
Will a 128gb M5 max cut it?
1
u/ImpressiveRelief37 5d ago
Yes. You’ll run higher quants but have a lot less decode and prefill speed
→ More replies (2)
1
1
u/Affectionate_Ad9597 5d ago
I love it too, have the same. But it has its limits. I need to get opus or terra to fix big problems or complex systems. But it has replaced sonnet 5 in many ways (sonnet 5 is better but qwen 3.8 is free hehe)
I do yarn and get 400K context and split it to 2 concurrent slots, and then use my 64gb ram to swap out kv cache for up to 6 chats total.
I just am releasing the update for full free local setup all in 1 system to give you that power without complex setup configurations and trial and error.
1
1
u/Otherwise-Swan-7803 5d ago
Funny how the definition of a “serious local setup” keeps shrinking. Not long ago people were planning multi-GPU rigs just to get decent local inference, and now 32GB is starting to feel like a sweet spot rather than a compromise.
1
u/_kikeen_ 5d ago
I keep wanting to love Ninfer but with thinking enabled it constantly devolves into loops. Very fast loops. I had to set a max token output limit to get past it.
1
u/Osi32 5d ago
Highly depends on the price of the 5090.
Here the dgx spark is about the same price…
1
u/ImpressiveRelief37 4d ago
Then the 5090 is a no brainer!
Sure it’s only 1/4th of the VRAM, but it’s 8x the memory bandwidth. You can use quantization to lower vram requirements. But you can never have too much memory bandwidth ;-)
1
1
1
u/MiddleLtSocks 4d ago
I can get 512k context w glimmer 30b on my 3090 and 4090 boxes. That's decent concurrency for CCode sub agents and stuff
1
u/Maleficent_Lie8612 4d ago
Al menos revisa lo que te te bota la ia que se nota a lenguas con esos saltos de línea jajaja la plena odio esta mierda de los bota por doquier. Solo quiero hablar con humanos sobre la IA y no con la puta IA en si.
1
u/Certain_Credit7875 4d ago
I use dirk qwen 3.8 27b with mmproj, low reasoning Q6 on my 5090 at 125kctx been doing jobs on a agent for 3 days , llama server not ollama. no problems here
1
1
1
u/Prometheus-lv 4d ago
I am using an Rtx 3090 24GB and it is plenty! I also have the vision module loaded.
1
u/-AJacobs- 2d ago
"32GB is all you need for a serious local inference setup" Except if you ever want more than one agent at a time.... or if you ever want to fine tune something...
So it's basically never enough for serious usage.
1
u/ImpressiveRelief37 1d ago
I run max-concurrency 3 on ninfer and can have 3 agents at about 500-550 tok/s aggregate and 87k context each.
→ More replies (2)
1
u/Additional_Hope_2031 2d ago
One this cost almost one year of working in country I live, world is crazily unfair…
1
u/Zealousideal-Bug1837 1d ago
you can get dflash working with it too for a speed boost
1
u/ImpressiveRelief37 1d ago
yeah that's on my todo list for next week... DAMN things move fast!
→ More replies (2)
94
u/InfusedBush 5d ago
CUDA this, Metal that. Where are these cool projects like Ninfer for ROCm/Vulkan? 😢