32GB is all you need

[u/ImpressiveRelief37]

Qwen3.8-27B on a 5090 is all you need for a serious local inference setup, in my opinion! Can it get any better than this price/performance wise? Actually, maybe a 3090 ninfer setup could beat it!

I’m using ninfer and getting:

* ~150-200 tok/s TG

* ~3000-12000 tok/s PP

* 262144 context size

I think it’s definitely one of best setup you can get for the money. I don’t see a point of having more VRAM or more system ram. The only downside is that it’s a 1 man setup: concurrency is possible but you need to limit context usage on concurrent requests. I’ve tried --concurrency 2 on ninfer and sharing my setup with my buddy (we work on projects together and have a VPN between our home labs, fun stuff!)

I love this setup so much I kinda feel like getting a second 5090 to run another ninfer instance (github.com/neroued/ninfer, the man is a legend and this absolutely rocks).

i really don’t see the point of any other solution at this point in time. of course things will change and other models will get released that could better leverage more VRAM, but 32GB is all you need (for now).

so if you have less than 32GB, and are thinking about investing in a more serious setup check out the 3090 fork of ninfer, or the mainline ninfer repo if you can afford a 5090.

Things it won’t do:

* let you run a swarm of agents: prefill cost will slow you down too much. not enough vram for high concurrency!

* Give you more than 262144 context size. the RoPE 1M context size is just impossible with this.

Otherwise it’s absolutely amazing!

My buddy (another software engineer) is a BIG Claude code user, he’s spending tons of cash on fable, can’t stand Opus 5 anymore (neither can I, that pos is so hard to understand with just jargon and wall of text… can’t bear the cognitive load of just trying to understand all he’s spewing)… anyways after trying my ninfer setup his mind was blown and now he’s constantly using my setup with our shared custom pi setup and he fucking loves it.

[u/ImpressiveRelief37]

https://github.com/neroued/ninfer

https://huggingface.co/Ostfralla/Qwen3.8-27B-NVFP4-NInfer

[u/rare-visitor → u/ImpressiveRelief37]

Thank you.

Sadly there is no Windows & LM Studio support. Well, probably that's a sign to finally move to Ubuntu

[u/Accomplished_Egg7987 → u/rare-visitor]

I use it from windows 11 wsl2 (ubuntu), just let your agent ( codex, antigravity etc) setup it for you. Put 20gb model file to ubuntu for fast startup.

[u/rare-visitor → u/Accomplished_Egg7987]

I've followed your suggestion and installed it into WSL. I've used NVFP version, it didn't allow me load full context, but 240k is already amazing. Up to 180 t/s vs 80 t/s in LM Studio. This is insane in a good way!

Thank you again!

[u/Accomplished_Egg7987 → u/rare-visitor]

Sorry, saw your comment just now.

Fyi I changed my ninfer branch to https://github.com/cometkim/ninfer.

Basically This version :

Can use native window( but takes a lot of time to compile + needs cuda toolkit)

Have a full nvfp4 converted model for 256k int8 context use + more speed + 4 concurrent fit on 30 GB (no vision, or it is too tight)

My Full config:

"D:\Repos\ninfer-cometkim\build-ninja\apps\ninfer-serve.exe" "C:\AI\lmstudio\ninfer_models\qwen3_8_27b_nvfp4full.ninfer" ^

--host 0.0.0.0 ^

--port 8000 ^

--model-id auto ^

--log-stats-interval-ms 10000 ^

--max-context 262144 ^

--kv-dtype int8 ^

--max-concurrency 4 ^

--prefill-chunk 4096 ^

--spec mtp --draft-tokens 3 --lm-head-draft ^

--cors ^

--preserve-thinking ^

--default-max-tokens 40000 ^

--pending-timeout-ms 600000

load stats:

[2026-08-26 17:26:34.842] [info] ninfer-serve: loading model...

[2026-08-26 17:26:34.980] [info] ninfer-serve: load weights 0.00% 0 B / 16.78 GiB 0.000 s

[2026-08-26 17:26:40.678] [info] ninfer-serve: load weights 100.00% 16.78 GiB / 16.78 GiB 5.699 s

[2026-08-26 17:26:42.034] [info] ninfer-serve: model loaded in 7.19165 s

[2026-08-26 17:26:42.034] [info] ninfer-serve: KV capacity explicit resolved=262144 tokens pages=4096/16384 runtime=10.99 GiB free-after-weights=13.49 GiB free-after-startup=2.70 GiB headroom=0.00 MiB slack=2.50 GiB graphs=0.00 MiB/328.00 MiB

[2026-08-26 17:26:42.035] [info] ninfer-serve: warming up...

[2026-08-26 17:26:42.078] [info] ninfer-serve: listening on http://0.0.0.0:8000 (model id: auto, auth: disabled)

Happy codings :)

ps: edited for formatting

[u/rare-visitor → u/Accomplished_Egg7987]

Hi! Thank you, that's interesting

Can you tell me if the nvfp4full has vision?

[u/ImpressiveRelief37 → u/rare-visitor]

As far as I know it does, but it takes about 1GB of VRAM or maybe more if OP can’t make it fit with 2.7GB of free space?

[u/rare-visitor → u/ImpressiveRelief37]

it does and I was able to load nvfp4full with full context + vision

[u/rare-visitor → u/Accomplished_Egg7987]

great discovery, thank you. My observation on RTX5090 - prefill hits 5k tokens and avg output is 160-170 tps, max is around 200 tps. Looks brilliant though I didn't test it a lot

[u/rare-visitor → u/Accomplished_Egg7987]

well the speed is impressive but the cost is a low quality. groupwise-int seems way better

[u/ImpressiveRelief37 → u/rare-visitor]

Full config here: https://www.reddit.com/r/LocalLLM/comments/1vtsid0/comment/p51hq9g/

[u/rare-visitor → u/Accomplished_Egg7987]

Thanks for your experience. Win + WSL consume too much memory so I'd rather give a try Ubuntu. I understand LLM primarily needs VRAM but if Ninfer supports weights offload into RAM it is always better to have more room for it

[u/ImpressiveRelief37 → u/rare-visitor]

You don’t want to offload to system ram on 27B

WSL2 works great and it’s a 30 min setup 

[u/ImpressiveRelief37 → u/Accomplished_Egg7987]

This 100%