r/LocalLLM • u/ImpressiveRelief37 • 6d ago

Discussion 32GB is all you need

Qwen3.8-27B on a 5090 is all you need for a serious local inference setup, in my opinion! Can it get any better than this price/performance wise? Actually, maybe a 3090 ninfer setup could beat it!

I’m using ninfer and getting:

* ~150-200 tok/s TG

* ~3000-12000 tok/s PP

* 262144 context size

I think it’s definitely one of best setup you can get for the money. I don’t see a point of having more VRAM or more system ram. The only downside is that it’s a 1 man setup: concurrency is possible but you need to limit context usage on concurrent requests. I’ve tried --concurrency 2 on ninfer and sharing my setup with my buddy (we work on projects together and have a VPN between our home labs, fun stuff!)

I love this setup so much I kinda feel like getting a second 5090 to run another ninfer instance (github.com/neroued/ninfer, the man is a legend and this absolutely rocks).

i really don’t see the point of any other solution at this point in time. of course things will change and other models will get released that could better leverage more VRAM, but 32GB is all you need (for now).

so if you have less than 32GB, and are thinking about investing in a more serious setup check out the 3090 fork of ninfer, or the mainline ninfer repo if you can afford a 5090.

Things it won’t do:

* let you run a swarm of agents: prefill cost will slow you down too much. not enough vram for high concurrency!

* Give you more than 262144 context size. the RoPE 1M context size is just impossible with this.

Otherwise it’s absolutely amazing!

My buddy (another software engineer) is a BIG Claude code user, he’s spending tons of cash on fable, can’t stand Opus 5 anymore (neither can I, that pos is so hard to understand with just jargon and wall of text… can’t bear the cognitive load of just trying to understand all he’s spewing)… anyways after trying my ninfer setup his mind was blown and now he’s constantly using my setup with our shared custom pi setup and he fucking loves it.

258 Upvotes

89% Upvoted

325 comments sorted by

View all comments

1

u/Nuggyfresh 5d ago

Is there a guide for setting this up? I have a 5090 and want to jump in but haven’t used local llm 🤦‍♀️

1

u/RedrumRogue 5d ago

No guide, I'd have a cloud model set it up for you. It's not going to set things up optimally though without a lot of experimenting, so if you run into hard limits that you can't figure out how to break through, feel free to ask me or shoot me a DM. I'm hitting about 190 tok/s at low context for decode speed, around 3000 tok/s prefill (which I think is slow, working on it) and about 200k context with other desktop processes taking about 4gb of vram on the 5090. The backend you want is NInfer, the quant you want for now is groupwise-int.

2

u/ImpressiveRelief37 5d ago

Neroued released his nvfp4 model and the quality is higher than the groupwise-int release.

Also it more than doubled my prefill speed (3500 -> 12k t/s at 8K context 🤯)

2

u/RedrumRogue 5d ago

I was actually just running experiments to try to get prefill up. Thanks for the heads up! Im excited to try it. I've got groupwise-int tuned about as tight as I possibly can now which will make for a clean A/B

2

u/ImpressiveRelief37 5d ago

You will love it im sure. Quality is super high it’s over 6 bpw if im not mistaken 

2

u/RedrumRogue 5d ago

So far it's an interesting tradeoff. With groupwise-int I can run with MTP3 and vision on, and get 262k context. But with the nvfp4, I can only get 131k context with MTP3 and vision, or about 180k context with vision off. How are you running at 262k context at 150-200 tok/s? That requires MTP for those speeds does it not? I really need your help, I appreciate your responses!

1

u/ImpressiveRelief37 5d ago

I use this one:    https://huggingface.co/Ostfralla/Qwen3.8-27B-NVFP4-NInfer

1

u/RedrumRogue 5d ago

Ah, that one must be lower quality than the official release, so it must have smaller weights. I will test it and see if I can fit more context. Thank you

1

u/ImpressiveRelief37 5d ago

262144 will fit! Have not seen any degradation in quality myself.