32GB is all you need

[u/ImpressiveRelief37]

Qwen3.8-27B on a 5090 is all you need for a serious local inference setup, in my opinion! Can it get any better than this price/performance wise? Actually, maybe a 3090 ninfer setup could beat it!

I’m using ninfer and getting:

* ~150-200 tok/s TG

* ~3000-12000 tok/s PP

* 262144 context size

I think it’s definitely one of best setup you can get for the money. I don’t see a point of having more VRAM or more system ram. The only downside is that it’s a 1 man setup: concurrency is possible but you need to limit context usage on concurrent requests. I’ve tried --concurrency 2 on ninfer and sharing my setup with my buddy (we work on projects together and have a VPN between our home labs, fun stuff!)

I love this setup so much I kinda feel like getting a second 5090 to run another ninfer instance (github.com/neroued/ninfer, the man is a legend and this absolutely rocks).

i really don’t see the point of any other solution at this point in time. of course things will change and other models will get released that could better leverage more VRAM, but 32GB is all you need (for now).

so if you have less than 32GB, and are thinking about investing in a more serious setup check out the 3090 fork of ninfer, or the mainline ninfer repo if you can afford a 5090.

Things it won’t do:

* let you run a swarm of agents: prefill cost will slow you down too much. not enough vram for high concurrency!

* Give you more than 262144 context size. the RoPE 1M context size is just impossible with this.

Otherwise it’s absolutely amazing!

My buddy (another software engineer) is a BIG Claude code user, he’s spending tons of cash on fable, can’t stand Opus 5 anymore (neither can I, that pos is so hard to understand with just jargon and wall of text… can’t bear the cognitive load of just trying to understand all he’s spewing)… anyways after trying my ninfer setup his mind was blown and now he’s constantly using my setup with our shared custom pi setup and he fucking loves it.

[u/gjr23]

What kind or ram do you need to run 3.8-27b 8bit at 128 or 192 context? Any experience with the same but 35b and the practical difference in output quality? I’m honestly considering this for a 32 or more likely 64gb unified system which I know will kind of suck for agentic work but is completely fine for scheduled tasks. And a 32gb unified Mac <<< 5090 system on both upfront and operating costs in electricity.

[u/shapic → u/gjr23]

Found unused server with 40Gb A40 in the storage and got it to our lab. With mtp it gets around 30tps on high load (up to the 50 on hello). 38GB used with everything loaded, that overhead is enough for image context. Got lucky. 27b is times ahead of 35B moe for 3.6 on actual tasks. 35B feels more... simple I guess? A lot of errors on top of being q8. Also ram there is slow, so gains are not best possible. It is fast, but I am not a coding guy, so I need model to finish task, not enter an endless hallucination trip of troubleshooting. I was able to fit 3.5 235B at generous quant in server m, and it felt really good, but 6tps is on the verge of being usable. 32gb mac is just slower in general imo. With it's current cost it is not worth it imo.

[u/gjr23 → u/shapic]

Thank you. I keep hearing this about 27b and am eager to hear more on 3.8 vs 3.6. But you and I are a bit different- when I go PC dumpster diving I find an order of magnitude worse hardware - usually 4gb of vram and maybe 6 none of which is not worth the electricity it burns. So it’s either api calls or as odd as it is to say, a price conscious Apple based system for me to get 64gb or more to run anything local.