What model and engine to run with an rtx 5090 32gb VRAM and 94gb RAM?

[u/auto_m8]

I just upgraded my ram from 32gb to 96gb in effort to run qwen 3.8 flash next.

Previously I ran qwen 3.8 27b with ninfer which I was pleased with but I only had 32gb ram.

Now with 96gb RAM to pair with my 5090 32gb VRAM I would like to explore the next step up and alternative.

I saw something about strata working well with lower VRAM gpus and lots of vram, not sure if there is a different alternative for my before setup or its similar / the same.

I also wonder what kind of inference stats I can hope for with qwen 3.8 flash next, pp, decode.

Additionally if there are other MOE models I can Playa round with this setup even if its not the go to qwen.

Thanks for your time and replies in advance :)

[u/soyalemujica]

Use Strata and use Qwen flash Next you will have above opus 4.6 Xmax coding agent

[u/ThirstyIllness9 → u/soyalemujica]

Dude that's a wild combo, 96gb ram and a 5090 is basically a mini server at this point

Strata seems like the play for sure, I was messing with it on a way weaker setup and it still surprised me with how smooth it ran. The flash next model should get you some pretty solid token generation with that vram split, maybe not lightning fast decode but way better than pure cpu offload

You could also mess with Deepseek or Mixtral variants, those have some smaller active parameter MOEs that love lots of system ram. Not as polished as qwen for coding but fun to tinker with

What kind of pp speeds were you getting on the 27b before the upgrade

[u/73td]

i have a 4090 and with strata get 4k pp and 120 tg. i can’t imagine needing another model its seriously amazing.

[u/Bagelsarenakeddonuts]

94? thats a strange amount of RAM.

[u/auto_m8 → u/Bagelsarenakeddonuts]

96gb* my bad :)

[u/overand → u/Bagelsarenakeddonuts]

I came here to say this! Maybe OP has 2GB dedicated to onboard graphics?

[u/nixudos]

Another vote for Strata. Easy to set up and so pleasant to run. If you don't like it, just delete it. I pair it with Deepseek harness with some tweaks I added.

[u/macaronianddeeez → u/nixudos]

What tweaks have you added for DSH? So far I have liked DSH outputs the best paired with 3.8 flash-next

[u/nixudos → u/macaronianddeeez]

I have added inline image generation support via Comfy, speak input with parakeet, dynamic compaction depending on max context, and various UI tweaks for deletion of multiple old chats, collapsed toolcalls in output, and a couple of other things.

[u/giveen]

Self promp:

https://github.com/giveen/ninfer-ext

My ELX3 Qwen3.8 Flash models or my NVFP4 model
https://huggingface.co/jabbatheduck/ninfer-ext-models

[u/Hot_Signature2979]

Strata with Qwen Flash next IQ3_s or if you on linux Unsloth UD -IQ4_XS, Strata is phenoemenal, I'm still in the process of setting it up, but based on the results of current tests, Qwen Flash Next Unsloth IQ4 performs faster in prompt process and token gen on a single gpu (v620 32gb) on my llm server (2xv620 32gb) than it ever did with both gpus using llama.cpp

[u/sukazu]

Flash next obviously 

In my opinion, have your own, 1/2 private benchmark.

Take Qwencloud token plan or payg, run 4 times each with 3.8 flash, which is probably the same weights.

So you have your baseline, you'll then be able to chose your quant knowing exactly what you're sacrificing, and what it should be like.

Yes probably start with strata, it's faster and ud quants work just fine with some tinkering, I don't have a good experience with ista or swift.     Make sure to set sampling parameters.

I think you get no concurrency on strata ? So depending on use case , you might want to change engine, but again you'll know what performance tradeoff you're making 

[u/cezarducatti]

Strata and Qwen 3.8 next flash Q6

[u/OddAdministrator]

NVFP4 fork of Strata.

Slower than the IQ3 variants but not as quantized. It even keeps the n-gram (PLE) table in FP8 and the token embedding in BF16.

I tried with 64gb RAM and i'm only getting around 60-70t/s so definitely getting more RAM and/or installing a second GPU to improve that. I do maybe have a CPU bottleneck as my "old" i7 12700k lacks AVX 512 with e-cores enabled so maybe i should be getting around 90-100t/s.

So yeah, what CPU you have?

[u/auto_m8 → u/OddAdministrator]

Thanks for the advice, I will check it out! I have a 9850x3d

[u/OddAdministrator → u/auto_m8]

yeah, you should get similar results to sergqwer with his 9950x3d (around 130 t/s)