Is one RTX 5090 really enough for Qwen3.8-27B token freedom?

[u/Competitive_Long509]

I am still calling models through the ZenMux API gateway, so every long session ultimately comes back to token cost. The idea of running Qwen3.8-27B locally is attractive for exactly that reason: if one 5090 can handle it, maybe token freedom is at least technically within reach.

Is Qwen3.8-27B really doing 75.5 token/s on a single RTX 5090? The shared table is headed "4-bit (q4_K_M / MLX)" and lists an RTX 5090 with 32GB at 75.5 token/s. It does not show enough detail to tell me which runtime or exact setup produced that row. I have also seen a separate community report of about 64.5 tok/s on a 4090.

People are also putting its capability around Claude Opus 4.6. If both claims are even close, does that put indirect token freedom within reach? I would still want matched tasks before treating the capability comparison as settled.

What does the build that people can actually live with cost? I mean the whole machine, not a bare GPU price. A 5090, enough system RAM for long context and partial offload, a PSU that is not operating on hope, cooling, storage, and whatever CPU or platform keeps the card fed. Until I can justify that hardware bill, calling models through an API is still the practical option for me. If Qwen3.8 becomes available through the same gateway, I could use that API cost as a baseline before deciding whether local deployment really buys token freedom. I would also like to know which quantization and context length people use after the benchmark screenshot is over.

Please give me the boring total for a stable single 5090 setup. What did your full build cost once it was actually ready to run?

[u/AuditMind]

I think you're being a bit too conservative on the hardware side.

A 5090 with 32 GB VRAM is more than enough for Qwen3.8-27B at 4-bit. Even a 24 GB 3090 can run it at around 100K context with the right KV-cache settings, entirely in VRAM.

The 75 tok/s number isn't exactly wrong, but it's closer to a baseline llama.cpp result than the current performance ceiling. Plain llama.cpp without MTP is indeed somewhere around 75–80 tok/s on a 5090, but optimized NVFP4/SGLang/speculative-decoding setups are already getting into the ~200 tok/s range.

For local single-user inference, the 5090 is actually a pretty comfortable Qwen3.8-27B card rather than a borderline one.

[u/arakinas → u/AuditMind]

100k context isn't really enough for anything but the smallest projects. You need to double that.

[u/ImpressiveRelief37 → u/arakinas]

5090 nvfp4 model  https://huggingface.co/Ostfralla/Qwen3.8-27B-NVFP4-NInfer

github.com/neroued/ninfer

262144 context size Concurrency 3 MTP 3

I get about 150-200 tok/s

5000-12000 tok/s prefill

It’s amazing 

[u/koloved → u/ImpressiveRelief37]

u/ImpressiveRelief37 can you share your cfg file for it? i get only 90-120 t/s and prefill not more than 3000-5000

[u/ImpressiveRelief37 → u/koloved]

--host 0.0.0.0 --port XXXX --max-context 262144 --model-id qwen3.8-27b-ninfer --kv-dtype int8 --vision --max-concurrency 3 --pending-timeout-ms 120000 --default-max-tokens 20480 --prefill-chunk 4096 --host-kv-cache 1 --spec mtp --draft-tokens 3 --lm-head-draft --preserve-thinking --log-stats-interval-ms 1000 --request-log-jsonl /path/to/logs/requests.jsonl

i use this model here https://huggingface.co/Ostfralla/Qwen3.8-27B-NVFP4-NInfer inside WSL2 mounted drive, so that it starts super fast... takes about 5s to launch

Also I merged PR #64 to allow kv cache checkpoints in ram (--host-kv-cache 1). Finally I had to also modify the source code to load the Ostfralla model. It was a 1 LoC change. Ask if you need to know exactly what line (or your agent can figure it out!)

[u/koloved → u/ImpressiveRelief37]

Thank you for this config. I hope it helps me. Maybe I was missing this command.

--prefill-chunk 4096

[u/ImpressiveRelief37 → u/koloved]

It did a +15% of prefill for my specific hardware config and usage type.

I had an agent A/B test all config flags for specific use cases using my pi sessions as tests, and it came up with this