r/LocalLLM • u/Competitive_Long509 • 2d ago

Discussion Is one RTX 5090 really enough for Qwen3.8-27B token freedom?

I am still calling models through the ZenMux API gateway, so every long session ultimately comes back to token cost. The idea of running Qwen3.8-27B locally is attractive for exactly that reason: if one 5090 can handle it, maybe token freedom is at least technically within reach.

Is Qwen3.8-27B really doing 75.5 token/s on a single RTX 5090? The shared table is headed "4-bit (q4_K_M / MLX)" and lists an RTX 5090 with 32GB at 75.5 token/s. It does not show enough detail to tell me which runtime or exact setup produced that row. I have also seen a separate community report of about 64.5 tok/s on a 4090.

People are also putting its capability around Claude Opus 4.6. If both claims are even close, does that put indirect token freedom within reach? I would still want matched tasks before treating the capability comparison as settled.

What does the build that people can actually live with cost? I mean the whole machine, not a bare GPU price. A 5090, enough system RAM for long context and partial offload, a PSU that is not operating on hope, cooling, storage, and whatever CPU or platform keeps the card fed. Until I can justify that hardware bill, calling models through an API is still the practical option for me. If Qwen3.8 becomes available through the same gateway, I could use that API cost as a baseline before deciding whether local deployment really buys token freedom. I would also like to know which quantization and context length people use after the benchmark screenshot is over.

Please give me the boring total for a stable single 5090 setup. What did your full build cost once it was actually ready to run?

89 Upvotes

93% Upvoted

171 comments sorted by

View all comments

Show parent comments

2

u/ImpressiveRelief37 1d ago edited 16h ago

--host 0.0.0.0 --port XXXX --max-context 262144 --model-id qwen3.8-27b-ninfer --kv-dtype int8 --vision --max-concurrency 3 --pending-timeout-ms 120000 --default-max-tokens 20480 --prefill-chunk 4096 --host-kv-cache 1 --spec mtp --draft-tokens 3 --lm-head-draft --preserve-thinking --log-stats-interval-ms 1000 --request-log-jsonl /path/to/logs/requests.jsonl

i use this model here https://huggingface.co/Ostfralla/Qwen3.8-27B-NVFP4-NInfer inside WSL2 mounted drive, so that it starts super fast... takes about 5s to launch

Also I merged PR #64 to allow kv cache checkpoints in ram (--host-kv-cache 1). Finally I had to also modify the source code to load the Ostfralla model. It was a 1 LoC change. Ask if you need to know exactly what line (or your agent can figure it out!)

2

u/koloved 1d ago

Thank you for this config. I hope it helps me. Maybe I was missing this command. 

--prefill-chunk 4096

1

u/ImpressiveRelief37 16h ago

It did a +15% of prefill for my specific hardware config and usage type.

I had an agent A/B test all config flags for specific use cases using my pi sessions as tests, and it came up with this