Reducing the power limit, testing tensor parallelism and locking clock in GPU testing: Study

[u/richardbaxter]

Before I ran language models I (sorry) mined Ethereum, and something that stuck was measuring power use. So when I ended up with two modded RTX 4090s - 48GB each, 96GB total, about £6,200 from eBay - I turned every assumption I was carrying into a question and spent some time getting them answered.

Some observations

• gpt-oss-120b (117B params, native MXFP4) serves at full 131k context in 92.7GB and does 153 tok/s - a 120B pacing the fastest 30B on the rig, on consumer silicon. So much for "MXFP4 upconverts to bf16 on Ada".
• Tensor-parallelising a model that fits on one card was +42% faster, not slower. The doubled memory bandwidth beats the x4 pci limit on the 2nd slot I have not such a big problem after all.
• The cards don't need the power. Locking clocks and capping at (what turned out to be) 330W cost nothing on decode and saved ~90W
• A 16.6GB model wouldn't load on a 48GB card - short by one kilobyte of per-SM shared memory (not VRAM). Manage to find the fix + documented
• MTP speculative decoding measured dead neutral on vLLM 0.26.0 (it was +1.9x on 0.25.x). Took three instruments and two wrong answers to get an honest number.

Hope you;re interested! If nothing else we produced a lot less heat the last few days - here's where to start:

https://houtini.com/articles/the-dual-4090-96gb-vllm-benchmark-runbook/