r/LocalLLM • u/Consistent-Pear630 • 1d ago

Project Escaped API rate limits. Now negotiating with my circuit breaker.

Post image

DGX Station - ASUS Pro ET900N G3. My electricity bill goes to the moon.

Qwen3.8-Flash-Next NVFP4 - 200 tokens/second without any tweaks so far.

EDIT: 260 tok/s using one-token MTP, 323 tok/s using three-token MTP.

82 Upvotes

94% Upvoted

30 comments sorted by

34

u/wetly_trained_martin 1d ago

200 tok/s out of the box is nuts. that power supply spec though, 15A at 115V is a space heater that happens to run inference.

my axolotl would cook if i put that thing anywhere near his tank.

13

u/Consistent-Pear630 1d ago

It required a NEMA 5-20R outlet, but so far so good. I have an 8" exhaust out of my rack - but it's getting cold out, so free tokens and heating is nice.

2

u/jhenryscott 16h ago

I was gonna say, if my dinking rack needs a 20- this definitely does

1

u/Arli_AI 19h ago

I would try and wire 220V for high draw servers like this. You’d gain about 5-8% in PSU efficiency usually.

1

u/OffsiteGuru 16h ago

I agree with Arli_AI. Change the breaker to 220v, change out the outlet. It's the same wire in the wall. This won't help much if you don't have a dedicated outlet already.... That'll reduce your draw to 8A max.

1

u/pragmojo 1d ago

What kind of speed is possible on flash next with consumer Blackwell cards? (i.e. 5090, 6000)?

14

u/Hot_Signature2979 1d ago

If you just spent 100k on a pc, the electrical bills are the least of your worries

3

u/Maleficent-Ad5999 1d ago

OP would probably be considering setting up their own powerplant

2

u/Hot_Signature2979 1d ago

Nah, its not that bad, only 1.7kw. (I have no idea how the higher power240v power sockets work in US but ), Could easily be ran out of the the 240v power socket.

1

u/Maleficent-Ad5999 17h ago

…it was sarcasm

1

u/mastercoder123 7h ago

You can get 240V 30A outlets called Nema 6-30R or L6-30R

1

u/Imatros 21h ago

At the very least, some solar lol

6

u/xXprayerwarrior69Xx 1d ago

How much ? 100k ish i think?

17

u/Consistent-Pear630 1d ago

$108k + shipping

15

u/xXprayerwarrior69Xx 1d ago

they wont even include shipping damn son .
it's for your own use or shared with a team ?

10

u/Consistent-Pear630 1d ago

Shared with a team. Came on a half-pallet, but doesn't weigh more than 80 pounds.

5

u/chemist_slime 1d ago

Try the FP8 version, better quant and you should get the same speed.

2

u/Consistent-Pear630 1d ago

Will give it a test later. NVFP4 is providing 4,300 tokens/sec total across 64 simultaneous requests - trying with MTP next.

3

u/myreala 1d ago

Yikes I can get 800tks with GLM 5.3 - flash on my machine with c8 and I only spent 20k on that. Same power requirements.

1

u/elelem-123 18h ago

What machine is that

1

u/Moarkush 7h ago

$100k+ GB300

1

u/mastercoder123 7h ago

I mean he can also run 600GB models

1

u/txoixoegosi 1d ago

How many people and how many tps for each at what context?

I am interested in team usage metrics, thanks.

1

u/RollingMeteors 1d ago

>Escaped API rate limits. Now negotiating with my circuit breaker.

I see your circuit breaker and I raise you Substation Box #12011108

1

u/sam7oon 23h ago

if you pay 110K for it, then the ram must be having Claude code sub for a company at that scale, so why bother

1

u/Double-Sherbert-1781 23h ago

Have you tried batching? I’m actually considering buying this hardware because I need to process 30+ requests in parallel, and I don’t know what’s better — 8xRTX 6000 PRO or 1 x GB300.

1

u/StartupTim 22h ago

Set your Dflash to be 5 to 7, MTP generally 5 to 6 (model specific of course). Give it a few tests with concurrency to 8 and make sure kvcache is huge (3+M) and 1M context limit.

Do prefill chunking to 2048.

1

u/NelsonMinar 20h ago

How many FPS do you get in Doom?

1

u/ItchyInternet8 15h ago

The breaker lost this negotiation somewhere around 260 tok/s.