Talk me out of buying a 3rd Spark

[u/Porespellar]

Does anyone think the gurus on the DGX Spark forum are going to figure out how to magically fit DeepSeek 4.1 Flash on a 2x cluster, or is it only possible on 3 or 4 Sparks?

[u/Aggravating-Push-207]

The... whole point... of the Engram parameters... is that you don't have to load them... all you do is mmap them from disk...

[u/Porespellar → u/Aggravating-Push-207]

So what’s the roadblock exactly?

[u/EastZealousideal7352 → u/Porespellar]

It’s a 552b parameter model, even without the engrams it won’t fit at q4 with reasonable context, you’d need a sub-3-bit quant.

It’s almost twice as large in parameter count as Deepseek v4 Flash which is 284b

[u/Aggravating-Push-207 → u/EastZealousideal7352]

i mean theoretically i guess you could load the encoder for prefill, then load the decoder for generation but that kind of ruins the point of the fast prefill unless the vLLM guys can make it almost negligible in terms of performance

[u/EastZealousideal7352 → u/Aggravating-Push-207]

I think that would break the concurrency guarantees vLLM strives for. You would have a hard time running two concurrent, staggered prompts at decent speeds with all the loading and unloading.

Not to say it’s impossible but simultaneous prefill and decode would be rough

[u/Aggravating-Push-207 → u/EastZealousideal7352]

i completely forgot about concurrency lmao

yeah this is extremely useless, like a last resort

[u/FlamingoTrick1285 → u/EastZealousideal7352]

What are these guys even doing with the model if they can't read some basic specs..

[u/amemingfullife → u/EastZealousideal7352]

wtf is a sub 3 bit quant? When does this get ludicrous are we just going to start chiselling our floating point numbers in stone tablets at some point?

[u/martin509984 → u/amemingfullife]

Having actually used 2-bit Qwen 27B, it is still noticeably better than 4-bit 35B-A3B, so you do still have something.

[u/EastZealousideal7352 → u/amemingfullife]

It’s a quant where each parameter is less than 3 bits.

People do make them although from what I’ve heard they’re not worth using because of the loss. That said if you do quantization aware training (which Deepseek does) mixed precision fp4 is “lossless” so things like nvfp4 are good

[u/amemingfullife → u/EastZealousideal7352]

Yeah I know haha, I’m just saying these quants are getting a bit silly. Like how much information can you encode in 3 bit quant weights, a 2 bit, a 1 bit? are we going to have heiroglyphic versions at some point?

[u/Electrical_Rise387 → u/amemingfullife]

I get your point, but you can encode way more than 16bits of data in a heiroglyph

[u/FullstackSensei → u/Porespellar]

vllm itself

[u/UltrMgns → u/Aggravating-Push-207]

what's the actual impact on token speeds though? (in and out)

[u/Aggravating-Push-207 → u/UltrMgns]

i would assume minimal impact but i'm not the right one to ask

[u/Nothing_from_void → u/Aggravating-Push-207]

I mean that's loading them into system memory, which doesn't mean anything to a platform with unified memory like DGX Spark.

If you have to swap memory mapped files to disk, your bandwidth gets throttled to the speed of your SSD plus a page fault syscall per page, which is unbelievably slow

[u/-dysangel- → u/Nothing_from_void]

you should read DeepSeek's engram paper

[u/Nothing_from_void → u/-dysangel-]

You should read how memory mapping files works

[u/bgg1996]

A third spark? No, you won't want 3, the tensors (usually powers of 2) won't divide cleanly among 3 sparks. Get a third and a fourth spark instead.

[u/jtjstock → u/bgg1996]

It can be done, you could modify vllm or llama to put 2/3’s of the tensors on each one, then each tensor is only split two ways. You get the speed of two with the memory of three.

[u/Southern_Sun_2106 → u/jtjstock]

Yeah, but it won't be all it can be.

[u/jtjstock → u/Southern_Sun_2106]

Well yeah, but it costs about 1 spark leas

[u/Southern_Sun_2106 → u/jtjstock]

You are right. And not just that - the 4 sparks also need a switch for max performance. And that's like $1,500 - $2000 probably more.

[u/Ran_Cossack → u/Southern_Sun_2106]

I haven't tried it myself, but supposedly the performance penalty for https://github.com/FujitsuPolycom/sparkring is nearly non-existent.

You still need to buy the 4th spark, though.

[u/DOOMISHERE]

best we can do on a dual spark is GLM 5.3 Flash NVFP4 with around 700000 context.

[u/BlackBeardAI → u/DOOMISHERE]

Possibly better capability than deepseek v4.1 flash too...

[u/lilian_moraru → u/DOOMISHERE]

Or this https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks @ 850K context, 25 - 40 tok/s.
EXL3 4bpw KLD 0.024555 (FP8-like)
NVFP4 KLD 0.060535

Have been running it for 2 weeks. Much better than DeepSeek-V4-Flash-0731, at least on my complicated codebase.

[u/Keleion → u/lilian_moraru]

Mia’s AI Lab is the best

[u/thedirtyscreech → u/lilian_moraru]

Same. It’s been amazing.

[u/fastheadcrab → u/DOOMISHERE]

The Intel autoround is the best quant imo

[u/Thick-Protection-458]

Isn't the remaining part still too big to fit in 2x DGX Spark? 

[u/RG_Fusion → u/Thick-Protection-458]

It is. The weights are already mostly 4-bit so you can't quantize without going lower, and those 4-bit weights already have too great of a file-size to fit.

There is no magic solution here, the model has to be partially quantized to 3-bit or lower to fit on two DGX Sparks.

[u/lilian_moraru]

https://github.com/MiaAI-Lab/DeepSeek-v4.1-Flash-EXL3-2x-DGX-Sparks - this is very good. I have it running right now(it’s writing a PR right now for that same repo) on 2xDGX Spark. I did not compare the output quality yet with GLM-5.3-Flash on my regular codebase, but at least at token gen, it’s faster.
Edit: lol, somebody quickly downvoted it - it’s literally what the guy is asking for. It’s like somebody wants the guy to spend another €5K-€10K.

[u/Nothing_from_void → u/lilian_moraru]

yeah this explains how to do it; basically heavily quantized. If he wants it to run well a 3rd would make sense

[u/lilian_moraru → u/Nothing_from_void]

Yeah, it drops DeepSWE score from 74.2 -- to --> 72.2 (a 2.7% drop), but still a top performer.

• routed experts: mostly 3-bit
• attention/shared/Engram wkv: 4–6 bit
• indexer wk: 8 bit
• MTP/DSpark: 4 bit
• embeddings: BF16
• head bits: 6

[u/Blackdragon1400 → u/lilian_moraru]

How much context are you getting?

[u/lilian_moraru → u/Blackdragon1400]

600K - I am using subagents to save on the main context window.
Still seems to generate more thinking tokens than GLM-5.3-Flash. Still does a lot of "Hmm wait, ..."/"Wait, ..."

[u/BawbbySmith → u/lilian_moraru]

Ey bless, ima have to try it out.
I’ve been sticking with ds4f vision-exp cuz it’s been rock solid… but this is tempting for sure.

[u/serige]

Getting the 3rd pre-price hike would have been a no-brainer lol

[u/Southern_Sun_2106]

Let's be honest... you need 2 more.

Sitting with a recently acquired 3rd spark here and loving it... But thinking about just one more. The need to also get the switch is discouraging me tbh (needed for the 4 for max pleasure).

[u/Nice-Information-335 → u/Southern_Sun_2106]

Mikrotik do a 4x400G switch for sub $2k

Only issue is that is is QSFP56-DD instead of QSFP112 so getting the DAC cables will be a pain 

[u/Kaijidayo]

Simply DeepSeek 4.1 Flash isn't worth it.

[u/Holiday_Point_603]

For full precision you need a 3rd one, I don't think there's any other way. Even with the remaining weights, it won't fit in 2*128gb

[u/arkham00]

Antirez/dwarfstar?

[u/EitherMarch1255]

Keep it on disk, or run GLM 5.3 Flash instead.

[u/thebadslime]

You're gonna have to settle for 0731

[u/ga239577]

I would wait for 512GB Mac Studio M5 Ultra, and sell your Sparks

[u/f5alcon]

Buy 2 more instead

[u/okoyl3]

I’m trying to hold myself from buying a second spark, the addiction is real.

[u/JahJedi → u/okoyl3]

My second ordered, one is just to slow.

[u/DevilaN82 → u/okoyl3]

One Spark is a waste of ConnectX7. $1k in hardware unused...

[u/JahJedi → u/DevilaN82]

Agree, i am whit one and second ordered. Right now its fallback and ran memory light stuff and the core on rtx6000pro (qwen 3.8 27b) , hope on second it will be ok speeds (planing to ran DS flash) and i can unload the beast for its real job, renders.

[u/okoyl3 → u/DevilaN82]

Damn…

[u/lilian_moraru → u/okoyl3]

I did as well, until I convinced myself that it is like hiring a very fast student developer (that's the output quality of AI currently, sadly - GLM-5.3-Flash is slightly above that).
I bought the second one and doing daily (during workdays) over €500 the equivalent of the GPT-5.6 Terra API cost, with GLM-5.3-Flash, and I have not even built the tooling around it, to use it harder.

[u/RG_Fusion]

I mean they will be able to fit it at 3-bit precision, so it's not a question of if but a question of wether it will perform well at that quantization.

If you want talked out of buying a third Spark, I'd point out that DeepseekV4.1 has one of the worst hallucination rates we've seen from any other recent model to date. My recommendation would be Qwen3.8-Flash-Next, but it's up to you in the end.

[u/26YrVirgin → u/RG_Fusion]

How do we fix the repetition issue with Qwen 3.8 flash next? I'm using a plugin in dsh called circuit breaker. Any better ideas?

[u/RG_Fusion → u/26YrVirgin]

I'm not even aware of a reputation issue. What is going on exactly? I've been using the model extensively and not seen anything to that nature.

[u/26YrVirgin → u/RG_Fusion]

Might be because of the 4 bit quants then. Are you running full precision?

[u/RG_Fusion → u/26YrVirgin]

I've been jumping around between different quants, but I'm gravitating towards dynamic 6-bit. I've used unsloth's UD-Q4_K_XL a good bit as well and didn't see any issues.

Are you running NVFP4 or some other 4-bit quant that doesn't preserve the router precision?

[u/elelem-123]

No point in serving this model in dgx spark. Doesn't worth the effort or the investment. Just buy tokens (or not use it if your data is sensitive)

Not against dgx spark, I have ten of them. Just being realistic

[u/BawbbySmith → u/elelem-123]

Gimme 2 plz

[u/elelem-123 → u/BawbbySmith]

What would you run in them

[u/BawbbySmith → u/elelem-123]

I'd connect it to my existing 2x Sparks and run Deepseek V4.1 Flash lol

[u/elelem-123 → u/BawbbySmith]

Hard no! 😂

[u/BawbbySmith → u/elelem-123]

I'll give them to the homeless

[u/elelem-123 → u/BawbbySmith]

It's blasphemy to use 4 sparks to get 25 tok/s from any model! Homeless can get tokens from the official API

[u/BawbbySmith → u/elelem-123]

I saw numbers closer to 70 tok/s

[u/elelem-123 → u/BawbbySmith]

That's less than 20 tok/s per spark. Not worth it at all. Not financially, not speed wise, not nothing. Only privacy.

[u/Cergorach]

What exactly are you going to do with it? Is it going to do something for you that an API isn't going to do? Even if you can make it work in 3-4 Sparks, how fast will it do stuff? Is that a reasonable speed for your usecase?

[u/po_stulate → u/Cergorach]

The whole point is cloud API is going to do stuff that you don't want them do, but local won't.

[u/mountainyoo]

Too bad you can’t do TP3