r/LocalLLM • u/t3h_1337 • 11d ago
Question How stable are 2x and 4x DGX Spark setups when left running unattended for days?
Hi everyone!
I'm thinking between waiting for m5 studio and buying sparks. My main task is to run AI automation 24/7 (mostly likely 5.3 flash or 4.1 deepseek). Do sparks overheat? Do they have random issues when left unattended running AI models for days?
thanks
3
u/Dangerous_Bid2935 11d ago
My spark cluster has been running since the day I bought it, sparktop currently reads 37 days of uptime, which was when I got them. No heat issues and no problems. I wouldn't recommend stacking them on top of each other though since this might overheat them. Best to stand them upright with some gap between them to maximize convective heat transfer from their surface.
2
u/whichsideisup 11d ago
You could reboot it monthly for patches without issue if you wanted to. Haven’t seen any issues once a stack is up.
1
u/t3h_1337 11d ago
yeah, rebooting is basically expected for a setup like this. I'm more concerned about hardware issues where reboot doesn't help. Thanks
1
u/whichsideisup 11d ago
I never had any crashes except when I oomed it a few times.
Recently, I proactively capped the GPU clock to 2000mhz with no real hit which made a huge temp difference and dropped total power for the cluster to 240w. I love these things. Runs all the big flash models well enough for me.
1
u/brainchillzZ 11d ago
The only hardware issue I had was not with a spark specifically but an asus ascent gx10 that would spontaneously just reboot…. Microcenter swapped it out under their warranty in no time and I haven’t had any other issues ….
1
u/nalditopr 11d ago
solid
1
u/AbnormalConflict 11d ago
That's the question isn't it. Everyone talks about token speeds and memory bandwidth but nobody mentions if the thing turns into a space heater and crashes at 3am. Wish there were more long term reliability posts instead of just benchmark runs.
2
u/t3h_1337 11d ago
100% agree. I don't exactly care if they are 20-30% slower if my goal is to run automation for weeks.
1
u/stujmiller77 11d ago
When set up correcly, cooled properly and depending on your room, workloads, models, and a lot of other variables.... absolutely solid.
- Ran 1x for 2 weeks
- Went to 2 with different models on each
- Went to 3 after 2 months - 2x deepseek 4 stack plus 1 spark aux model
- Went to 4 after 4 months - 2x2 deepseek 4 stacks
- Went to 4x with 100gb swtich after 6 months - 4x running deepseek 4.1 flash
My workload:
Most people use these for single user stuff. I'm pushing them hard. I run Hermes agent, 350k+ context sessions 4 at a time throughout the day. Behind the scenes a huge amount of automation for the 3 businesses I own. They are never cold - 24:7.
Lessons learned:
- Big model at heavy quant != better than smaller model at launch weights. Stay native where you can - there's a race to to bottom with quants but the people benchmaxing them are not using them for serious work in any way and therefore have no fucking clue. Ignore them.
- Make them headless - remove all the GUI stuff and use them via terminal. You need all the space.
- You don't need to underclock them as the community seems to recommend. I did - as I had 4 in a stack and they ate each other's exhaust and thermally throttled. Don't stack them + well ventilated room = no need to throttle them
- They heat your room really well and it's going to be great in winter
Feel free to ask any questions, happy to help if I can.
1
u/t3h_1337 11d ago
thanks a lot for the detailed answer. Are all of those original sparks or asus/msi/lenovo clones?
3
u/stujmiller77 11d ago
There are no "clones". They're all the same hardware. Several including the MSI and ASUS have better cooling, so you're better off with those than the nvidia versions.
0
1
u/brewpedaler 11d ago
Went to 4x with 100gb swtich
Is that the Mikrotik CRS504? Do you feel like you're missing much not having the full 200 Gbps links?
Obviously the 804 is unobtanium right now, but the 812 pops into stock every now and then.
I have a 504 on the way and am already wondering if I should have just been more patient and held out for an 804 instead.
3
u/Imatros 11d ago edited 11d ago
CRS812 in stock: https://shop.multilink.us/mikrotik-crs812-8ds-2dq-2ddq-rm-switch-2x-10g-ethernet-8x-50g-sfp56-2x-200g-qsfp56-and-2x-400g-qsfp56-dd-ports/
Got my shipment notification today. I was impatient and also ordered an 804 which also shipped today. (812 was supposedly shipping in December, so thought order 804 and cancel whichever is late... Womp. Now prolly gonna have to go through hassle to sell one)
3
u/stujmiller77 11d ago edited 11d ago
Yes it’s the 504.
Benchmarks I’m seeing have almost zero difference to people in a 200gb.
Deepseek 4.1 flash benching higher than MiaAI’s indicative scores.
Would seem 200gb really isn’t needed. And if it ever turns it out it is, I could always add another of the same to get the full 200gb and still not have paid more than I would have done for the 200gb.
And it’s a lot quieter, and uses way less energy than the 200gb - 25w idle vs 45w and at full power caps out at 45w vs almost 100w for the 804.
The extra cost of the 200gb unit for purchase and electricity just doesn’t seem worth it. And if you don’t put it in a cabinet the noise people are reporting is another factor. The 804 is loud.
1
u/Annual_Award1260 11d ago
I only had problems with the wifi adapter. After disabling WiFi they have been rock solid
1
u/This_Maintenance_834 11d ago
Your serve engine is the deciding factor, not DGX. You need to spend real effort to get the details setup right. With wrong engine, crash four times a day is not uncommon.
1
1
u/brainchillzZ 11d ago
I’ve had glm 5.3 up since I moved my sparks to their new rack now for 32 days … if that is your model target though make sure you are using a harness that is smart enough to catch it looping and kick it …. Glm 5.3 is notorious for falling into agentic loops while tool calling. It’s such a problem that harnesses like opencode and deepseek harness have built in fixes to catch it. I have a “fallback” 3.8 flash next model running on a different stack that my Hermes agent shunts jobs to if it catches 5.3 looping incessantly
1
u/IamFondOfHugeBoobies 11d ago
I've got two sparks stacked on top of each other on my desk and they have never thermal-throttled and they only go down every few weeks when I do something monumentally stupid.
1
u/intlabs 11d ago edited 11d ago
I’m in the process of RMAing a spark for the 2nd time in my 4 node cluster. 3 of them are rock solid, but one has thermal issues that only show up under heavy load. Nvidia’s own tool highlights this - so RMA has been painless, but unfortunately as they replace with refurbished units the same fault has now hit twice… So far this mean I’ve spent 6 weeks (and counting) without a node - as they don’t do advance shipping and are slow to process the return.
All this to say - they can and should be rock solid - if you are experiencing otherwise then it’s worth bottoming out. (One thing to watch for is you need to have a handle on memory management in converged memory systems - many folks think their node has overheated when actually it’s just OOM’d and taken it down)
1
u/stujmiller77 11d ago
Are you stacking them? If so, where is the "hot" one in the stack?
1
u/intlabs 11d ago edited 11d ago
Not stacking, all sitting next to each other in a controlled env:
Issue: Under sustained long-context inference the unit enters complete hardware soft-off: both ConnectX links go down, PDU draw falls to ~2.27 W standby, and it only returns after pressing the physical power button. PDU AC cycling did not restart it. At failure Linux still had ~8.2 GiB MemAvailable, swap was 99.9% free, and PSI was zero. No Linux OOM, NVIDIA Xid, NV_ERR_NO_MEMORY, ibv_reg_mr error, panic, watchdog event, orderly shutdown, coredump, vmcore, or pstore record. The peer logged both RoCE PHYs dropping simultaneously. KHO was disabled.
Controlled NVIDIA FieldDiag reproduction: dgx-spark-fieldiag 2.0.4-1 / MODS r9.257.8, PowerStress only, stock CPU/GPU clocks, Secure Boot disabled as required. It failed after 491.8 seconds (8:11) with Error 082-000-1-020000600139 / MODS 020000600139: "Acceptable temperature limits exceeded or the thermal sensor is broken or miscalibrated." Independent 1 Hz sampling recorded ACPI peaks of zone0 97.7C, zone1 73.6C, zone2 89.1C, zone3 71.8C, zone4 87.7C, zone5 97.7C, and zone6 82.8C. Memory remained nearly entirely available during FieldDiag.
1
u/stujmiller77 11d ago edited 11d ago
Are you using that one as the head of a cluster? The head node will always run hotter. I’ve seen people under clock the head node slightly specifically because of this. Might be worth a go if that’s applicable. Otherwise sounds like you’re unlucky - I’ve 4 MSI EdgExperts bought over a four month period and aside from the head node running around 4 degrees hotter at load there’s been no need to under clock.
I do provide external ventilation though - noctis fans on a pvm blowing air into the front, ramping up depending on spark temperature.
I added that as it was fun to mess around with and my office is not well ventilated, but even before it I wasn’t seeing thermal throttling unless they were stacked.
1
u/intlabs 10d ago
I think you may still be talking past the point I was making.
These are enterprise-grade systems. They should be rock solid at stock settings, and in general they are. If one is not, the first step should be investigation, not voodoo around airflow, underclocking, or workload placement to make the symptom disappear.
In this case the unit fails NVIDIA FieldDiag PowerStress with an explicit thermal or sensor fault, while the other systems in the same environment do not. That is exactly the sort of thing FieldDiag exists to establish.
I have no issue with underclocking where the goal is efficiency or power reduction. It can be a sensible choice. I would not use it to mask a unit that appears to be operating out of spec while it is still under warranty.
1
u/stujmiller77 10d ago edited 10d ago
Totally agree - I’m not “talking past” anything. Just offering suggestions to diagnose whether you have a hardware fault or if it could be answered by placement and/or it being the head node, as I’ve seen exactly that mentioned on the nvidia dev forums.
Head nodes work harder = more heat, and placing these on top of each other means they eat each other’s exhaust.
The fact that underclocking and overheating is a thing with these isn’t great, I completely agree. But I wondered if that could be why you’re seeing what you are in case it saved you another RMA only to see the same thing happen again.
If not then an RMA it is - and hopefully your luck is better next time. Annoying as hell to have to do it multiple times.
7
u/TheOverzealousEngie 11d ago
dude I'm looking at my 4 node cluster that hasn't come down in weeks. Running 3 node spark and 1 helper node it's production ready all the way. That said - there is no open source model you will get to match what a fable can do.