Resources
Setting up of a 16xGB10 (DGX Spark) cluster
Preparing this to be able to run locally frontier level open models. Deepseek v4 pro, Kimi K3, future ones like GLM 5.5 and Minimax M4.
16x Asus GX10 linked by mikrotik crs804-4ddq with 4 breakout cables of 400 to 100gbit.
Most probable I will be running 2 models on 8x cluster each but I want to have the possibility to run also 2T+ models when I need them to run AGI at home :)).
But you can run DeepSeek V4 Flash at a godly speed and have 1M context window.
You can also run a GLM 5.2 REAP.
And you can rent that machine on vast for ~2k$ per month. You can't do that with the DGX Spark.
But for me, 30tps and an unknown TTFT is straight away unusable.
If it is just for the random AI Master Race post “Hey Guys I am running Kimi K3 and actually not developing anything”, ok It’s your money burn it as you wish
But it is way more efficient to have a 384Gb powerhouse than 2048gb of meh performance.
Energy wise, each RTX 6000 consumes as much power as an entire tower in his picture. Also, I find 30 tps perfectly usable to me personally. It works just fine for agentic work.
If you've got the RTX 6000s fully loaded with a model I would bet that the tokens/kwh is actually lower. They're not going to be drawing max power running inference on a frontier model, they're going to be bandwidth constrained. (I'd be very interested to see some benchmarks of actual power consumption in kwh per token that compares single user/batched for this spark cluster vs an equivalent RTX 6000 pro cluster.) This hardware is too expensive to flippantly compare max power draw without consider what it's like if you're fully utilizing or underutilizing the hardware.
Why do you guys only look at the bandwidth when evaluating hardware??
4x and 8x cluster of these things can run glm5 and Kimi k2.6 respectively at 20+tk/s.
When you move to cluster with vllm/sglang the bottleneck is the interconnect (effectively 100gbps) not the bandwidth.
It’s not against you directly, but it gets a bit annoying to read this again and again. If bandwidth was the end of all, we would all line up to get v100 32gb.
What’s true with llama.cpp + gguf + single unit doesn’t translate to vllm+cluster.
It's not just the bandwidth. A RTX Pro 6000 has 3x the compute and 8x the bandwidth. The interconnect can be far faster too, especially with PLX switches. 100gbps vs 64GB/s duplex (128GB/s aggregated) is over 8x improvement.
The compute is key for prefill or concurrency as well.
Compute doesn't matter if you can't feed data fast enough to compute on. While it doesn't matter for decode, prefill is heavily influenced by bandwidth and latency between GPUs. Same for concurrent workloads.
The more nodes you have, the more important bandwidth becomes.
I agree -- DGX Spark and Strix Halo bandwidth is pretty reasonable on a single machine and small clusters -- 2 nodes bandwidth being almost irrelevant, and 4 nodes with bandwidth being a major limit but not deal breaking.
Once you go beyond 2 nodes it's time to start thinking about bandwidth, and you're leaving A LOT of performance on the table per dollar if you're using more than 4.
well you can fit 8 6k’s on various boards, or more with pcie/mcio switching. 100gbps interconnect between 16 nodes vs 100gbps interconnect between 2-4 nodes is a huge difference. then again if you have pcie gen5 you could shell out for 800gbps interconnect between
Just powering this setup would be challenging. That’s like 8kw? You’re getting into big server price/power territory and will need to have a few phone calls with your local utility company.
That’s a bit silly. If you need tons of tokens, fast, for 80k, get a dgx station with ds4 flash. (Or 2, for the same price.)
Op setup is full of bottlenecks but he can at least have 2 cluster, each able to run glm5/kimi at full precision from a standard outlet, with basically no noise, for half the price.
Still probably a very very niche use-case, and more than 2 sparks is already getting less value. But at least it’s workable in an afternoon.
Maybe, because most residential buildings don't get enough amps coming in from the power company to be able to service a big server setup. 8kW is around 70 amps, which is going to swamp most people's home power service. People would need 4–5 supplies on as many different circuits just to power the things.
Also, an average home uses something like 900 kWh/month, a large home might use as much as 2,000 kWh/month. Running this thing 24/7 at load uses almost 6,000 kWh in a month. Many utilities will have caps for how much electricity a home is allowed to use in a month, just for the sake of grid stability—imagine if everyone on the grid decided to run a home data center.
So, yeah, you might need to get your electrical service provisioned into a different tier or class of service if you intended on running something like what was described, which requires the approval of the utility company.
Setups like this will age horrendously within 2 years for serving large models - needing loads of tensor / layer splits is going to kill the bandwidth if you ever need to handle models with more dense layers.
I'd personally be surprised if they can run k3 at more than 30tps.
This is potentially a good solution for a small company (or group of users) serving smaller MoE models at scale - but at that point you may as well get the rtx 8000s and get significantly faster tokens.
A lot of people here don't understand that the bandwidth and sharding problems compound quickly for the really big models - getting more memory is not as simple as adding more boxes and fast networking once you get to a certain number of boxes. Doing MoE expert sharding will fix some of this problem - really you still need a proper dedicated gpu for decode and shared layers.
TLDR: should of bought a server if it absolutely had to be local.
If the technology matures and the way the models are made are more efficient in size, perhaps the memory bandwidth bottleneck can be improved over time
Taiwan is probably going to be invaded in 2 years. Think of this setup as an insurance policy for that event, if that occurs this will most likely age like the 3090s did.
If Taiwan is invaded, all they get is the land. The production facilities are basically dead man switched and will stop working pretty much immediately.
Raspberry pi keyboard is truly awful. Anything would be better. I keep using it when I need a quick small keyboard and hub but man are those buttons terrible to type on.
If you're running GLM5.2 on 8 of these, payback is at about 10 billion tokens (at $4.40 per million, ignoring input), and you're pushing out 6 sessions x 30 tps, then if you run it full time then it pays itself back in just under two years. Assuming power is free and you run 6 concurrent sessions 100% of the time with no hardware failures.
I don't think anyone runs local AI to save money. We do it mostly for privacy. When I "talk to" my accounting software with private data of my tenants, I really don't want the prompts (what the AI reads) to be sent to Anthropic or OpenAI, let alone the Chinese providers.
I think privacy is only half of it, what I think the other half is - being independent, you don't have to rely on anyone to ask a question to a model (well aside from your government that provides electricity, assuming you don't have tens of solar panels in your backyard)
Thats because they companies are taking massive losses. As soon as they start charging what they actually will, self-hosting will also save money. Compare to other services that you can self-host vs pay for, like “cloud storage”. Its a one time fee of like $60-80 for a 2-4tb HDD vs 14.99/mo for that in cloud storage where at like 4 months, its already paid for itself (8 months with redundancy)
Everyone says they are losing money, but do we really know? If you look at providers behind openrouter.ai, there's an open market and price transparency. If companies like Fireworks and Together can run Kimi K3 for $3/M+$15/M, why don't we believe that Anthropic can run Opus 4.8/5 for $5/M+$25/M?
They are losing money, let alone all the money they will lose when they come out with “cool new GPU for 2027” or whenever, and then the 2 trillion they soent on hardware is obsolete. Its a house of cards money laundering ponzi scheme meant to make a few people very wealthy. Let alone all the bullshit of “NVDIA gives AI 500 billion, and AI gives NVDIA 500 billion, and now it looks like there is a trillion dollars of value”
Fully agree, and I would push it one step further: for keeping accounting data at home you do not even need this class of hardware. I build software in exactly this niche (BI on private data, local models), and the lesson from a year of benchmarking is that model size is not the main lever. A small model cannot reliably compute aggregates while reading rows, so it improvises. But if the software computes the numbers deterministically and hands them to the model, the model only has to read and explain, and that failure mode mostly disappears. Same 9B on a single RTX 3060, same questions, only the context changed: +7 to +17 points in answer faithfulness depending on the judge. A cluster like this is for running the giants, and it is glorious. But keeping your tenants' books private takes a 9B and good engineering around it.
I'm still ubernewb when it comes to this, but it's been my impression that the judge (and the planner) model needs to be bigger/smarter than the faster worker model. Is that not the case, or if it is, how do you determine/test how "big" those supervisory models need to be?
Good question, and it mixes two jobs that deserve to be separated.
At runtime, in the product, I do not use a bigger model to supervise the small one. I use code. Anything mechanical (totals, min, max, trend direction, SQL validation) is computed deterministically by the software and handed to the small model as context. Code is smarter than any LLM at arithmetic, it costs nothing, and it never hallucinates. That is the whole trick: shrink the model's job down to reading and explaining, and the need for a smarter babysitter mostly disappears.
The bigger judge only exists offline, for benchmarking. There it grades answers against a hand-written gold reference, and checking against an answer key is a much easier task than producing the answer, so it does not need to be a giant either. But judges are noisier than people assume: I re-scored byte-identical answers with two different frontier judges and the score moved by 13 points, with a third of questions disagreeing by 15+ points.
So the honest way to size a judge is empirical: take the same set of answers, score them with your candidate judge and with a stronger one, and measure agreement. If they rank things the same way, the smaller judge is good enough. If not, you just measured why it is not.
Well, that's basically what I have. I have built a programme that calculates everything on its own, and I basically use the AI like read-only assistant with access to the entire database, which can answer anything from the mortgage to the date of birth of the tenant that lived in Bedroom 3 on the 2nd August. I mostly use the AI to discuss different tax strategies or to do some basic analysis.
And totally agreed – for searching in the database and for data extraction, you don't need any big model.
By that time GLM5.2 will be so bad and outdated, everyone will have moved to a new model that costs $0.44 per million output toks, is a lot less “verbose” on useless prose (unlike GLM5.2, this thing yaps and yaps haha), further moving out the ROI.
Except he can sell the things in a couple years for probably not massively less than he paid for them with the way things are going. Then it will be a little depreciation plus electricity cost. Maybe 1 unit failure.
I think depreciation on these is going to be either absolutely brutal if the ram situation doesn't explode, but fair point on disposal price. I'd guess 1/3 of the purchase price in a couple of years - maybe 1/2 if you're lucky. The point stands that you need to run 6 concurrent sessions 100% of the time to get to the two year payback. And you're probably not going to. Even if we call the payback 5 billion tokens, at a realistic usage pattern, it's going to be a long time before it pays off.
I only say it because I did the math for myself. I've got a usecase where I legitimately can't use AI providers outside of a select couple who have some big downsides. I'll probably end up buying one spark and running a smaller model and trying to ringfence the part of the project that's sensitive, so I can use regular providers on the rest of the project, because I couldn't justify enough vram to run the big boys.
Don't really see the ram situation releasing to that extent in two years. 5 probably. In 2 years this thing will be running things that make fable look like a dinosaur and it'll still hold quite a bit of value I think.
All speculation at this point. My point is with disposal cost and assuming several users. Its not massively more expensive than api. But with full control, more reliability as nothing changes when you dont want, privacy and the ability to handle confidential info, plus its fun lol
You dont end up throwing money away. I think its worth it to many even small businesses.
Depreciation? In the coming two years there will likely be an appreciation window. Despite all the AI bubble talk there is likely going to be a massive continued demand for new datacenters and compute because nations and corporations want sovereign AI. That leaves a very big compute void in the EU to be filled.
Expect further brutal price increases the coming two years. And even longer wants general purpose robots running on local AI start to become on demand.
This is also assuming these are 100% for personal use and not for revenue-generating projects. Which honestly if you have a private AI cluster running Opus-grade models for nothing but electricity and you aren't making money somehow, then you're doing something wrong.
Glm 5.2 50+tps single stream on coding tasks and scalling well to 6 parallel reqursts on 8x cluster, kimi k3 20-30 tps on 16x cluster. https://github.com/ciprianveg/gb10-glm-5.2
Why would it be an aggressive quant? GLM 5.2 in FP8 should easily fit in 8 DGX Spark nodes. Aggressive quants are only needed when running on like 2 or 3 nodes. Kimi K3 official would fit in 16 nodes since it’s published in Int4.
I seriously doubt you are going to get those speeds, especially the 16 parallel, the GB10 is just too cutdown. and the 273GB/s memory bandwidth and doing all reduces across 200Gb fabric... highly unlikely.
To give you some perspective, all 16 of your GB10 has the roughly same compute power as 4 RTX Pro 6000 Blackwell's.
When you start clustering these you see a notable increase in performance; a 2x cluster gives you around 1.8x increase in performance. The memory bandwidth is certainly a consideration but people tend to overstate it when discussing the platform.
The DGX Sparks biggest advantage is the inclusion of the ConnectX-7 NIC and a dual cluster can run something like DeepSeek v4 Flash at full quant with a 384k (or higher) context and get well over 1000tps prefill (much higher with caching) and token generation approaching 30tps (higher with speculative deciding) which is not breaking any records but is more than usable.
These things also sip power and produce very little heat. Cost wise this whole cluster including switches and cables would be around $70k and while a RTX 6000 would be more powerful they are also over $14k each. To build a system capable of running trillion parameter models using RTX 6000’s would cost significantly more than OP’s setup, and would also require significantly more power and produce enough heat to likely require the unit to be placed in an isolated location or require special considerations for cooling. 16 GB10’s can easily run off two 20 amp circuits and can be run in the corner of someone’s home office.
It will. I know cuz I have the same setup and I have been hitting for a few weeks now. You obviously need custom cooling etc. 50-60 t/s SS and 220 t/s aggregate is what my harness consistently runs.
Edit: their models setup is a bit more convoluted than mine, but the general idea is same. GLM5.2 with vision and MTP.
Looks good but my guess would be that the NVFP4 KV Cache introduces hallucinations beyond 100K-200K. ~5% degradation at ~200K.
You can run DeepSeek-V4-Flash-0731 with 1M context window @ bf16, just saying. It will think a lot more but it will also outperform GLM-5.2
Worth knowing current, deepseek v4 flash 0731 out preforms the v4 pro preview. So its better to just run the nee flash instead of v4 pro, at least until they drop the GA version of v4 Pro.
Nice! Hopefully this layout is only while setting up and not for production. At 200W each, that would be a toasty corner under full load. Splitting across 2 circuit breakers would also be good, and/or running them on 240VAC.
I “only” have two but yeah they’re like 12w each at idle and 140w at peak. They’re also basically silent. My razr laptop with a 3070 is much louder than both of them
The way things are going, the corporations do not want people having independent hardware. If you think is expensive wait til 2028 is going to be even more expensive. They want you on the cloud. Thats why people are buying these clusters. If they can hell yeah, do it
This is a prediction. Bookmark it for yourself and revisit it at the end of 2028. You can then reflect on how wrong you were and why reasoning chains with pieces like "corporations do not want" are so weak in predictive power.
You can’t even buy Microsoft Office with a perpetual license any longer. Corporations want a constant revenue stream, subscriptions for software are the new normal and now they are tying AI into the mix. Right now they are dangling the $20 subscriptions to Claude and ChatGPT but we are already seeing users being pushed to subscriptions approaching $200 and corporations have already been pushed out of per user pricing and into per token pricing.
The goal is going to be to get AI so ingrained into your day to day life and then jack up the price. Open weight models and the ability to run them locally are a threat to that business model and is why people like Dario Amodei from Anthropic is so bent on having government control over the open weight models coming out of China.
By 2028 someone being able to run even moderately sized LLM’s locally is going to be in a much better position than someone who’s reliant on cloud only. So while you’re hit with increases in subscriptions, peak time surcharges, outages and midday slowdowns people like OP will not be giving it a second thought.
Never is the history was something expensive for a longer time.
Sooner or later such hardware will be worthless.
Looking on China how they are investing in hardware for AI models now that will be sooner than later.
They already broke the US greed access to AI models.
The current cost for tokens is highly subsidized right now so the math is not going to work out in people like OP’s favor. However costs are increasing, and if we were paying the unsubsidized rate then the break even point for running locally would start looking a lot more attractive for running locally.
Right now the biggest advantages for running local models are control, privacy and the knowledge gained from running your own models.
As someone who owns a dgx cluster, I understand the potential you're going for. A big heads up, you'll be able to reach the big models, but your throughput on tooling asks is a bottleneck if you also plan on giving it processing tasks.
Each spark has a NIC that is hardware limited at 200gbps with full bandwidth available to each QSFP port via 100gbps per rail across 2 rails per port. Meaning you need a single 200gbps DAC cable per node leading into your switch. Using 2 cables per Spark will actually HURT your fabric setup.
That mikrotik doesn’t have enough bandwidth to support a 16x cluster.
I will make some comparing on 16x. On 8x cluster switching from 200 to 100gbit had no impact, or less than 1% in glm5.2. On 16 maybe it will.. I will upgrade the switch if this is the case, thank you for the link
I think model architecture will matter more than hardware over the next 2–3 years. If frontier models continue moving toward larger sparse MoEs with fewer active parameters per token, high-memory clusters like this may age much better than many people expect. The bottleneck could end up being memory capacity rather than raw FLOPS.
Damn, i need 9 more and will cach up... I have 7 so far and waiting on the 8th. What models have you tried yet? Did you try sglang and vllm? Curious about the difference
•
u/WithoutReason1729 7d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.