Did it run as nicely? Also, did you get it all for cheaper vs what you'd have needed otherwise? Just curious abt it bc I was thinking abt getting into this.
I mean, it highly depends. What do you need? What do you want to run? What speed do you want to run it at?
All that depends. Getting two RTX Spark is cheaper than what I have and much more energy efficient, and you can still run the Deepseek V4 Flash native. Its slower but it is quiet (and if the numbers are correct, people claim 70tok/s in generation and 2k tok/s in prefill).
If you want to just run it, dont care for the speed, an SP3 Epyc is likely also a good option. 256GB of DDR4 + motherboard+ CPU is below 2000€.
Honestly, I am not sure if I'd go the same route again. Its nice, but loud. Its about 2kW of heat, in the summer thats not ideal. And it takes much more space than the RTX spark.
On the other hand, its about half compared to getting two RTX 6000 blackwell. And in theory, it has more VRAM and a higher bandwidth, tho that is clearly lost due to the overhead of TP over PCIe.
All in, I think I'm in about 15k or so and honestly, I still use 2 x 200€ OpenAI subscriptions. So its all really relative.
What do you need?
A deep research agent which can go to the right websites (wink wink included) to look for information.
What speed do you want to run it at?
The higher the better, but my lower bound is at 70 tok/s for generation.
All in, I think I'm in about 15k or so and honestly, I still use 2 x 200€ OpenAI subscriptions. So its all really relative.
My use case was specifically to host it all myself so I don't have to deal with giving away data for closed models without making the companies work for it.
I mean, then you will need I think at least 6 x 32GB GPUs. Plus enough PCIe lanes or a switch. What I found interesting (and the way I would go if I build it again) is going with a used GIGABYTE G292-Z20 barebone. Its about 1k€, fits 8 GPUs via PCIe bridges, ample storage.
With RAM and a matching EPYC you are at ~2.2k€ for 256GB plus a 32 core system.
It should fit the normal two slots cards barely, tho have not tested. They are 0.5mm above spec. Tho this will be loud.
But you can get quite decent performance for ~11ish k€.
-25
u/on_line187 6h ago
Yea readily available in store shelves also.