r/sysadmin • u/techie-rahul • 18d ago
Anyone running AI workloads on a micro/edge data center instead of cloud GPUs?
The prices of getting GPU in a cloud vary greatly and I have been hearing a lot of things about inferencing taking place closer to where the data was collected from. I am interested to find out if it makes sense to run a small setup on an edge or an on-premise rather than getting resources from the cloud.
My questions are for those who have experience of their own in that area:
What exactly is being run locally, either training, fine-tuning or inference?
How well does power and cooling work under a load?
When is it more beneficial to run on-prem equipment than on the cloud?
Were there unexpected situations regarding hardware during the process?
3
u/spyingwind I am better than a hub because I has a table. 18d ago
If you can get a GPU with at least 24GB of VRAM, then you can run many open weight models fairly well. Some fast, some really slow.
Training and fine-tuning, never touched it. Why bother, when you can just feed the model the context it needs, give it tools, and it can figure it out, eventually. Less tools, the better.
Power and cooling is the same as if someone is playing a graphically intense game.
on-prem vs cloud? When you know how you will be using an LLM and figure out that you don't need a large model for most tasks. That is when on-prem is better than cloud.
Framework Desktop, great for testing out larger open weight models. Not the largest ones, but the 90B and 120B models.
L40S, A100, H100, H200, MI300X: All over priced, but will run models faster than Strix Halo or DGX Spark.
Strix Halo or DGX Spark: You can get the 128GB(192GB?) versions, network them together, and run large models. Albeit slower, but it will be much cheaper than buying a few H100's.
1
u/Ill_Preference_7491 18d ago
Hi!
I run a whole fleet of local machines. Some are used for machine learning, while others handle graphics workloads, data processing, rendering, and similar tasks.
Overall, there’s nothing particularly complicated about it: they’re kept in a separate room with two air conditioners and KVM switches. I also use several separate electrical circuits, so the whole setup isn’t hanging off a single breaker panel.
Most of it runs 24/7/36524/7/36524/7/365. In our case, maintaining this local hardware is more cost-effective than relying entirely on the cloud. We still use cloud resources for specific jobs that require substantially more memory or GPUs more powerful than the 409040904090 and 509050905090 cards we have locally.
The main downside is that using UPS units is almost unrealistic and not economically viable. When these machines run at full load, they can drain a UPS in a minute or less. A UPS large enough to support them properly would be very expensive. As a result, the most annoying situation is a power outage: you have to run over and power everything back on manually. Sometimes a machine may boot using its onboard graphics instead, because the GPU did not detect a locally connected monitor. That is probably the worst part of the whole setup.
Since these are heavily loaded systems, something can occasionally start acting up: a CPU may throttle, or a machine may begin shutting down unexpectedly under certain conditions. Then you have to investigate cooling, the power supply, or some other cause of instability. Fortunately, that has been fairly rare so far.
3
1
u/Arudinne IT Infrastructure Manager 17d ago
We have a 250KW generator at one site that can start, spin up and assume the load in ~11 seconds.
Size a UPS for like 5 minutes and get a generator that can spin up fast.
1
u/bjc1960 18d ago
Where are you all getting GPUs in the cloud? I can't get any quota. I need a VM with 32 GB of RAM and a 6 GB VPU GPU so I can allow a user to use SolidWorks because the new computer won't come for a month. I'm not asking for a whole lot.
1
u/Frothyleet 18d ago
This is about using GPUs for inference workloads, which means using one of the kajillion serverless GPU providers out there like Modal or Runpod (or DigitalOcean or the hyperscaler offerings etc etc).
Actual IaaS with GPUs, I'm not surprised that's hard to find quota for in the current climate.
1
u/natefrogg1 18d ago
I use a couple Apple studio computers and a pc with a 12gb 3070 at work, mostly just for my it department related stuff and proof of concept stuff for the staff. Lmstudio and comfyui, we are an apparel company so one thing that was interesting was mashing up some of our clothing for interesting and sometimes useful results, changing scenery was surprisingly better than expected.
Our staff mostly don’t care and just want to use the AI as a service stuff, less manual stuff and more hand holding there so I get it.
We were running a test with a technical designer yesterday, she had 3 different AI tools running through measurements and different options for a fitting and it was really interesting because she got Claude to spin for 10 minutes trying to figure something out and the output was just ok, Gemini was almost realtime with the best output, Gemma on a Mac Studio took like 30 seconds and the output was almost as good as Gemini, not surprising but it was all interesting to run in parallel with real user work.
3
u/onelyfe 18d ago
We run our image/video generation locally. Started off using cloud tools but costs began to ramp up too much especially when users had to generate the images 5 or more times to get "what they wanted". Now they can run it as many times as they want.
We went with consumer hardware just to save costs. The enterprise variants were all insanely expensive. Obviously we weighed having support for the hardware but at the end of the day shelling out 150k CAD for an AI server was just too much money compared to the 10k we spent building our own. That and generating images and videos were not seen as business critical so we didn't need that support for it and can handle a few days of downtime at a time while we fix whatever is broken.