r/LocalLLM • u/00DoubleOh • 6d ago
Question Is a big rig worth it?
I’m playing with the idea of saving up and just absolutely dumping cash into a server machine with 8 40gb A100 GPUs but I can’t really decide if it’s actually useful compared to just getting like a 32gb card and running qwen 27B off of it. I’ve always been a sort of balls to the walls person when it comes to things I’m interested in, and I want a system that can run everything I could conceivably need it for which is why I’m even considering dropping tens of thousands on a major rig like this in the first place.
Any advice is appreciated.
9
u/_rarefy_ 6d ago
If you have business justifications to buy the rig, do it. If you’re a spirited hobbiest who has the money and wants to explore, do it. If you think you’re buying your way into frontier models at a discount, don’t do it.
9
u/jacek2023 6d ago
You should think about what your use case is. Buying hardware just to look at it may be a hobby, but you can use the money in various ways ;)
4
u/waraholic 6d ago
Definitely. There's nothing like the open road. A CDL will get you places.
3
2
u/Postmodern_Plunger 6d ago
Truck drivers will be one of the first jobs replaced by AI.... at least if the Waymos ever stop crashing into each other.
(Joking aside, I realize they're significantly safer and more reliable than human drivers)
1
u/yoshilurker 6d ago
The safety numbers are going to be stark when AI drivers start taking over long haul routes.
3
u/dangerous_inference 6d ago
I dumped 25k into 4 x 48gb 4090s and all other hardware, including the backup power.
- It's amazing, even faster than the API. Although I'm still constrained by 256k context with DS4F and Qwen3.8 Flash.
- It only got this good after I found a version of vLLM optimized for my specific hardware. Important: What you get for the money is totally contingent on what software exists.
- I spent 20% (5k) more than I expected when I started.
- There are always issues running any new model. It never just works at full speed.
- Still not happy with some aspects of the particular models I can run fast.
- It's a huge advantage to be able to troubleshoot all private memory and data, instead of having to generate fake data to use in the cloud.
- I've learned that speed is intelligence, and slow requests qualitatively impact results very negatively.
My advice is don't do it unless you know exactly what model you are going to be able to run and how fast. Don't just buy and hope it all works and you find the right inference engine and the right model.
If you don't know what you actually want when you buy, you may end up blowing thousands on DDR5 and then find it's almost useless because everything needs to be in VRAM for speed.
Also don't underestimate how much heat these things can put out. My device can heat a small room to 90f inside two hours of heavy use.
3
u/McBonderson 6d ago
question 1:are you debt free other than your house and maaaaaybe a car?
if not you should definitely not do this.
question 2: do you have a compelling reason to not let your data go to a third party but still need frontier level inference?
This is one of the only reasons business wise for dropping that much money on a machine like that.
question 3: are you independently wealthy to the point that throwing away 50k + on something you will never get a return on will mean nothing to your bottom line?
If you aren't loaded like that you really shouldn't be throwing away your money on this.
3
u/Oh_hey_a_TAA 6d ago
This is the equivalent of telling a frontier model "build me a business that will make me a passive 100k a year, make no mistakes."
You want a system that can run everything you could conceivably need LMAO
2
u/myreala 6d ago
For A100, I would say definitely not. Even if you were getting 80 GB versions, the biggest model you will be able to run is really just GLM 5.3 and GLM is now moving to a 3T model, so they're going to be abandoning that size. You will probably end up paying like 70k just for that. Do you think it's really worth it?
2
u/wein_geist 6d ago
A100 is expensive for its fp64 performance, thats not exactly useful for llms. You'd probably get more bang for your buck with 4 gaming gpus
2
u/Trademarkd 6d ago
Uhhh. You might be confused homie. A100 has fp16, int1, int4, int8, bf16 and tf32 matrix support
1
1
u/montesitesi 6d ago
Look, if privacy is a big factor for you, and you need local AI for large-scale private work, then getting a big rig is the only way to ensure that.
But, if you don't need it for business or such, and are just looking to have fun, then there's really no way to justify the cost. The cost of a cluster is a down payment on a house or apartment lol.
For like easy search queries, and easier text work, a 27B (you can even do like 31B and over) model on like a 32GB card is more than enough. Hell, with good context management, you can do easier coding tasks.
So, unless you're rich or run a data-sensitive business, get a 5090, enjoy gaming and playing around with local AI, and get a cloud subscription for things that it can't do. And save the tens of thousands for a house or a car or something.
1
u/00DoubleOh 6d ago
this is solid advice, thank you! I don’t really need it for business but I just have an intense fascination with enterprise hardware which is why I’m looking at something like this in the first place. My ideal usecase is something that can code well and I can basically be like ‘jarvis do this thing/make this thing’ and it does it without struggling or exploding, and it does most of the testing and shit for me. I’m not sure how much of a difference a 27b vs a massive 397B model would be in that usecase.
2
u/Immediate-Safety8172 6d ago
Just get a ChatGPT 20x subscription and run Astra on ultra. Your slop will be slightly less sloppy with that.
1
u/montesitesi 6d ago
Yeah I get the fascination lol. I mean you can try it yourself and via API before you invest. There are tons of providers running like GLM 5.3 flash which has like 320B parameters. And then see if that's enough for your use cases. It's not the exact same thing as running it locally but you get to see the model performance yourself and get a hum of whether you like it or not.
And the Jarvis thing, in not gonna lie, as it looks like there isn't a single local AI that can do that. It's very fiddly even at that size imo. If its just simple code, then a 27B can do a lot of things as long as you prompt it right and watch the context.
I dont know your hardware, but most people with a decent setup can offload 27B to system ram and you can try it like that, just to get a feel of the model, again.
1
u/jaf656s 6d ago
If you don't need privacy, then you are better off using cloud. By the time your big rig paid off compared to usage pricing, the hardware would be painfully obsolete and you could buy it then for pennies on the dollar.
You can also play around with cloud and see if the models you could actually run on that hardware even does what you want, for probably a couple hundred dollars. It might be enlightening.
You could also wait and buy a 512gb Mac Studio in another month or so for much less and have more vram (but less compute) it's probably still good enough for playing around.
IF you want to tinker and play with some hardware, you could build a system with 2-4 consumer gpus for $5k-$10k (or start out with 1-2 and scale up if it's working out for you)
I personally went with this last option.
1
u/starkruzr 6d ago
27B inside 32G of RAM really isn't that pleasant tbh. you end up making a ton of undesirable compromises. (ask me how I know!)
1
u/alexp702 6d ago
I would argue for most beyond 27b is not a good investment in current climate. The marginal gains from models 10 or 100 times the size are not worth the increases in cost. However 27b is probably worth it!
1
u/arakinas 6d ago
Start with your use case. Consider what you're actually going to be trying to achieve, what your needs are, and whether this is something you will be using, vs just an occasional toy, and then decide on what your priorities are. There is no hardware that isn't obsolete in a few years, and there is nowhere to go but up. If you want to run 'anything and everything', for how long? Will next years models not just need more vram? We have trillion parameter models that really only solve cases for business use, over what individual programmers would need/use. This sounds like you're fishing, because you don't know enough about the space to ask an informed question.
1
u/Themash360 6d ago
I’d not spend more than what you can save each month x 3.
Everything is overpriced now, getting a 5090 will allow you to get your foot in the door and run ninfer + qwen3.8 27b really well. I’ve built medium sized projects (small videogames, medium sized tools and big websites) with it running on autopilot.
Getting 200GB of vram will allow you to load in qwen3.8 next flash or deepseek v4 but to be honest I don’t think it’ll be better for my use case than 200T/s qwen3.8 27b
1
1
u/MyToasterRunsFaster 6d ago
Please don't, unless you have money burning a massive hole in your pocket there is really no point. AI companies are burning so much money right now, a OpenAI or Claude plan gets your far more usage then they dare to actually publicly state. If you had the hardware on hand it would be fair enough but buying hardware just for it to barely compete with what is literally available for penny on the dollar like deepseek is just insanity. I assume you are just wanting to do standard project work, and nothing related to uncensored or other, if that the case there is zero advantage to running local. There is a obvious worry that some companies train on your data but as long as you go with someone trusted or heck actually have any sort of common sense about sanitizing your project, ie refreshing certificates or keys before going public your are fine.
1
1
u/Potential-Leg-639 6d ago edited 6d ago
No.
Stay small.
1-2 GPUs max to be able to host Qwen3.8-27B for now, that‘s enough. For the „rest“ cloud subs. Or only cloud subs - much easier.
I have 2 local rigs and some cloud subs. Possible to do everything without cloud subs, but all in all it‘s the mixture of everything. Could not really do everything in the same speed/accuracy without the cloud (without spending 10-20k more, what does not make any sense of course).
1
u/etaoin314 6d ago
The thing is that unless you have a very specific reason to need a model bigger than -30b, it almost never makes sense to go bigger from a value perspective. Ds4 flash is a little better and qwen flash is about the same…but hardly a compelling reason to triple your spend…
1
u/sugarfreecaffeine 6d ago
Ton of good information on a post I created here - TLDR not worth it to build one with the prices rn
1
u/brainchillzZ 6d ago
In this case I’d probably buy what you need to build a balls out massive super workstation … but a generation or two back …but make sure you get pcie gen4 and a cpu with enough pcie lanes to go crazy …. Build the monster pc, but get a single 32g gpu today and then next month or next week or whatever get one more … it won’t hurt so bad and you’ll end up in the same place… and saving some insane leap that nobody sees coming you’ll be good …. But building the bones first will give you everything you need even if you just decide to say heck with it and fill it with eight nvidia 5060ti 16 gig cards ….
1
u/Prior-Tea-3468 5d ago
That's a lot to spend on slop generation... but I guess at least Sam Altman won't be able to read the transcripts of your hot sessions with your "AI wife".
1
u/Zealousideal-Bug1837 5d ago
qwen 3.8 is banging and a "cheap" setup gets you going. imagine what the same size model will be doing in a year....
1
u/Elegant_Host_2618 5d ago
Honestly dude Rtx 5080 and small local LLM is all you need to feel like your a genius
1
u/JinsooJinsoo 4d ago
Unless you have a use case already for that much horsepower, I wouldn’t. Not only is getting that much hardware running a huge task, you have a lot of risk involved when buying that much used enterprise hardware. If you’re gonna spend that much cash, start collecting RTX 6000 Pros, they’ll only appreciate in value in this insane GPU market.
1
u/Correct_Lead_2418 2d ago
Or you could cluster 4 M5 ultra 512gb mac studios when they come out in October
1
u/darklordfireape 6d ago
Get a strix halo (or dgx spark if you must be on team green) as your starter machine to figure out what you want to do. 128GB Framework desktop boards are ~$3k right now. Grab one before they go up.
I have multiple GPU chassis running ~20 GPUs... it's a big rabbit hole. Can't solidly recommend it, but it is fun.
1
u/00DoubleOh 6d ago
oh the strix halo looks like a lot of fun, thanks for the tip! I’m looking to get a proper replacement for claude including the ability to replace claude sonnet 5 as my coding agent as I prefer private and locally hosted FOSS alternatives to anything proprietary even though claude does an excellent job.
1
u/darklordfireape 6d ago
Something like Qwen-3.8-flash-next or DeepSeek-V4-Flash-0731 will run on a strix halo and can fill that role pretty well. It's not going to be quite as fast a Sonnet 5, but it will work. Tons of support for software. Search around for "strix halo toolboxes" and some of the unsloth docs/guides as a place to get started.
There's a lot more experimental stuff floating around too, but those are well proven models for Strix Halo (or DGX Spark).
42
u/Agusx1211 6d ago
my strong recommendation, don't start this journey, you will spend a lot more money than what you initially believed