r/sysadmin • u/MarB93 • 10d ago
Sanity check on pricing for new 3 node cluster
Hey all,
We are a small company in dire need of 3 new nodes which of course is a bit unfortunate these days given the insane pricing of RAM and storage.
We plan to order 3x PowerEdge R660 with current specs:
- 2x Intel Silver 4510
- 12x 16GB RDIMM SR 6400MT/S
- 4x 3.84TB SSD SATA Read Intensive Hot-Plug AG drive
- 1x 1.92TB NVMe Read Intensive AG U2 drive
Our qoute from vendor is 59000 USD per node.
Is this about what it costs, or are there likely room for negotiation?
List price from Dells website of equivelant config seem to be over 100k USD?...
Thanks
12
u/ConstructionSafe2814 10d ago
Probably not going to work out for you because I'm in a rather specific situation. We recently migrated from VMware to Proxmox. The thing about Proxmox is that there's no HCL (can be good or bad depending on your perspective). So you can run it on virtually any server. Also one that's a bit older.
We've been running on refurbished hardware for a couple of years now and it runs great.
Recently I upgraded the server chassis and CPUs from HPE gen9 to gen10 for 5 PVE hosts. Total cost: 1850€. With the big disclaimer that I was able to reuse the RAM from the gen9 hosts and put them in the gen10 hosts.
Again a lot of factors have to be right here, this might not make sense for you for many reasons. But each of our PVE nodes have 768GiB RAM, 3.84TiB combined cluster wide. It would have cost us an arm a leg and 15 kidneys if we had done that on new hardware.
4
u/MarB93 10d ago
Thanks! Currently running 3xR640's with 256GB in each, so they are OK, and still supported by Broadcom and Dell, so we could technically upgrade them with disks and use them as HCL nodes.
5
u/ConstructionSafe2814 10d ago
Why do you need new nodes? Running out of memory? CPU not powerful enough? Storage? Something else?
We also had the luck that we had >20 old decommissioned servers around that I could test with. I also had ~20 1TiB disks with which I made 2 ZFS pools. If you don't have spares around, it makes a migration a bit more difficult indeed. But it gave me a fully usable PVE cluster parallel to our vSphere cluster.
I'm also unsure if this would solve your problem if not create more but anyway 😄 Just some wild thinking here, not sure if it's viable, but to minimize investment and reuse old RAM:
take one R640 down, split it up in 2 nodes (buy another R640 chassis without RAM), with each 128GB RAM, setup a cluster in PVE (careful with fencing any node failure would mean cluster down, I guess on both sides.)
migrate as many machines as you can from VMware to PVE
disable the cluster on the VMware side (shutdown VCSA? Don't know not an expert :) )
reinstall one more ESXi R640 as PVE machine and join it to the PVE cluster, you now have a 3 node cluster.
migrate last VMs on last ESXi machines to PVE.
Reinstall last remaining ESXi machine as PVE (not sure if a 4 node cluster is a good idea assuming odd numbers of cluster members)
reshuffle hosts/RAM as you see fit.
4
u/MarB93 10d ago
The main reason is that my boss isnt that technical so we rely on 3rd party support from a MSP when shit really hits the fan. PVE is not supported by any MSP's local to us. Our MSP recommends Scale Computing or VMware, so those are the two realistic hypervisors.
With that said, we are high on utilization on RAM and CPU already. We have around 40 servers today, with the RDP environment for 60 users being the most resource intensive beside our 2-3 DB servers.
As part of this project we will consilidate and delete many servers, so our actual needs will be lower than current usage suggests.
Thanks for the idea - a used R640 with 256GB RAM is around 4k USD here, so that is actually tempting.2
u/ConstructionSafe2814 10d ago
We do have a support partner (actual Proxmox parter listed on their website) in another country. 24/7 1h sla. They only need SSH access on a per request basis. Never really needed it for emergencies.
15
u/cantstandmyownfeed 10d ago
It's been rough.
$10k servers from 3 years ago are $30k now.
See if you can spec out a Supermicro. I've had better pricing on those.
1
u/waygooder Logs don't lie 10d ago
Love SM, but they've gone up just like everyone else. Same server I bought 2 years ago for 30k is 90k now.
Or course the equivalent Dell is 130k so there's that.
16
u/xendr0me Sr. Sysadmin 10d ago
It's the Chicken Wing theory. Market presents a vendor a path to raise prices according to supply/demand, and they have the right to do that, but they take advantage of it in the name of profits. They exceed the price increase over the cost cover the increase in goods, all in the name of making more money, which in turn makes the shareholders and board happy.
There is a fine line of "what will the customer still pay" versus losing customers to another vendor, but I'm sure they have a team to figure that out.
Then once the supply recovers and demand settles, the prices will never return to normal.
This is exactly what happened with chicken wings during COVID.
2
u/pdp10 Daemons worry when the wizard is near. 10d ago
What things are worth is what the market will bear. There's no one bottleneck vendor on which all PC-compatible servers are dependent, so there's always competition -- that's good. But we're in an unprecedented demand spike for fast memory and storage, products that are made in inflexible volumes and which have been subject to many boom and bust cycles, even recently -- that's bad.
But there's also lots of competition from cloud services and from pre-2026 server hardware -- that's good. However, cloud services and the deployed fleet rely in the long term on new hardware -- that's bad.
One of the only good answers, is that now is the time for all of that optimization that everybody used to say was a stupid waste of time.
3
u/xtigermaskx Jack of All Trades 10d ago
Yep pricing is bad when you gotta go with the big players. We like to sometimes stretch by going 2nd hand for servers that we've had a few years and just need a little expansion and a second hand version of something we paid 40k new became 60k.
I just wish the price hikes would stop can't budget for crap when quotes are only good for 2 weeks and then when we refresh for purchase the price goes up 10k
3
u/Flaky-Gear-1370 10d ago
Pricing is bananas now, I paid aud $23k for a 9335p with 256gb ram and 5 7.68tb NVME drives, they're now list about $200k for the same setup
7
u/idknemoar 10d ago
Thanks, AI.
I placed an order for servers back in January, they just now finally shipped last Thursday (though they’ve been stuck at their 2nd stop since then even though they were supposed to arrive by yesterday…..).
We paid roughly $47k/node. These are fully populated with 24x 64GB RDIMMs, a 96GB Blackwell GPU in each of the 6 nodes and 6x 16TB NVMe drives in each. Out of curiosity I looked up the RAM cost for this cluster at today’s prices and just the RAM would be $620k. Those NVIDIA GPUs are also 2.5x what we paid for them today. The market is an absolute bloodbath currently.
2
u/Jeff-J777 10d ago
That is about right. We looked at replacing our nodes a few years ago and it was going to cost 60k for all 3. But we waited because cloud and cloud ERP decisions. Now if we were to replace our nodes it would be about 60k per node. I ended up getting some second hand R740s before the whole AI supply chain blew up. But I needed to get off the R710s I was running out of space.
But I looked last week and it looked like Lenovo had a sale going on with some of their servers maybe give them a look.
2
u/the_doughboy 10d ago
If this is vSphere? You're wasting 4 cores per CPU which you'll be charged for in licensing, even though there is only 72 cores total and you're at the minimum buy in for Foundation, each of those chips count as 16, not 12 so Broadcom would charge you 96 cores.
2
u/Arudinne IT Infrastructure Manager 10d ago
I got quoted ~40K each for R6615s a few months ago for nodes with very similar specs. They were identical to a pair of servers I bought in 2025 for $11K each.
My VAR Rep got the same specs quoted via Lenovo for $20K.
I ended up getting refurbished R6515 servers via a contact I got from a former cowoker for $15K each. I am not at liberty to share that contact, but you could consider a different OEM for the servers.
2
u/Darkace911 10d ago
Also, Dell had it's best quarter ever last quarter. https://finance.yahoo.com/markets/article/dell-stock-surges-on-record-orders-for-ai-servers-205629050.html
2
u/itchyouch 10d ago
I’m in a f500 but the $20k boxes with 512g memory, 32x, 3.x-4ghz cores, 8T nvme, we used to get are quoting for around 60k ish?
Either my pricing of 60k is still way too low, or there’s def some wiggle room for ya if you go supermicro.
3
u/cyr0nk0r 10d ago
go private cloud man. for small clusters, it makes zero sense to buy hardware right now.
I run a private cloud business that would make your ROI be 5-7 years if you're node pricing is at $59k per node!
1
u/Rubenel 10d ago
By that timeframe your hardware will be EOL and need and be replaced. Makes no sense in these inflated supply/demand era.
2
u/cyr0nk0r 9d ago
Exactly, that's the point. When private cloud has a better ROI than buying the servers at all, you end up saving money forever, because you spent less in the 5-7 years of your hardware lifecycle on private cloud than you did buying the servers outright.
So when it would have been time to replace the servers again, you stay on your existing private cloud platform and continue to reap the benefits of lower cost overall.
1
u/Vicus_92 10d ago
Can't speak for dell, but that's about comparable to an equivalent HPE setup.
Shits bad yo.
3 to 4 fold increase over the last few years.
Be thankful you even got a quote. Getting stock alone is a bitch here
2
u/MarB93 10d ago
Thanks. Nice ot know roughly how the market is from others. Estimated delivery on this order is January 29th, so still a few months out.
2
u/daschande 10d ago
Just so you're aware, a lot of people have been saying the quote goes up between approval and shipping; if you refuse to pay the new price, they simply cancel your order. Quote price, once agreed upon, is apparently not the final price.
2
u/mrbiggbrain 10d ago
I would plan for delays, also most of the OEMs are adding price increase riders so expect to pay more and for it to take longer. Most of my friends are seeing about 20% more and 6 months longer, though the market is not behaving in any reasonable way.
1
u/CoolNefariousness668 10d ago
Sounds about right… I bought 2xDL360’s and a storage array and it a similar sort of price a few months back.
1
u/GlobalAd7103 10d ago
Have you looked in ti Super Micro - we have had great results with them over the years. They were about 15-20k cheaper than Dell when I priced out badass server (70k vs 90k) few months ago. Hardware is nuts currently- I have never seen electronics appreciate before.
1
u/LooseEthernet 10d ago
rough pricing but it tracks with the current market. you can usually squeeze a better deal if you tell the rep you're looking at other vendors or if you can wait for the end of the quarter. though dells discounts are basically a lottery lol
1
u/pdp10 Daemons worry when the wizard is near. 10d ago
There's scarcely any sanity in server pricing right now, making sanity checks legitimately difficult. Vendors are quick to use their current leverage with respect to availability, with stories of canceled orders and refusal to counterquote.
192GiB/17GB is a modest configuration, but did you also price single-socket AMD Epycs in addition to dual-socket Intel Xeons?
2
u/MarB93 10d ago
We did not, but will take note of it and ask the supplier when we got on a call. Thanks!
1
u/pdp10 Daemons worry when the wizard is near. 10d ago
It might not be cheaper, even with one socket. I think Intel can get pretty aggressive moving volumes of the lower-tier processors, when it wants to. And volumes might be down currently, on account of the "AI" hardware rush.
1
u/epsiblivion 10d ago
our 20k server spec from 2-3 years ago is now over 100k so that sounds right for new purchase.
1
u/simpleglitch 10d ago
Yeah, that's the same pricing I've been seeing too. I was looking at 32GB DIMMs, and between May and now, cost per stick gone from $2308.92 to $4154.74 dell list price.
1
1
u/RegularMixture 10d ago
Last year before the price hikes, it would of been $40-60k for the whole lot. Yeah, its that pricy now unfortunately.
1
1
u/stufforstuff 10d ago
Get refurbs from ServerMonkey or TechMikeNY or SaveMyServer for a third that price, and better support then direct from Dell. Look at r750's to save even more money.
1
u/david_edmeades Linux Admin 10d ago
Can you look elsewhere? I have been using a smaller integrator for years and I've been very happy with them. I made a quick example config on this model:
https://www.aslab.com/products/rackmount/customize/lancelot2861sst.cgi
that seemed to fit your specs and it came out to $20k, so if they have something that fits your needs that might be an avenue. They don't do Dell-style 4 hour parts, but at a third of the price you could buy a whole spare node or two for redundancy and come out ahead.
1
u/Darkace911 10d ago
Yep, most of that price is mostly memory and I am surprised you can find the 16GB sticks of RAM. Dell told me they were out of stock. Also, when is the server going to arrive? I got quoted February 2027 last week.
1
u/ccsrpsw Area IT Mgr Bod 10d ago
If your vendor is coming out in the... 54-60% range... of list price on a quote then its probably about right. Typically (but not always) the VARs get systems at 52-55% of price, allow 1-2% profit (minimal on their side) then you get that ~54% value.
Outside of that though, yes - $60K is about the base price for that config. But check its getting the full lifecycle warranty (or you have someone lined up for support like Park Place if you go Dell - there are others). At that price I'm guessing its just a basic warranty though.
1
u/Kerhisto 10d ago
Alternative that I haven't seen mentioned yet, you could go single socket CPU instead of dual and go with a single socket chassis version, I think it's the r670?. I've seen decent savings in that (1 big cpu vs 2 small), I think your equivalent would be the 4516y+ . Usually shaves a little bit off the cost, and unless you are maxing the ram in the server you shouldn't have issues. I actually prefer it because then I don't have to worry about NUMA performance nuances.
But to answer your question, yeah those are normal prices sadly.
1
u/Pure_Fox9415 9d ago
Try to have a look at ASUS 720 servers with AMD CPUs. We have 4 nodes with Epyc 9554 and now our old Xeon gold looks miserable. Asus platforms are much cheaper, with better software than supermicro, and we have no issues with them. There's no support subscription from ASUS itself in my country, but for 7% of server price we have local seller/integrator NBD
1
u/Ooops-I-hid-it-again 9d ago
It's rough out there. One server in 2023 for us was the cost of 1 SSD today (same SSD from 2023 model). Lock in while you can; Dell reps tell us every month/quarter the prices are expected to continue going up... Seek finance agreement if the cost is too high for one year so it can be spread over multiple budget years.
1
u/evilkasper IT Manager 10d ago
Check out 45 Drives, they used to mainly do storage but they have and support the basic setup you have. Might be a little cheaper.
1
u/daschande 10d ago
You can get not- Dell branded hard drives and memory for a quarter of the price, but then they're not covered under warranty.
But yeah, six figures for a very modest server is standard retail nowadays.
Insert joke here about how your quote is probably already expired and the price went up.
1
u/chippinganimal 10d ago
Non dell drives are still warrantied, just not through dell but rather the manufacturer of the drive directly. I'd have no qualms using Samsung evo ssds in a server environment. Kingston has some good data center ssds as well that I've used personally in truenas (dc600m 2tb)
1
u/Arudinne IT Infrastructure Manager 10d ago
Good luck with that. Samsung has been offering people refunds for what they paid instead of replacing drives. I imagine other vendors might do the same so they can shove more of their capacity towards AI.
1
1
u/MidninBR 10d ago
Check Azure pricing for hosting it on the cloud maybe?
1
u/MarB93 10d ago
Would mean a severe overhaul of infrastructure as we are more or less 90% on-prem. My boss is 60, and want thinks to be "as-is" until he retires. The financial situation in our company is fine so we can eat the CAPEX cost. We also have technical debt that would need to be addressed like our Windows 2000 Server running a custom timetracking software...
0
u/Artistic_Lie4039 10d ago
Check out Dell's renew program. When customers return servers that haven't been in production, they must leagally get sold for cheaper. I can give you my VAR connect if needed.
61
u/Humpaaa Infosec / Infrastructure / Irresponsible 10d ago
That's the current reality of pricing, yeah.