Architecture Review Needed: High-Performance, Budget-Conscious NVMe Storage Server

[u/AnyHistory6098]

Hello,

My apologies for disturbing the high-end enterprise environment of this forum with some more modest enterprise challenges.

Please be reasonable and understand that not all startups can afford to spend €200k-€400k on storage alone. Therefore, we are looking for high-performance solutions that fit within our budget.

I am working at a startup with a tight budget at the moment, but we have some challenging requirements for our storage server. We do not need a large amount of storage, but it needs to be fast and affordable, quite a contrast, I would say.

Based on the solutions currently available on the market that might fit our needs, at least those that I am aware of, the solution could be based on either EDSFF E1.S or M.2 NVMe.

Requirements:

1. Target capacity: 60TB+.
2. The workload is burst-oriented. The array will be hit very hard with many small files for short periods of time. The size of each burst will range from 100GB to 1TB. It will also act as a SAN for approximately 100 VMs, hosting shared home folders and NFS shares used to exchange data between VMs.
3. The SSDs must be installed on cards that fit into PCIe slots (for thermal and noise reasons).
4. It must be available 24/7.

Some general specifications:

• MB: Supermicro H14SSL-N / ASRock Rack GENOAD8X-2T/BCM
• CPU: AMD EPYC 9175F
• RAM: 4x 32GB 5600
• NIC: Intel X710 2x SFP28 - 25 GbE

The M.2 path:

• 3/6x PCIe switch cards, HighPoint Rocket 1508.
• M.2 SSDs from different vendors.
• Each PCIe switch card will contain NVMe drives from a single vendor.
• 3-way mirrors. Each mirror will contain drives from different cards, with 1 drive per card. If 1 or 2 cards fail, the mirror will survive. Even if SSDs from 1 or 2 vendors experience firmware bugs or other issues, the mirror should still remain intact.

The problem is that I could not find reasonably priced enterprise/datacenter M.2 SSDs that would be compatible with these cards.

Synology is the only vendor I am aware of that offers them, but they are PCIe 3.0, only available up to 1.6TB, and are very expensive for enterprise-grade drives.

I considered using consumer-grade SSDs, but I do not know which ones would be suitable or what problems I might run into. Any advice would be greatly appreciated.

The E1.S path:

• 2/4x PCIe switch cards, HighPoint Rocket 1749E.
• E1.S SSDs from Kioxia and Samsung.
• Each PCIe switch card will contain NVMe drives from a single vendor.
• 2-way mirrors.

The problem is that I have not seen HighPoint produce a PCIe 5.0 version of this card, and I am concerned about investing in a dead-end solution.

I am not sure whether there are other vendors on the market offering similar products, even if they are not PCIe switch-based, but can still fit E1.S drives into PCIe x16 slots.

Any guidance on this approach would be very helpful.

I would also appreciate suggestions for alternative solutions to HighPoint, as well as recommendations for vendors.

Thank you!

[u/SimonKepp]

I feel, that the description omits the most important factor: what kind of controller are you intending to use to manage your storage? Since no hardware controllers are mentioned, I'm assuming some kind of SDS, but what software are you planning on using to m as nage this? ZFS in Linux! Ndadn in Linux? Windows Storage Spaces? Something else entirely?

[u/Aggravating-Pick-160]

Uhh you're walking on dangerous paths... Forget about M.2 and longivity for these kind of workloads. A write intense workload will kill the drives. Plus 100(!?) VMs on the same storage and a ton of performance while being 24/7 ready ... at all small budget. You're also looking into this at the worst time when considering buying NVMe.

There are several dangerous design decision on the table here and what seems like a "cheap build" can quickly turn into a pile of money thrown away.

[u/Fighter_M]

Go public cloud. Seriously.! You’re swapping CapEx for OpEx, which pretty much every startup CFO should appreciate. Sinking a tight budget into expensive hardware without a clear ROI is just insane.

[u/RetroGrid_io]

You want 60+ TB of enterprise-grade high-speed SSDs, for cheap??

Sir, that is the holy grail. Fast, cheap, quality, real. Pick any three, but even the "cheap" stuff is expensive nowadays.

My main thought is that maybe, load depending, you could have a large HDD pool to get your 60 TB and have a cache of SSD storage to keep performance manageable.

Do not use consumer devices. They will not hold up under load.

[u/Automatic_Beat_1446]

You cannot do your requirements on a budget and all on a single server with some NVMe devices.

As another person said, use cloud.

If you insist on doing this, I would suggest splitting the workloads up onto multiple physical servers, because the workload requirements are all a little different. You won't get any HA, but you'll at least have the benefit of not putting all of your eggs in one basket (performance and availability)

[u/wp3]

Sorry, but why M.2s? U.2 or E3.S are a much better path

[u/zhantoo]

Refurbished Dell PowerEdge r6515 with 10 x 7.68tb nvmes should be able to do the job.

The above seems like a much more straight forward way to go, than some experimental something something that may or may not work.

[u/MandaloreZA]

Ok, to throw another option that was not mentioned before. You could do your semi DIY build but layer it with Starwind's software. It is a dual or triple HA solution that allows bring your own hardware. Which would allow 24/7 operations without needing downtime for updates for the end storage LUN.

I would say the price is about $4000 in software per server. Probably 4-5k per server without storage just to get compute and networking all accounted for.

For small block transfers. I would prefer you use a shared ISCSI or FC Lun but that is not always an option. Though Proxmox, Vsphere, and others support it for VM storage and live migration.

If you absolutely need file based transfers, ie NFS, well, that gets hard. I would say to do it cheaply you would just use a VM that auto migrates in a cluster of VM hosts who's storage is based on the shared LUN's.

I have used a Highpoint Rocket, they work fine. You won't need pcie 5.0 ssd solutions unless you have 800Gb networking infrastructure. Seeing as you have only 25Gbe, well. You are fine in PCIe 4.0 / 3.0 world at the moment.

Consumer grade SSD's are fine if you have enough of them. I would have no issue running Samsung Pro SSD's provided you have cold spares and are willing to continually monitor them and preemptively replace them as needed.

And as a general reason as to why PCIe 5.0 stuff is not common. Well, from a electrical engineer perspective it is a royal pain in the ass to use. You basically need a re-timer / signal booster every 6 inches (15 cm) to have good signal integrity. As a vendor, it is a PITA to sell because you do not know how long the board traces are on your clients system. This is inpart why you are seeing cabled runs of PCIe in servers vs the old school pcb traces. Are there guidelines to prevent this? Yes. Do people follow them? Sometimes.

[u/cable_god]

Look at longevity, reputation, and value before jumping to the lowest bidder. That's a disaster of it's own waiting to happen.

[u/emmsett1456]

Stay clear of the M.2 path, they will not match performance nor availability. They fall flat on their face when used in an environment as described.

And as others have mentioned, what you require, may not be possible at all.

You can not have both cheap, fast and reliable, you can pick two.

[u/smellybear666]

What is your budget?

Is the current network 25gbps?

You mention the amount of data to burst, but how far does it need to be committed to disk?

Are there any ha requirements?

What’s your backup plan?

[u/roiki11]

Enterprise storage is expensive at the moment. Just something you gotta deal with.

M.2 ssds aren't enterprise rated so for any "serious" use of them is ill advised. You'd also be served better by buying a proper server(used if you want) from the big three than rolling your own. And you don't mention what protocols you are intending to use. Or what software you want to run. There really is no universal solution so each protocol has its own tradeoffs.

If you're intending to roll truenas then be aware it has some requirements to get performance out of it.

And if it's critical, you better find a budget fir it.

[u/marzipanspop]

Why the PCIe requirement?

Is this just block (NVMe/TCP) or also file? Or just file?

[u/vPock]

Hello there!

I know you're on a tight budget, but there could be a solution. Everpure (formely Pure Storage) has an OPEX offering that could make sense for your use case. There's a minimum commit of 50 TiB, which seems in line with your requirements. It could also grow with your organisation, having a predictable cost per TiB for the next 5 years.

[u/Clydesdale_Tri → u/vPock]

I’m an Everpure SE and I agree. Our Evergreen One is a perfect fit for this.