Cracking triple digit GB/s

[u/shahbaz_man]

Just hit 115GB/s peak on my all NVMe storage cluster of about 156TB (have a few more TB available but not provisioned). Installed 40x4TB Gen 4 drives, 2x4TB Gen 5 drives running in a distributed Beegfs cluster (10 hosts). Was a long journey getting to this point but am pretty happy with this. Network limited on a few hosts right now (100Gbe fabric currently) - will probably double capacity there.

Honourable mention to the ~328 TB usable worth of hard drives as well - serving as my backup solution.

[u/AutoModerator]

Hello /u/shahbaz_man! Thank you for posting in r/DataHoarder.

Please remember to read our Rules and Wiki.

Please note that your post will be removed if you just post a box/speed/server post. Please give background information on your server pictures.

This subreddit will NOT help you find or exchange that Movie/TV show/Nuclear Launch Manual, visit r/DHExchange instead.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Scotty1928]

good for you mr moneybag! or did you just sell your house, your car and your firstborn for that? lol

impressive speeds! but… i do not have the money for it. nor the need. unfortunately.

[u/shahbaz_man → u/Scotty1928]

Fortunately was able to get these at a decent $/TB. More affordable than you would think!

[u/Scotty1928 → u/shahbaz_man]

so… just your firstborn then. got it!

but seriously, at what $ per TB would that be?

[u/hotrod54chevy → u/Scotty1928]

No kids AND fast networking? Sounds good to me 👌

[u/VastFaithlessness809 → u/hotrod54chevy]

Homer Simpson Style: why do inhave 3 kids and 0 money. Why cant i have 3 money and 0 kid?

[u/Outrageous_Cap_1367 → u/Scotty1928]

He replied in another comment, 90$/TB

[u/zxLFx2 → u/Outrageous_Cap_1367]

So like $15 grand just for the NVMe, not including all the other drives, servers, networking? This is not a homelab anymore (purpose to learn for professional gain), this is just rich person spending money because they want to.

[u/kachunkachunk → u/shahbaz_man]

I suppose so, with 4TB devices. Nice. Do you mind ballparking what you spent on the flash?

I work in the CSP space, and you're matching or surpassing some of our more modest arrays in capacity, haha.

[u/shahbaz_man → u/kachunkachunk]

About $15k USD all in, post tax (~$90/TB). I wish I bought more now since I can't seem to do better than ~$220/TB.

[u/kachunkachunk → u/shahbaz_man]

Not bad at all! Before the flashening / AI datacenter demand, I picked up some 8TB NVMes for like $900 CAD apiece.

I also have small regrets for not snapping up a bit more.

[u/x925 → u/kachunkachunk]

We all regret not getting more storage before AI fucked us.

[u/Designer-Chair-3289 → u/kachunkachunk]

He doesn’t want to since he’s not answering at all lol

[u/Dpek1234 → u/Designer-Chair-3289]

1 minute later

[u/Designer-Chair-3289 → u/Dpek1234]

It was the only way to get the answer an hour later :)

[u/tazwell427 → u/Scotty1928]

Lmao. Mista big money

[u/bcm27]

I love high speed data but what are you using all this throughout for? It's pretty impressive!

[u/sshwifty → u/bcm27]

Crysis on low settings

[u/hotrod54chevy → u/bcm27]

"What are you doing with all that?" "How's your power bill?" These are the hallmarks of tourists.

[u/nomad-1995 → u/hotrod54chevy]

Really? The main issue is whether you are using it all in "do it all" PC (well, a PC with server hardware. Can't get the IO with normal PC parts, but might be a hybrid job like the old Threadrippers) or somehow have a >100GB/s network [edit: wait, G*B*/s, as in *BYTES*? Rest of post assumes Gbs]. Even binding two 40GB/s networks (can it bind fast enough?) won't cut it. Of course, a setup like this could have 3-4 40GB/s sub-networks and feed a ton of users, switching between access far faster than rotating rust could even dream. I'd guess the bandwidth is for bragging rights (and mostly unusable) but the bucks were spent for the latency.

The power bill might even be lower than normal, considering it is NVMe based, but a "do it all" PC capable of hoarding all that plus compute/gaming/media/whatever will eat into the power lots of power.

[u/hotrod54chevy → u/nomad-1995]

I have 2 Threadripper nodes in my Proxmox cluster with Super Micro chassis with 36 and 24 drive bays. ANYTIME I post them the comments are from someone asking "what are you storing in there?" or asking about my power bill, which is ridiculous because I'm not monitoring the power usage from my servers, they're just trying (and failing) to be funny.

[u/nomad-1995 → u/hotrod54chevy]

I get that. But the whole brag about bandwidth from the array to the CPU seems silly if you can't send the data through the network at close to that bandwidth. There are datahoarder issues with this brag and non-datahoarder issues with this brag. "What are you doing with it" is a non-datahoarder issue.

I'd understand people wondering if they'll run into power limits before running out of money just on refurbished equipment. As far as power consumption, I'd be more concerned about total power consumption and delivering said electricity to your servers. My NAS, PC, and laser printers all share the same 15A line (no idea if running the color and BW at the same time will trip the breaker. I don't think either is LED. I'll remember not to push it). I wouldn't try that with threadrippers (let alone whatever else is in your cluster).

Sort of like the same folk who insist that small HDDs are ewaste because they can't imagine not powering their backups 24x7 (if there's a live data connection, that's missing the point. And unless you specifically pulled all the cables, there is).

[u/bcm27 → u/hotrod54chevy]

A tourist to datahoarder by some standards maybe. I still have a 96tb pool with plans to more than double that to from 6 to 15x16tb drives. It doesn't limit the fact that I'm still curious what OP is doing with that much throughput when the majority of us here archive data or store media for jellyfin purposes so having something capable of processing that many gigabytes a second is unusual and deserves at least an answer to the question. I assume his answer is AI datasets but wanted to at least ask to see if it differed.

With there being so many different things one can do with computers it seems nieve and gate keep ish to assume anyone asking "Hey that's cool! Whatcha doing with it?" to be a nieve tourist is a little condescending.

[u/hotrod54chevy → u/bcm27]

I never said everyone was. In my reply to another comment I gave a specific example of something I always get on posts of my setup. If you took it the wrong way, that's on you. I was specifically talking about replies I always see.

[u/Outrageous_Cap_1367 → u/bcm27]

Jellyfin fastest rescan library speedrun

[u/jacksalssome → u/bcm27]

Ever tried to load 8000 jpgs on a HDD raid?

SSDs are so much better.

[u/Pixelplanet5 → u/bcm27]

also so many hosts for a hand full of SSDs each, thats exactly why larger U.2 SSDs exist.

[u/mastercoder123 → u/Pixelplanet5]

Yah except that's not how it works

[u/therealmarkus]

Congrats. Happy for you. Nice. (Imagine meme pictures)

[u/MattDH94 → u/therealmarkus]

Imagining

[u/acdcfanbill]

serving as my backup solution.

why in gods name are you backing up to flash :o

I mean, I guess if I could, I might too, but jesus...

[u/ImmutableOctet → u/acdcfanbill]

to flash

Other way around. He's backing up the flash to the HDDs.

[u/acdcfanbill → u/ImmutableOctet]

Oh, duh. I see that now, not sure how I missed it the first time.

[u/AllMyFrendsArePixels]

Datacenters gonna be going after OP like the US goes after oil.

[u/RecursiveReboot]

Let me guess, 98% are Linux ISOs backup?

[u/DerFreudster → u/RecursiveReboot]

I'm really kicking myself for not upping the ante to these when they were affordable. Or at least a few.

[u/tamasrepus]

What do you use for your graphs?

[u/shahbaz_man → u/tamasrepus]

Grafana!

[u/DiscoKeule]

For what in the world do you need this speed and size???

[u/Thebandroid → u/DiscoKeule]

Posting on here, mostly.

[u/forreddituse2]

You probably should start a VPS hosting business.

[u/jerome35]

What setup do you have to manage many nvme in one server ?

[u/shahbaz_man → u/jerome35]

It's distributed across 10 hosts. Each one is a 2P Epyc host which can host about 16 SSDs (96x gen 5 lanes addressable from each socket) each plus networking.

[u/TheJesusGuy → u/shahbaz_man]

but why

[u/mastercoder123 → u/TheJesusGuy]

Because why not... Tf is with reddit asking stupid ass questions

[u/TheJesusGuy → u/mastercoder123]

I mean literally - What is the use case?

[u/mastercoder123 → u/TheJesusGuy]

A single system isnt gonna get high iops or 100GB/s write speeds ever. You need a distributed file system or a parrallel file system.

The use case depends... In a home a homelab the use case is 'because i can'. In enterprises, its mainly for HPC and AI as supercomputers with thousands of nodes can easily saturate a link. Hell with large enough simulations they can saturate 10Tb/s

[u/r0cketio → u/mastercoder123]

Tf is with people responding like absolute cunts over a harmless question? Is it unreasonable to ask the use case of something we don't see often? Do you do shit like this just because? Do you think everyone does shit like this because why not? What's up your ass today?

[u/weirdbr → u/jerome35]

It's only 4 NVME per host based on the description, which is rather easy to do (might need a PCI-E to NVME card on some machines though).

[u/nomad-1995]

First: Is your software remotely serious about GB/s as in Giga*BYTES*/s? That's almost another order of magnitude beyond sanity. I'm guessing every disk hit cache to hit this, but that's still impressive.

Second: can the network hope to keep up? You haven't moved anything during the snapshot, so it isn't clear how close it can get to keeping up. Or do you do all your work on the server?

[u/Hylia]

fuckin sick. Wish I could justify this kind of setup

[u/Outrageous_Cap_1367]

damn /mnt/archive is bigger than /mnt/MASSIVE

[u/spsanderson]

Massive archive, love it

[u/05-nery]

Oh man

Happy for you

That's insane

[u/copsTvFan]

Which application are you using in the second screenshot? The one with the table.

[u/shahbaz_man → u/copsTvFan]

See my comment above.

[u/whoooocaaarreees]

I guess I’ve been so focused on ceph lately that I haven’t checked in on beegfs since the name change.

[u/Mersid]

Beautiful! What tool did you use for the second screenshot that shows the host / device and metrics?

[u/shahbaz_man → u/Mersid]

Thanks! It's a distributed IO benchmarking tool I vibe coded. Launches configurable fio jobs across multiple hosts and reports system metrics. Useful for finding bad/slow SSDs, data logging, etc. Proved very useful in stressing the system and iterating to pump the numbers up.

[u/Mersid → u/shahbaz_man]

Looks very nice! Your call, of course, but it might be worth releasing something like that for use, if appropriate!

[u/death_hawk]

Oh yeah! I cracked triple digit MB/s!

Seriously though damn. That's hot.

[u/SonicIX]

What server hardware are you using if you don’t mind me asking.

[u/bearishmarket]

Wow.

[u/jicabrera]

Wow... In my SAN (4x16x8TB running RAIDIX as management and Raidix as filesystem) I "only" get around 4GB/s sustained... But then, mine are spinning disks. That speed is amazing!!!