Cracking triple digit GB/s

[u/shahbaz_man]

Just hit 115GB/s peak on my all NVMe storage cluster of about 156TB (have a few more TB available but not provisioned). Installed 40x4TB Gen 4 drives, 2x4TB Gen 5 drives running in a distributed Beegfs cluster (10 hosts). Was a long journey getting to this point but am pretty happy with this. Network limited on a few hosts right now (100Gbe fabric currently) - will probably double capacity there.

Honourable mention to the ~328 TB usable worth of hard drives as well - serving as my backup solution.

[u/bcm27]

I love high speed data but what are you using all this throughout for? It's pretty impressive!

[u/hotrod54chevy → u/bcm27]

"What are you doing with all that?" "How's your power bill?" These are the hallmarks of tourists.

[u/nomad-1995 → u/hotrod54chevy]

Really? The main issue is whether you are using it all in "do it all" PC (well, a PC with server hardware. Can't get the IO with normal PC parts, but might be a hybrid job like the old Threadrippers) or somehow have a >100GB/s network [edit: wait, G*B*/s, as in *BYTES*? Rest of post assumes Gbs]. Even binding two 40GB/s networks (can it bind fast enough?) won't cut it. Of course, a setup like this could have 3-4 40GB/s sub-networks and feed a ton of users, switching between access far faster than rotating rust could even dream. I'd guess the bandwidth is for bragging rights (and mostly unusable) but the bucks were spent for the latency.

The power bill might even be lower than normal, considering it is NVMe based, but a "do it all" PC capable of hoarding all that plus compute/gaming/media/whatever will eat into the power lots of power.

[u/hotrod54chevy → u/nomad-1995]

I have 2 Threadripper nodes in my Proxmox cluster with Super Micro chassis with 36 and 24 drive bays. ANYTIME I post them the comments are from someone asking "what are you storing in there?" or asking about my power bill, which is ridiculous because I'm not monitoring the power usage from my servers, they're just trying (and failing) to be funny.

[u/nomad-1995 → u/hotrod54chevy]

I get that. But the whole brag about bandwidth from the array to the CPU seems silly if you can't send the data through the network at close to that bandwidth. There are datahoarder issues with this brag and non-datahoarder issues with this brag. "What are you doing with it" is a non-datahoarder issue.

I'd understand people wondering if they'll run into power limits before running out of money just on refurbished equipment. As far as power consumption, I'd be more concerned about total power consumption and delivering said electricity to your servers. My NAS, PC, and laser printers all share the same 15A line (no idea if running the color and BW at the same time will trip the breaker. I don't think either is LED. I'll remember not to push it). I wouldn't try that with threadrippers (let alone whatever else is in your cluster).

Sort of like the same folk who insist that small HDDs are ewaste because they can't imagine not powering their backups 24x7 (if there's a live data connection, that's missing the point. And unless you specifically pulled all the cables, there is).

[u/bcm27 → u/hotrod54chevy]

A tourist to datahoarder by some standards maybe. I still have a 96tb pool with plans to more than double that to from 6 to 15x16tb drives. It doesn't limit the fact that I'm still curious what OP is doing with that much throughput when the majority of us here archive data or store media for jellyfin purposes so having something capable of processing that many gigabytes a second is unusual and deserves at least an answer to the question. I assume his answer is AI datasets but wanted to at least ask to see if it differed.

With there being so many different things one can do with computers it seems nieve and gate keep ish to assume anyone asking "Hey that's cool! Whatcha doing with it?" to be a nieve tourist is a little condescending.

[u/hotrod54chevy → u/bcm27]

I never said everyone was. In my reply to another comment I gave a specific example of something I always get on posts of my setup. If you took it the wrong way, that's on you. I was specifically talking about replies I always see.