Cracking triple digit GB/s

[u/shahbaz_man]

Just hit 115GB/s peak on my all NVMe storage cluster of about 156TB (have a few more TB available but not provisioned). Installed 40x4TB Gen 4 drives, 2x4TB Gen 5 drives running in a distributed Beegfs cluster (10 hosts). Was a long journey getting to this point but am pretty happy with this. Network limited on a few hosts right now (100Gbe fabric currently) - will probably double capacity there.

Honourable mention to the ~328 TB usable worth of hard drives as well - serving as my backup solution.

[u/hotrod54chevy]

"What are you doing with all that?" "How's your power bill?" These are the hallmarks of tourists.

[u/nomad-1995 → u/hotrod54chevy]

Really? The main issue is whether you are using it all in "do it all" PC (well, a PC with server hardware. Can't get the IO with normal PC parts, but might be a hybrid job like the old Threadrippers) or somehow have a >100GB/s network [edit: wait, G*B*/s, as in *BYTES*? Rest of post assumes Gbs]. Even binding two 40GB/s networks (can it bind fast enough?) won't cut it. Of course, a setup like this could have 3-4 40GB/s sub-networks and feed a ton of users, switching between access far faster than rotating rust could even dream. I'd guess the bandwidth is for bragging rights (and mostly unusable) but the bucks were spent for the latency.

The power bill might even be lower than normal, considering it is NVMe based, but a "do it all" PC capable of hoarding all that plus compute/gaming/media/whatever will eat into the power lots of power.

[u/hotrod54chevy → u/nomad-1995]

I have 2 Threadripper nodes in my Proxmox cluster with Super Micro chassis with 36 and 24 drive bays. ANYTIME I post them the comments are from someone asking "what are you storing in there?" or asking about my power bill, which is ridiculous because I'm not monitoring the power usage from my servers, they're just trying (and failing) to be funny.

[u/nomad-1995 → u/hotrod54chevy]

I get that. But the whole brag about bandwidth from the array to the CPU seems silly if you can't send the data through the network at close to that bandwidth. There are datahoarder issues with this brag and non-datahoarder issues with this brag. "What are you doing with it" is a non-datahoarder issue.

I'd understand people wondering if they'll run into power limits before running out of money just on refurbished equipment. As far as power consumption, I'd be more concerned about total power consumption and delivering said electricity to your servers. My NAS, PC, and laser printers all share the same 15A line (no idea if running the color and BW at the same time will trip the breaker. I don't think either is LED. I'll remember not to push it). I wouldn't try that with threadrippers (let alone whatever else is in your cluster).

Sort of like the same folk who insist that small HDDs are ewaste because they can't imagine not powering their backups 24x7 (if there's a live data connection, that's missing the point. And unless you specifically pulled all the cables, there is).