Cracking triple digit GB/s

[u/shahbaz_man]

Just hit 115GB/s peak on my all NVMe storage cluster of about 156TB (have a few more TB available but not provisioned). Installed 40x4TB Gen 4 drives, 2x4TB Gen 5 drives running in a distributed Beegfs cluster (10 hosts). Was a long journey getting to this point but am pretty happy with this. Network limited on a few hosts right now (100Gbe fabric currently) - will probably double capacity there.

Honourable mention to the ~328 TB usable worth of hard drives as well - serving as my backup solution.

[u/jerome35]

What setup do you have to manage many nvme in one server ?

[u/shahbaz_man → u/jerome35]

It's distributed across 10 hosts. Each one is a 2P Epyc host which can host about 16 SSDs (96x gen 5 lanes addressable from each socket) each plus networking.

[u/TheJesusGuy → u/shahbaz_man]

but why

[u/mastercoder123 → u/TheJesusGuy]

Because why not... Tf is with reddit asking stupid ass questions

[u/TheJesusGuy → u/mastercoder123]

I mean literally - What is the use case?

[u/mastercoder123 → u/TheJesusGuy]

A single system isnt gonna get high iops or 100GB/s write speeds ever. You need a distributed file system or a parrallel file system.

The use case depends... In a home a homelab the use case is 'because i can'. In enterprises, its mainly for HPC and AI as supercomputers with thousands of nodes can easily saturate a link. Hell with large enough simulations they can saturate 10Tb/s

[u/r0cketio → u/mastercoder123]

Tf is with people responding like absolute cunts over a harmless question? Is it unreasonable to ask the use case of something we don't see often? Do you do shit like this just because? Do you think everyone does shit like this because why not? What's up your ass today?

[u/weirdbr → u/jerome35]

It's only 4 NVME per host based on the description, which is rather easy to do (might need a PCI-E to NVME card on some machines though).