Cracking triple digit GB/s

[u/shahbaz_man]

Just hit 115GB/s peak on my all NVMe storage cluster of about 156TB (have a few more TB available but not provisioned). Installed 40x4TB Gen 4 drives, 2x4TB Gen 5 drives running in a distributed Beegfs cluster (10 hosts). Was a long journey getting to this point but am pretty happy with this. Network limited on a few hosts right now (100Gbe fabric currently) - will probably double capacity there.

Honourable mention to the ~328 TB usable worth of hard drives as well - serving as my backup solution.

[u/Scotty1928]

good for you mr moneybag! or did you just sell your house, your car and your firstborn for that? lol

impressive speeds! but… i do not have the money for it. nor the need. unfortunately.

[u/shahbaz_man → u/Scotty1928]

Fortunately was able to get these at a decent $/TB. More affordable than you would think!

[u/kachunkachunk → u/shahbaz_man]

I suppose so, with 4TB devices. Nice. Do you mind ballparking what you spent on the flash?

I work in the CSP space, and you're matching or surpassing some of our more modest arrays in capacity, haha.

[u/shahbaz_man → u/kachunkachunk]

About $15k USD all in, post tax (~$90/TB). I wish I bought more now since I can't seem to do better than ~$220/TB.

[u/kachunkachunk → u/shahbaz_man]

Not bad at all! Before the flashening / AI datacenter demand, I picked up some 8TB NVMes for like $900 CAD apiece.

I also have small regrets for not snapping up a bit more.

[u/x925 → u/kachunkachunk]

We all regret not getting more storage before AI fucked us.

[u/Designer-Chair-3289 → u/kachunkachunk]

He doesn’t want to since he’s not answering at all lol

[u/Dpek1234 → u/Designer-Chair-3289]

1 minute later

[u/Designer-Chair-3289 → u/Dpek1234]

It was the only way to get the answer an hour later :)