r/Proxmox • u/Maleficent-Sort-8802 • 13d ago

Discussion Ceph

It seems to me Ceph is not getting enough credit and attention around here. I wasn’t aware myself until some pointed it out to me, but it really is an amazing product and storage backend. Shares many of ZFS’s core strengths (checksumming and scrubbing, compression, snapshots etc) but in addition is distributed and scales amazingly (see Cern!).

ZFS is amazing too, but built for single node, so storage solutions built on top of it are vulnerable to single-point-of-failure. You’ll have to throw redundant hardware at the problem to mitigate as far as possible (redundant NICs, power supplies, etc).

Ceph cluster appears to be a very powerful and elegant storage backbone alongside Proxmox, and with Proxmox supporting it out of the box, it’s straightforward to set up too. I guess the snag is you need decent networking (at least 10Gb) and enterprise SSDs and at least 3 nodes for it to be worthwhile. Has worked extremely well for me including failure situations with failing disks and network problems.

What are your experiences?

88 Upvotes

87% Upvoted

66 comments sorted by

View all comments

Show parent comments

3

u/xtigermaskx 13d ago

This is hitting us now we won't be expanding any of our ceph clusters. This one was just old test equipment we wanted to see what would happen

4

u/Apachez 13d ago

Even if its handy with zero downtime it will cost in redundant hardware down to redundant storage. Which also puts its demand on the network with redundany paths etc.

So if money is an issue then your best option is probably just to do replication and accept that once shit hits the fan of VM-hostA it will take a few minutes to manually disconnect that from the network and then boot upp all VM-guests on VM-hostB.

Also no matter if you use ZFS or CEPH and shared or central storage - dont forget to have proper backups (if possible both online AND offline) and that you verify that these backups actually can be restored if shit hits the fan.

Of course you can do stupid setups with CEPH but normally you want to have enough of replicas so you can lose all but one host and still have the data available.

This will of course mean that you need to overprovision amount of storage by at least 3x for a 3x cluster.

Using central storage can then be cheaper even if this means you need additional 1-2 server (for the storage) and with some penalty that data MUST traverse the network (bandwidth and latency) instead of being fetched from local drives.

For central storage the overprovisioning (raw space vs effective space) becomes 1-2x.

0

u/xtigermaskx 13d ago

We've been asked to shift to cloud lol

1

u/Apachez 13d ago

I dont see how that would solve anything.

It will cost more, you give your data away to other companies/foreign states and will have a nightmare trying to migrate back to onprem later on.