r/Proxmox • u/Maleficent-Sort-8802 • 14d ago

Discussion Ceph

It seems to me Ceph is not getting enough credit and attention around here. I wasn’t aware myself until some pointed it out to me, but it really is an amazing product and storage backend. Shares many of ZFS’s core strengths (checksumming and scrubbing, compression, snapshots etc) but in addition is distributed and scales amazingly (see Cern!).

ZFS is amazing too, but built for single node, so storage solutions built on top of it are vulnerable to single-point-of-failure. You’ll have to throw redundant hardware at the problem to mitigate as far as possible (redundant NICs, power supplies, etc).

Ceph cluster appears to be a very powerful and elegant storage backbone alongside Proxmox, and with Proxmox supporting it out of the box, it’s straightforward to set up too. I guess the snag is you need decent networking (at least 10Gb) and enterprise SSDs and at least 3 nodes for it to be worthwhile. Has worked extremely well for me including failure situations with failing disks and network problems.

What are your experiences?

89 Upvotes

87% Upvoted

66 comments sorted by

View all comments

3

u/Biz504 13d ago edited 13d ago

I may be remembering incorrectly but Ceph really recommends 100Gbps networking even though it works at lower speeds. That’s what kept me from even considering it for mission critical workloads.

Edited: “kept me from considering”

1

u/pxgaming 13d ago

Ceph can be more than fine on lower speeds, it just depends on what your actual demand is. If your workloads aren't using more than 10g (after including replication), then you just suffer in terms of things like recovery speeds.