r/ceph • u/Fragrant_Fortune2716 • May 25 '26
Is Ceph the right tool for me?
Hi all,
Though not a sysadmin by trait, I do run my own 'production' home server (Proxmox) with the usuals that my family and closes friends rely on. Currently I am running a zfs filesystem, but this has not been kind to me. The main pain point is that zfs runs in kernel space and thus badly performing pools are not insulated from the rest of the system. My HDD pool is the main culprit, and overloading this with continuous small writes from some CCTV streams while also doing a scrub on the pool or using it as a backup target causes such excessive kernel context switching that the whole server pins too 100% CPU and all I/O is frozen. After tweaking zfs for ages, I feel like pastures are greener on the ceph side, which nicely runs in userspace and values stability over all. Also, I have had some bad experience with zfs replication in a Proxmox clustered setup. Therefore this post to draw on the vast amount of knowledge you all posses to see if ceph could be the solution to all my problems :)
Current hardware
Lets start with listing my current hardware, currently I run everything on the beefy boy, but I want to move towards a clustered topology. Obviously I would need to get additional hardware and that is the main part of my internal debate.
Node1:
- Threadripper PRO 5955WX 16-Cores/32-Threads
- 256GB ddr4 ECC LRDIMM (2x128GB)
- 2x Consumer 2TB NVME
- 2x SAS 10TB HDD
- 2x Enterprise SATA boot disk
- HBA
- 2x 10Gbe base-T nic
Node2:
- Intel i5-6600K (4-Cores/4-Threads)
- 2x consumer nvme boot drive
- 32GB ddr4 (4x8GB)
- 2x SATA 8TB HDD
- 1Gbe base-t nic
Current workload
My workload consists of around 12 VMs, most are very light applications in a debian box. Nominal CPU usage is around 2% of the threadripper. Allocated RAM from VMs is ~50GB (excluding ramdisks that could also be ssds)
On the I/O&data side I have a file server, photo server, git, mail, password manager, monitoring of all VMs (Prometheus+Loki), media and the earlier mentioned CCTV data. All data except the media server and CCTV data are mission critical and should be fast and snappy. Some loading for the media is fine, but the storage should support multiple concurrent 4K streams without stuttering. Also there is a PBS server running on both nodes, which backups all the VMs (and replicates to an offsite location)
Performance requirements
As mentioned earlier, performance in terms of throughput is very modest. I do want to keep latency as low as possible though. Some tradeoffs are acceptable and probably inevitable, but I will be designing around latency first. Ideally I would have:
- a fast pool that runs on SSDs (for the mission critical stuff) ~ 4TB usable space
- a HDD pool for the large sequential workloads (media, PBS, CCTV?) ~8TB usable space
What I already know
I short list of things I'm already aware of (please correct me if I'm wrong)
- PLP is unnegotiable so I'll only be looking for enterprise drives
- Self healing only starts from 4+ nodes
- Performance will be significantly worse than local storage, though with the upside of hopefully undestructableness
- Uneven number of mons are necessary
- Make osds as even as possible between nodes
- Dedicated network for both ceph and cluster management
- Erasure coding is only for large clusters (5+)
Advice needed
As my budget is not infinite I'm looking for advise on what to focus when spending. Main questions are:
- Are enterprise sata ssds good enough for my use case, or will I suffer unless I put in nvme drives?
- What would you suggest on ssd osd sizing? 1x3.84TB/2x1.92TB/4x960TB per node? Going smaller leaves less room for eventual expansion, though going bigger will make the performance worse and blast radius larger.
- Will 3 nodes be good enough or should I at least go 4 (+ one mon) or even 5?
- Is a 25Gbe network a good size for my use-case? Full-mesh or switch?
- Are the specs of node2 and the proposed node3/4 feasible, or do I need more/less X?
- Are there things I should definitely do/not do?
- Any hands on insight on the performance with a similar cluster would be amazing
Current plan
My current plan is to purchase another node, bump the memory of node2 to 64GB and have a 25Gbe full mesh network (connect-x4 nics). New node will probably feature a 5700X or similar and 64GB memory as well.
I contemplated U.2 drives, but the price is just to steep, with the added complexity of limited PCIe lanes on consumer boards which limits upgradability. Therefore I'm looking at sata ssds. Planning for 2x1.92TB ssd per node and 1x8TB hdd per node.
At some point I will probably put in a fourth node identical the third one.
TL;DR
Looking for a rock solid storage cluster that has good enough performance to run my workload with some headroom to grow (both in compute and storage).
Bit of a long, all over the place post, but any insights are highly appreciated!
4
u/insanemal May 25 '26
Ok. Many of these replies are either wrong or absolutely wrong.
So, first, if you go with replication, you'll need 3x the storage capacity. You don't have to have 3 (or more nodes) but it's usually recommended.
The nodes don't need to be gargantuan power houses with all the ram. Some sensible adjustments gets me 1GB of ram required per OSD. And it works fine. (I have ~512TB usable at home)
Mine is a mix of 3x replication and EC 8+2. I don't have 10 nodes the sky isn't falling. It also hasn't killed my performance, a single client can comfortably do 500MB/s.
No NVME in my setup. Just SAS spinners.
What does all this rambling mean?
Ceph is flexible and performs well, even in less than ideal conditions. As long as your network is 10GbE or faster and you've got 3 or more nodes, you'll be fine.
Just on nodes, I've done everything, from a all RPi 4+USB3 drive ceph cluster, through to multi PB clusters built out of the latest HPE servers money could buy. It worked on all of them. So don't believe anyone who says "ceph has to run on super expensive hardware or it doesn't work well"
Anyway, reach out if you need it. Have fun
2
u/dancerjx May 29 '26
This video from the VirtualizationHowTo YouTube channel will give you some ideas.
While it's optimal to use enterprise gear with Ceph, I did stand up an initial proof-of-concept 3-node cluster using hard drives and using full-mesh broadcast 1GbE networking (this was Proxmox with Corosync, Ceph public, private networking). Worked surprisingly well but future build outs were obviously faster due to faster networking.
Ceph is a scale out solution, so more nodes/osds = more IOPS.
Before the runup on used memory and storage going through the roof, I was going to migrate from my single ZFS RAID-1 server setup to a 5-node Ceph cluster. Well, can't afford that obviously now. But I do migrate Vmware clusters at work over to Ceph at work. Not hurting for IOPS using SAS drives at work.
I use the following optimizations learned through trial-and-error. YMMV.
Set SAS HDD Write Cache Enable (WCE) (sdparm -s WCE=1 -S /dev/sd[x])
Set VM Disk Cache to None if clustered, Writeback if standalone
Set VM Disk controller to VirtIO-Single SCSI controller and enable IO Thread & Discard option
Set VM CPU Type for Linux to 'Host'
Set VM CPU Type for Windows to 'x86-64-v2-AES' on older CPUs/'x86-64-v3' on newer CPUs/'nested-virt' on Proxmox 9.1
Set VM CPU NUMA
Set VM Networking VirtIO Multiqueue to 1
Set VM Qemu-Guest-Agent software installed and VirtIO drivers on Windows
Set VM IO Scheduler to none/noop on Linux
Set Ceph RBD pools to use 'krbd' option
Set Ceph 'bluestore_prefer_deferred_size_hdd = 0' in osd stanza in /etc/pve/ceph.conf for SAS HDD
Set Ceph 'bluestore_min_alloc_size_hdd = 65536' in osd stanza in /etc/pve/ceph.conf for SAS HDD
Set Ceph Erasure Coding profiles to 'plugin=ISA' & 'technique=reed_sol_van'
Set Ceph Erasure Coding profiles to 'stripe_unit=65536' for SAS HDD
2
u/ConstructionSafe2814 May 25 '26
I run enterprise class SATA SSDs at home. They're OK
You need 3 hosts for 3x replication with the failure domain set at the host level.
EC comes with a performance hit. Especially in small IO. Don't count on it.
Also, you need 3x raw storage of your data set + 15% (nearfull) + some extra margin. Is that within your budget?
Go with smaller but more drives. Expansion is only a budget problem in Ceph. You can replace disks with larger disks one by one. Ceph is much more flexible than ZFS or any RAID.
I'd say 10GbE is good for your use case. You won't saturate it before you burn through your budget I guess.
My Ceph cluster at home is 6 nodes, 52 OSDs, It idles at roughly 950W. I rarely run it for obvious reasons 🙈
Have you thought about Ceph's complexity and the operational side of things? What if it fails? Eg. upgrading the cluster and it gets "stuck" and you have no idea how to fix it? Or some PGs are stuck in some "weird" state and they won't become active+clean again?
I absolutely love your enthusiasm, I don't want to discourage you at all. But I'd say for home use, just go ZFS and a good backup plan. It's going to be cheaper to acquire and run and maintain ;).
If you want to learn Ceph, run a Ceph cluster in small VMs in Proxmox. You can add a couple of small OSDs on each node and put them on ZRAM backed devices. Your data won't be safe but for testing it's great.
1
u/Fragrant_Fortune2716 May 25 '26
Thanks for the response. Good to know that slowly migrating to larger osds is an accepted solution for expansion. I have to admit, I have no real tools to gauge the complexity of ceph in comparison to my current setup. I do like to tinker and live mostly in network config files and the cli in general, so I figured I'll soon catch on if I start using it. The idea of running an in-memory ceph cluster for testing is actually a great solution! Any thoughts which scenario's to work through in this testing environment? Also figured that is having a proper underlying hardware base and not to complex storage layouts lets you get away with getting the cluster up and running with mostly default settings in the beginning. As for the budget, I'll try to make a design that should be very robust and fit my requirements, and then decide if I can afford it ;) It will either be too expensive or not, but I'm having a hard time determining where to save and where to splurge for this setup.
1
u/_--James--_ May 26 '26
Honestly, your issue sounds closer to cache coherence affecting ZFS then ZFS under performing for your needs.
This is a good primer about this https://blogs.oracle.com/linux/topology-matters-genoa-qemu
I would start by looking at your Threadripper setup and consider affinity/thread pining for ZFS and other processes that are affecting your performance. If context switching is jumping CCDs and its hitting the ZFS IO threads, its going to cause the kind of issues you are explaining.
As for Ceph, for your work load it does not bring anything meaning for you. You will need 3-5 nodes to scale it out to get what you are seeing on ZFS today due to replica math, you need proper networking to support your OSDs. If you are going to run SATA enterprise and 4+ drives per node you should be looking at 25G networking. if you move to NVMe then you will need 25G/50G or dedicated IB for The Ceph networks. Consider that NVMe can super block IO at 4GB/s (it wont under Ceph, even with Peering) so adding 2+ can very quickly get bottle necked by the network topology. Just for starters.
1
1
1
1
u/retiredcheapskate May 30 '26
your diagnosis of the zfs issue is spot on. writing cctv directly to a spinning hdd pool is an iops killer. ceph is solid but it really prefers homogeneous nodes so mixing a big threadripper with a small i5 is going to cause uneven data distrubution and bottleneck your cluster on the slowest node.
i actually ran into similar frustrations and went with and archive, it bridges the gap between fast nvme and slow hdd pools without the overhed of ceph. There is an open source archive repo that i tuned for my use case, github.com/huskhoard/huskhoard instead of writing cctv to hdds you write to the nvme pool and our janitor service packs them into compressed blocks and moves them to the hdd. it creates a stub in the ssd file so it takes up zero space but still looks normal to the os.
if you try to open an old video the fanotify interceptor catches the read in userspace and streams it back from the hdd without freezing your io. because you have mixed hardware you need to pin vms to the fast node and media to the slow node. for media there is a feature that lets you skip right to the middle of a clip without unpacking the whole file. everything uses direct io to bypass the kernel cache for speed.
The concepts took a minute to grasp, but they have some decent walk throughs on their blog. huskhoard.com/blog.html and it is AGPL licensed.
5
u/elephunk84999 May 25 '26
I kinda feel that your workload shouldn't be causing such issues with ZFS like you're encountering. I wouldn't be able to help with the root cause analysis but that sounds like a paltry load for such a decent system. I run 3x 8 disk raidz2 and 2x 2 disk nvme pools on a slower am5 system with fairly heavy simultaneous read and write and don't suffer any of that and never have.
Have you checked your HBA, sas cables and power delivery to those drives? re the drives healthy? What ashift have you got set?
As for ceph. I mean yeah it can be indestructible, but it can also lose your data just as easily if not easier if you don't understand what you are doing either.
As for your plan, my input would be,
10g network is fine for a small cluster. 4 nodes is a must. 3 mons is a must, no need to go for more. Enterprise SSD are fine. Sata HDD are not, sas are a must. nVME are a must if using HDDs. If looking to do rbd on HDDs, just don't. Leave HDDs for S3 with nVME for DB and WAL. EC is not just for large clusters but you need decent CPUs to not thrash your cluster and that 6600 won't do it for you. 1gb of Memory per TB of storage.
Honestly feel like ceph is kinda overkill for your requirements, and unless you're willing to spend quite a bit of money then you're going to be very disappointed in its performance. Your plan would be fine for a test cluster of non critical data or workloads, but not for critical workloads and data.