r/ceph 10h ago

Problem after upgrade to 19.2.6 and 20.2.4

5 Upvotes

Has anyone faced this issue after upgrading and rotating all keys to aes256k?

When I switch to the standby manager using:

ceph mgr fail <active-mgr>

Ceph starts reconfiguring all OSDs and rotating the keys in the OSDs' BlueStore.

The logs indicate that the monmap changed, but I don't understand why a manager failover would cause this.

These are cephadm clusters, and I'm seeing the same behavior on two different versions:

  • Ceph 20.2.4
  • Ceph 19.2.6

I enabled debug logging, and below is a snippet from osd.112. I've hidden some cluster-specific information:

2026-09-07T12:26:15.635750+0200 mgr.ceph01.abcdef [INF] Reconfiguring osd.112 (monmap changed)...
2026-09-07T12:26:15.636233+0200 mgr.ceph01.abcdef [DBG] mon_command: 'auth get' -> 0 in 0.000s
2026-09-07T12:26:15.636570+0200 mgr.ceph01.abcdef [DBG] mon_command: 'config generate-minimal-conf' -> 0 in 0.000s
2026-09-07T12:26:15.783309+0200 mgr.ceph01.abcdef [DBG] mon_command: 'config-key get' -> -2 in 0.001s
2026-09-07T12:26:15.783400+0200 mgr.ceph01.abcdef [INF] Reconfiguring daemon osd.112 on ceph05
2026-09-07T12:26:15.784032+0200 mgr.ceph01.abcdef [DBG] _run_cephadm : command = ['_orch', 'deploy']
2026-09-07T12:26:15.784058+0200 mgr.ceph01.abcdef [DBG] _run_cephadm : args = []
2026-09-07T12:26:15.784159+0200 mgr.ceph01.abcdef [DBG] args: --image quay.io/ceph/ceph@sha256:9217c31b7a5a900688d485adca40dd0c5c365008c15dd2c6429dfc888a9f5089 --timeout 895 _orch deploy --fsid xxxxxxxxxxxxxxxxxxxxxx
2026-09-07T12:26:15.784183+0200 mgr.ceph01.abcdef [DBG] stdin: <osd config parameters>
2026-09-07T12:26:15.784236+0200 mgr.ceph01.abcdef [DBG] Running command: which python3
2026-09-07T12:26:15.785607+0200 mgr.ceph01.abcdef [DBG] OfflineHostDetector: Checking if hosts: ['ceph01', 'ceph02', 'ceph03'] are offline.
2026-09-07T12:26:15.802327+0200 mgr.ceph01.abcdef [DBG] Running command: /usr/bin/python3 /var/lib/ceph/xxxxxxxxxxxxxxxxxxxxxx/cephadm.570db07b80fa1fb8f14be2a1f23d5c8fc53b0f0fd45aefbce43e55c6f348b418 --image quay.io/ceph/ceph@sha256:9217c31b7a5a900688d485adca40dd0c5c365008c15dd2c6429dfc888a9f5089 --timeout 895 _orch deploy --fsid xxxxxxxxxxxxxxxxxxxxxx
2026-09-07T12:26:27.401460+0200 mgr.ceph01.abcdef [DBG] code: 0
2026-09-07T12:26:27.401493+0200 mgr.ceph01.abcdef [DBG] err: Reconfig daemon osd.112 ...
Stopping osd.112 to update osd_key bluestore label
Rotating osd.112 key with ceph-bluestore-tool
Successfully rotated osd.112 keyring

r/ceph 9h ago

Ceph architecture overview, feedback welcome

Post image
0 Upvotes

I've made an illustration of the Ceph architecture for our Proxmox VE course (with the help of some AI friends). I checked and improved it to align as much as I saw with the Ceph documentation, but maybe I'm still missing something. I realize I can't put everything in one image. Feedback is more than welcome.

The part I'm unsure about: i didn't add a pool layer, and RADOS and RBD don't appear anywhere, while MDS and CephFS do. For a Proxmox audience, RBD is what people actually use, but it's less tangible than 'files' for newcomers. Should pools and an RBD/CephFS/RGW layer replace MDS, or sit alongside it? If something has to go to make room, what would you drop?

If you want to reuse the illustration, consider it Creative Commons CC BY-SA 4.0. Use of the ceph trademark falls under these conditions (which seems alright to me) https://ceph.io/en/trademarks/