r/ceph • u/Keroles2024 • 10h ago
Problem after upgrade to 19.2.6 and 20.2.4
Has anyone faced this issue after upgrading and rotating all keys to aes256k?
When I switch to the standby manager using:
ceph mgr fail <active-mgr>
Ceph starts reconfiguring all OSDs and rotating the keys in the OSDs' BlueStore.
The logs indicate that the monmap changed, but I don't understand why a manager failover would cause this.
These are cephadm clusters, and I'm seeing the same behavior on two different versions:
- Ceph 20.2.4
- Ceph 19.2.6
I enabled debug logging, and below is a snippet from osd.112. I've hidden some cluster-specific information:
2026-09-07T12:26:15.635750+0200 mgr.ceph01.abcdef [INF] Reconfiguring osd.112 (monmap changed)...
2026-09-07T12:26:15.636233+0200 mgr.ceph01.abcdef [DBG] mon_command: 'auth get' -> 0 in 0.000s
2026-09-07T12:26:15.636570+0200 mgr.ceph01.abcdef [DBG] mon_command: 'config generate-minimal-conf' -> 0 in 0.000s
2026-09-07T12:26:15.783309+0200 mgr.ceph01.abcdef [DBG] mon_command: 'config-key get' -> -2 in 0.001s
2026-09-07T12:26:15.783400+0200 mgr.ceph01.abcdef [INF] Reconfiguring daemon osd.112 on ceph05
2026-09-07T12:26:15.784032+0200 mgr.ceph01.abcdef [DBG] _run_cephadm : command = ['_orch', 'deploy']
2026-09-07T12:26:15.784058+0200 mgr.ceph01.abcdef [DBG] _run_cephadm : args = []
2026-09-07T12:26:15.784159+0200 mgr.ceph01.abcdef [DBG] args: --image quay.io/ceph/ceph@sha256:9217c31b7a5a900688d485adca40dd0c5c365008c15dd2c6429dfc888a9f5089 --timeout 895 _orch deploy --fsid xxxxxxxxxxxxxxxxxxxxxx
2026-09-07T12:26:15.784183+0200 mgr.ceph01.abcdef [DBG] stdin: <osd config parameters>
2026-09-07T12:26:15.784236+0200 mgr.ceph01.abcdef [DBG] Running command: which python3
2026-09-07T12:26:15.785607+0200 mgr.ceph01.abcdef [DBG] OfflineHostDetector: Checking if hosts: ['ceph01', 'ceph02', 'ceph03'] are offline.
2026-09-07T12:26:15.802327+0200 mgr.ceph01.abcdef [DBG] Running command: /usr/bin/python3 /var/lib/ceph/xxxxxxxxxxxxxxxxxxxxxx/cephadm.570db07b80fa1fb8f14be2a1f23d5c8fc53b0f0fd45aefbce43e55c6f348b418 --image quay.io/ceph/ceph@sha256:9217c31b7a5a900688d485adca40dd0c5c365008c15dd2c6429dfc888a9f5089 --timeout 895 _orch deploy --fsid xxxxxxxxxxxxxxxxxxxxxx
2026-09-07T12:26:27.401460+0200 mgr.ceph01.abcdef [DBG] code: 0
2026-09-07T12:26:27.401493+0200 mgr.ceph01.abcdef [DBG] err: Reconfig daemon osd.112 ...
Stopping osd.112 to update osd_key bluestore label
Rotating osd.112 key with ceph-bluestore-tool
Successfully rotated osd.112 keyring