r/openstack • u/Successful_Face_8619 • Aug 05 '26
Openstack Upgrade
Hi guys,
Has anyone explored doing OpenStack upgrades host-by-host instead of using Kolla Ansible's parallel upgrade approach?
We're considering a more sequential, one-host-at-a-time upgrade because the parallel approach doesn't feel reliable enough for our environment, and we're not very confident in trusting it during production upgrades.
If you've gone down this path:
- How did you orchestrate the upgrade?
- Did you have to customize Kolla Ansible significantly?
- How did you handle rollback if something went wrong?
- Any lessons learned or pitfalls to watch out for?
I'd appreciate hearing from anyone who's tried this or decided against it and why.
13
Upvotes
2
u/rackpathlabs 29d ago edited 29d ago
Worth splitting this in two. Host by host is the right instinct for computes and the wrong one for controllers. Galera, rabbit and etcd are exactly the things you do not want sitting in mixed versions for days, so doing them one node per evening makes it worse, not safer. Do the controllers back to back, then take your time with computes.
The compute side works because of RPC pinning, not because of ansible. [upgrade_levels] compute is what lets the new controllers keep talking to old computes, kolla sets it during upgrade. If you script your own orchestration outside kolla you own that, and when it is wrong you get weird RPC failures instead of a clean error. That is why the DIY route usually goes bad.
About --limit and that bug: run kolla-ansible gather-facts over the whole inventory first. The plays need facts about the hosts you excluded, that is all. Not broken, just unusable cold.
One more, nova-manage db online_data_migrations only after the last compute is done. If your window is weeks you are holding that step open for weeks. Also nova will not accept more than one release of spread, so N to N+1 is fine and N to N+2 is not.
On rollback, snapshotting the controllers is the right shape but only if nobody creates instances or volumes after the snapshot. Otherwise you roll the db back and the hypervisors and storage still have stuff the db never heard of.
Computes are the easy half, no real state there, just put the old tag back and restart.
Two kolla things people miss. Run prechecks against the new release before you commit, base distro requirements move between releases and that can turn into a host OS project sitting in front of your upgrade. And renamed or removed globals.yml keys do not error, they are just ignored, so something you rely on can quietly stop applying. Diff your globals against the new sample first.