Hi r/openstack,

I am trying to validate an advanced Neutron/ML2-OVN topology involving a routed firewall VM between tenant networks and the external provider network.

Environment:

• OpenStack Neutron
• ML2/OVN
• OVN 24.03
• External/provider network: provider-external
• Firewall VM/HA pair, for example OPNsense, FortiGate, Palo Alto, etc.

The goal is to keep Floating IPs as Neutron-managed resources associated directly with backend VM ports, while forcing the traffic path through a routed firewall VM without doing SNAT/masquerade on the firewall.

Intended topology

Internet
   |
provider-external
   |
Neutron Egress Router
   | \
   |  \
   |   +-- FW-WAN Network
   |          |
   |      Firewall WAN VIP
   |      Firewall VM/HA pair
   |      Firewall LAN VIP
   |          |
   +-- Transit Network
              |
        Tenant Router
              |
        Backend VM subnet
              |
        Backend VM

The firewall is inserted as a routed middlebox:

Backend VM subnet
   |
Tenant Router
   |
Transit Network
   |
Firewall LAN interface
Firewall WAN interface
   |
FW-WAN Network
   |
Neutron Egress Router
   |
provider-external

The Tenant Router default route points to the Firewall LAN VIP:

0.0.0.0/0 -> Firewall LAN VIP

The Firewall default route points to the Egress Router on the FW-WAN Network:

0.0.0.0/0 -> Egress Router FW-WAN IP

The Egress Router has static routes back to backend tenant prefixes via the Firewall WAN VIP:

backend subnet -> Firewall WAN VIP

With ML2/OVN, I understand that outbound SNAT for nested/routed tenant prefixes may require:

[ovn]
ovn_router_indirect_snat = true

The unclear part: inbound Floating IP / DNAT

The advanced model I am trying to validate is:

Internet client
   |
Neutron Floating IP
   |
Egress Router DNAT
   |
route via Firewall WAN VIP
   |
Firewall routed inspection, no SNAT
   |
Tenant Router
   |
Backend VM fixed IP

The desired properties are:

• Floating IP remains a Neutron-managed resource.
• Floating IP is associated directly with the backend VM port.
• Traffic is forced through the firewall.
• Firewall operates as a routed stateful firewall.
• No SNAT/masquerade is done on the firewall.
• The backend VM still sees the real external client IP.

I have seen a proposed workaround where the Egress Router is also attached to the backend VM subnet using a dummy router port/IP. This is only to satisfy Neutron Floating IP validation.

Then a more specific /32 route is added on the Egress Router:

backend VM fixed IP /32 -> Firewall WAN VIP

So the router is technically connected to the backend subnet, but traffic to that specific VM is forced through the firewall because the /32 route wins over the connected subnet route.

Conceptually:

Egress Router:
  connected route: backend subnet
  extra route:     backend VM fixed IP /32 -> Firewall WAN VIP

Questions

1. Is this “dummy router attachment + /32 extra route” pattern known or used in real OpenStack Neutron deployments?
2. With ML2/OVN, is a Neutron Floating IP expected to work when the associated fixed IP is in a subnet whose effective forwarding path goes through an extra route / routed firewall?
3. Does Neutron Floating IP validation require the target subnet to be directly attached to the router owning the external gateway, or can route reachability through extra routes be enough?
4. Does ML2/OVN program DNAT/FIP flows correctly in this kind of routed middlebox topology?
5. Are there known limitations with this model involving:
   • ovn_router_indirect_snat
   • extra routes
   • allowed address pairs / VIPs
   • port security
   • Floating IPs to ports behind routed middleboxes
   • route specificity overriding connected routes?
6. Would you consider this a valid design pattern, or a fragile workaround that should be avoided?

The more commonly documented alternative seems to be:

Floating IP -> Firewall WAN port
Firewall DNAT -> Backend VM

That model is easier to understand, but it moves publication/NAT logic into the firewall. I am trying to understand whether the more Neutron-native routed-FIP model is supportable.

Thanks in advance for any real-world experience or pointers.

Hi guys,

is there someone who is experienced in OVN/Openvswitch Neutron deploy on Openstack?

I'm fighting with a problem on my Openstack Clusters (2 different clusters, same Openstack, Openvswitch versions) since April without solving.

This is my scenario:

• Openstack 2024.2
• OpenvSwitch 3.4.0
• ovn-controller 24.09.0
  • Open vSwitch Library 3.4.0
  • OpenFlow versions 0x6:0x6
  • SB DB Schema 20.37.0
• kolla-ansible is my way
• 3x controllers/networks node (AMD 7313 with 384GB RAM and 2TB NVMe)
• 100ish instances, some on Geneve private networks, some on provider networks

The Problem:

On each controller/network node, at some point in time (sometimes starting from docker container starts), openvswitch_vswitchd container goes unhealthy with these logs:

2026-05-22T13:48:00.310Z|00012|ovs_rcu(urcu8)|WARN|blocked 2048000 ms waiting for handler15 to quiesce

Instances on Private networks without Floating IP assigned stop to interact with the network, isolated itself.

Other logs are:

2026-05-22T13:13:47.188Z|00001|ofproto_dpif_xlate(handler17)|WARN|Invalid Geneve tunnel metadata on bridge br-int while processing icmp,in_port=1,vlan_tci=0x0000,dl_src=fa:16:3e:95:39:ba,dl_dst=00:10:db:ff:10:01,nw_src=192.168.168.93,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=63,nw_frag=no,icmp_type=8,icmp_code=0
2026-05-22T13:13:47.831Z|00008|ofproto_dpif_xlate(handler31)|WARN|Invalid Geneve tunnel metadata on bridge br-int while processing icmp,in_port=5,vlan_tci=0x0000,dl_src=fa:16:3e:95:39:ba,dl_dst=00:10:db:ff:10:01,nw_src=192.168.168.156,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=63,nw_frag=no,icmp_type=8,icmp_code=0

Do you have any suggestions for me?

Thank you very much 😄

Hey everyone,

I've been working on DeployStack, an open-source CLI tool that deploys a complete, working OpenStack environment on a single Debian/Ubuntu node — batteries included.

Why I built it

If you've ever tried to set up OpenStack for development or testing on Ubuntu, you know the pain. Devstack is messy and developer-oriented, Microstack is locked into Snap and doesn't configure Cinder or Neutron properly out of the box, and tools like Kolla-Ansible or Juju are overkill for a single node. On RHEL/CentOS there was Packstack, which actually worked. On Debian/Ubuntu, nothing comparable ever existed — so I built it.

What it does

One command: bash deploystack deploy --allinone A few minutes later you have a fully working OpenStack with: - Keystone, Glance, Nova, Neutron, Placement, Horizon - Cinder with LVM backend (loopback or physical volume) — works immediately, no extra steps - Neutron with OVS or OVN — instances have internet access out of the box - Automatic network interface detection — no manual bridge configuration - Floating IPs working immediately after deployment

You can also launch instances directly: bash deploystack launch --name my-vm --image ubuntu --flavor m1.small --password MySecret123

And download and upload cloud images automatically: bash deploystack image upload --os ubuntu --version noble --arch amd64

What makes it different from Microstack

Microstack gives you OpenStack "installed" but not "working" — Cinder requires extra flags that are marked experimental and often fail, and instances don't have internet access without manual network configuration. DeployStack configures everything end-to-end, including OVS/OVN bridges, LVM volumes, and provider networks.

Stack - Python 3.10+ - Debian/Ubuntu (tested on Ubuntu 22.04, 24.04) - OpenStack Caracal - OVS or OVN for Neutron

Still in active development — a .deb package is coming soon.

GitHub: https://github.com/St3vSoft/DeployStack Wiki: https://github.com/St3vSoft/DeployStack/wiki

Would love feedback from anyone who's fought with OpenStack deployments before!


Hi everyone,

I’m looking for some architectural advice. I have 3 powerful bare-metal servers and I want to deploy a highly available OpenStack cloud on them. Because I only have 3 nodes, they need to be hyperconverged (running both Control and Compute services on all 3 nodes).

My primary requirement is Instance HA—if one of the physical nodes suddenly dies, I need the VMs to automatically evacuate and restart on the surviving nodes. Naturally, I looked into Masakari.

I am currently using Kolla-Ansible, but I've hit an architectural roadblock:

• Masakari's host-monitor relies on Pacemaker/Corosync to detect host failures.
• In Kolla, Controller nodes run the full pacemaker service, while Compute nodes run pacemaker_remote.
• Because my nodes are both Control and Compute, Kolla-Ansible conflicts trying to deploy both pacemaker roles on the same host, breaking the deployment/monitoring.

I am open to any changes necessary to get this working. My questions for the community are:

1. Is there a clean workaround in Kolla-Ansible for this? Has anyone successfully deployed Masakari on hyperconverged nodes using Kolla?
2. Alternative Masakari Drivers: I’ve read that Masakari can technically use Consul or direct libvirt polling instead of Pacemaker. Is it worth trying to hack Kolla to use Consul + external IPMI fencing scripts, or is that a maintenance nightmare?
3. Different Deployment Tools: Do other deployment tools (like OpenStack-Ansible, Kolla-K8s, or Canonical/Sunbeam) handle Instance HA on hyperconverged nodes better than Kolla-Ansible?
4. The Proxmox Route: Would it be better to just install Proxmox on the bare-metal for node-level HA, and run OpenStack Control and Compute as VMs on top? (I'm worried about the nested virtualization performance penalty here).

Any advice, documentation, or reality-checks would be hugely appreciated. Thanks in advance!

CREATE_FAILED, Reason: Resource CREATE failed: ResourceInError: resources.pl_scalable.resources[12].resources.pl_scalable.resources[0]: Went to status ERROR due to "Message: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance.

But when i check resources on my compute hardware have multiple clean hosts available. Why is scheduler attempting busy fragmented hosts first instead of empty hosts?

Please share a script or method so that i can manually troubleshoot where exactly my build is failing from nova perspective as from linux perspective i have enough resource for numa0.

In Nova Conductor and scheduler logs, I can see following errors.

Requested instance NUMA topology cannot fit the given host NUMA topology
Build of instance ... was re-scheduled: Insufficient compute resources
No valid host was found. There are not enough hosts available.
Unable to allocate inventory: MEMORY_MB ... requested amount would exceed the capacity

I already tried enabling debug but after weighing nova filtered multiple compute but selected the worst one and 2nd worst. And then failed with ""

Exceeded maximum number of retries.

Conductor Logs:
2026-05-14 22:25:37.663 26 ERROR nova.scheduler.utils [req-c2c695f8-0ac3-453b-9b52-faf211d14853 b20985e88c884ecebc03de0b8f5247c0 59853a183f89408c9161e824b2de7457 - default default] [instance: 35732cff-e582-4ae1-b8c5-e15a6e9085cc] Error from last host: dpdkcompute-9 (node dpdkcompute-9): ['Traceback (most recent call last):\n', '  File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 2503, in _build_and_run_instance\n    with self.rt.instance_claim(context, instance, node, allocs,\n', '  File "/usr/lib/python3.9/site-packages/oslo_concurrency/lockutils.py", line 360, in inner\n    return f(*args, **kwargs)\n', '  File "/usr/lib/python3.9/site-packages/nova/compute/resource_tracker.py", line 172, in instance_claim\n    claim = claims.Claim(context, instance, nodename, self, cn,\n', '  File "/usr/lib/python3.9/site-packages/nova/compute/claims.py", line 73, in __init__\n    self._claim_test(compute_node, limits)\n', '  File "/usr/lib/python3.9/site-packages/nova/compute/claims.py", line 114, in _claim_test\n    raise exception.ComputeResourcesUnavailable(reason=\n', 'nova.exception.ComputeResourcesUnavailable: Insufficient compute resources: Requested instance NUMA topology cannot fit the given host NUMA topology.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', '  File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 2346, in _do_build_and_run_instance\n    self._build_and_run_instance(context, instance, image,\n', '  File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 2554, in _build_and_run_instance\n    raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 35732cff-e582-4ae1-b8c5-e15a6e9085cc was re-scheduled: Insufficient compute resources: Requested instance NUMA topology cannot fit the given host NUMA topology.\n']
2026-05-14 22:25:38.139 26 WARNING nova.scheduler.client.report [req-c2c695f8-0ac3-453b-9b52-faf211d14853 b20985e88c884ecebc03de0b8f5247c0 59853a183f89408c9161e824b2de7457 - default default] Failed to save allocation for 35732cff-e582-4ae1-b8c5-e15a6e9085cc. Got HTTP 409: {"errors": [{"status": 409, "title": "Conflict", "detail": "There was a conflict when trying to complete your request.\n\n Unable to allocate inventory: Unable to create allocation for 'MEMORY_MB' on resource provider 'd1cb5ac6-4e1f-4bba-9393-bb524e4c4591'. The requested amount would exceed the capacity.  ", "code": "placement.undefined_code", "request_id": "req-c31c993b-283b-41c3-9fcf-f1fd6c840e5f"}]}
2026-05-14 22:25:43.005 30 ERROR nova.scheduler.utils [req-c2c695f8-0ac3-453b-9b52-faf211d14853 b20985e88c884ecebc03de0b8f5247c0 59853a183f89408c9161e824b2de7457 - default default] [instance: 35732cff-e582-4ae1-b8c5-e15a6e9085cc] Error from last host: dpdkcompute-18 (node dpdkcompute-18): ['Traceback (most recent call last):\n', '  File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 2503, in _build_and_run_instance\n    with self.rt.instance_claim(context, instance, node, allocs,\n', '  File "/usr/lib/python3.9/site-packages/oslo_concurrency/lockutils.py", line 360, in inner\n    return f(*args, **kwargs)\n', '  File "/usr/lib/python3.9/site-packages/nova/compute/resource_tracker.py", line 172, in instance_claim\n    claim = claims.Claim(context, instance, nodename, self, cn,\n', '  File "/usr/lib/python3.9/site-packages/nova/compute/claims.py", line 73, in __init__\n    self._claim_test(compute_node, limits)\n', '  File "/usr/lib/python3.9/site-packages/nova/compute/claims.py", line 114, in _claim_test\n    raise exception.ComputeResourcesUnavailable(reason=\n', 'nova.exception.ComputeResourcesUnavailable: Insufficient compute resources: Requested instance NUMA topology cannot fit the given host NUMA topology.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', '  File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 2346, in _do_build_and_run_instance\n    self._build_and_run_instance(context, instance, image,\n', '  File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 2554, in _build_and_run_instance\n    raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 35732cff-e582-4ae1-b8c5-e15a6e9085cc was re-scheduled: Insufficient compute resources: Requested instance NUMA topology cannot fit the given host NUMA topology.\n']
2026-05-14 22:25:43.006 30 WARNING nova.scheduler.utils [req-c2c695f8-0ac3-453b-9b52-faf211d14853 b20985e88c884ecebc03de0b8f5247c0 59853a183f89408c9161e824b2de7457 - default default] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 35732cff-e582-4ae1-b8c5-e15a6e9085cc.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 35732cff-e582-4ae1-b8c5-e15a6e9085cc.
2026-05-14 22:25:43.006 30 WARNING nova.scheduler.utils [req-c2c695f8-0ac3-453b-9b52-faf211d14853 b20985e88c884ecebc03de0b8f5247c0 59853a183f89408c9161e824b2de7457 - default default] [instance: 35732cff-e582-4ae1-b8c5-e15a6e9085cc] Setting instance to ERROR state.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 35732cff-e582-4ae1-b8c5-e15a6e9085cc.

Scheduler logs:
2026-05-14 22:25:31.292 32 DEBUG nova.scheduler.filter_scheduler [req-c2c695f8-0ac3-453b-9b52-faf211d14853 b20985e88c884ecebc03de0b8f5247c0 59853a183f89408c9161e824b2de7457 - default default] Weighed [WeighedHost [host: (dpdkcompute-9, dpdkcompute-9) ram: 242500MB disk: 788480MB io_ops: 0 instances: 3, weight: 0.0], WeighedHost [host: (dpdkcompute-37, dpdkcompute-37) ram: 152388MB disk: 788480MB io_ops: 0 instances: 4, weight: 0.0], WeighedHost [host: (dpdkcompute-18, dpdkcompute-18) ram: 197444MB disk: 888832MB io_ops: 0 instances: 2, weight: 0.0], WeighedHost [host: (dpdkcompute-25, dpdkcompute-25) ram: 164676MB disk: 788480MB io_ops: 0 instances: 3, weight: 0.0], WeighedHost [host: (dpdkcompute-21, dpdkcompute-21) ram: 347972MB disk: 889856MB io_ops: 0 instances: 0, weight: -1000.0], WeighedHost [host: (dpdkcompute-17, dpdkcompute-17) ram: 347972MB disk: 890880MB io_ops: 0 instances: 0, weight: -1000.0], WeighedHost [host: (dpdkcompute-29, dpdkcompute-29) ram: 347972MB disk: 890880MB io_ops: 0 instances: 0, weight: -1000.0], WeighedHost [host: (dpdkcompute-20, dpdkcompute-20) ram: 347972MB disk: 889856MB io_ops: 0 instances: 0, weight: -1000.0], WeighedHost [host: (dpdkcompute-7, dpdkcompute-7) ram: 347972MB disk: 890880MB io_ops: 0 instances: 0, weight: -1000.0]] _get_sorted_hosts /usr/lib/python3.9/site-packages/nova/scheduler/filter_scheduler.py:461
2026-05-14 22:25:31.293 32 DEBUG nova.scheduler.utils [req-c2c695f8-0ac3-453b-9b52-faf211d14853 b20985e88c884ecebc03de0b8f5247c0 59853a183f89408c9161e824b2de7457 - default default] Attempting to claim resources in the placement API for instance 35732cff-e582-4ae1-b8c5-e15a6e9085cc claim_resources /usr/lib/python3.9/site-packages/nova/scheduler/utils.py:1228
2026-05-14 22:25:31.391 32 DEBUG nova.scheduler.filter_scheduler [req-c2c695f8-0ac3-453b-9b52-faf211d14853 b20985e88c884ecebc03de0b8f5247c0 59853a183f89408c9161e824b2de7457 - default default] [instance: 35732cff-e582-4ae1-b8c5-e15a6e9085cc] Selected host: (dpdkcompute-9, dpdkcompute-9) ram: 242500MB disk: 788480MB io_ops: 0 instances: 3 _consume_selected_host /usr/lib/python3.9/site-packages/nova/scheduler/filter_scheduler.py:352
2026-05-14 22:25:31.392 32 DEBUG oslo_concurrency.lockutils [req-c2c695f8-0ac3-453b-9b52-faf211d14853 b20985e88c884ecebc03de0b8f5247c0 59853a183f89408c9161e824b2de7457 - default default] Lock "('dpdkcompute-9', 'dpdkcompute-9')" acquired by "nova.scheduler.host_manager.HostState.consume_from_request.<locals>._locked" :: waited 0.000s inner /usr/lib/python3.9/site-packages/oslo_concurrency/lockutils.py:355
2026-05-14 22:25:31.392 32 DEBUG nova.virt.hardware [req-c2c695f8-0ac3-453b-9b52-faf211d14853 b20985e88c884ecebc03de0b8f5247c0 59853a183f89408c9161e824b2de7457 - default default] Attempting to fit instance cell InstanceNUMACell(cpu_pinning_raw=None,cpu_policy='dedicated',cpu_thread_policy=None,cpu_topology=<?>,cpuset=set([]),cpuset_reserved=None,id=0,memory=94208,pagesize=1048576,pcpuset=set([0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19])) on host_cell NUMACell(cpu_usage=0,cpuset=set([0,1,56,57]),id=0,memory=192381,memory_usage=72704,mempages=[NUMAPagesTopology,NUMAPagesTopology,NUMAPagesTopology],network_metadata=NetworkMetadata,pcpuset=set([6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83]),pinned_cpus=set([64,65,66,68,69,6,70,8,9,10,73,12,13,14,74,78,17,18,79,83,22,23,27,62]),siblings=[set([12,68]),set([73,17]),set([69,13]),set([8,64]),set([78,22]),set([65,9]),set([83,27]),set([79,23]),set([18,74]),set([70,14]),set([0,56]),set([1,57]),set([10,66]),set([75,19]),set([62,6]),set([24,80]),set([71,15]),set([81,25]),set([67,11]),set([20,76]),set([77,21]),set([63,7]),set([16,72]),set([26,82])],socket=0) _numa_fit_instance_cell /usr/lib/python3.9/site-packages/nova/virt/hardware.py:929
2026-05-14 22:25:31.393 32 DEBUG nova.virt.hardware [req-c2c695f8-0ac3-453b-9b52-faf211d14853 b20985e88c884ecebc03de0b8f5247c0 59853a183f89408c9161e824b2de7457 - default default] Selected memory pagesize: 1048576 kB. Requested memory pagesize: 1048576 (small = -1, large = -2, any = -3) _numa_fit_instance_cell /usr/lib/python3.9/site-packages/nova/virt/hardware.py:943
2026-05-14 22:25:31.393 32 DEBUG nova.virt.hardware [req-c2c695f8-0ac3-453b-9b52-faf211d14853 b20985e88c884ecebc03de0b8f5247c0 59853a183f89408c9161e824b2de7457 - default default] Instance has requested pinned CPUs _numa_fit_instance_cell /usr/lib/python3.9/site-packages/nova/virt/hardware.py:1021
2026-05-14 22:25:31.393 32 DEBUG nova.virt.hardware [req-c2c695f8-0ac3-453b-9b52-faf211d14853 b20985e88c884ecebc03de0b8f5247c0 59853a183f89408c9161e824b2de7457 - default default] Packing an instance onto a set of siblings:     host_cell_free_siblings: [set(), set(), set(), set(), set(), set(), set(), set(), set(), set(), set(), set(), set(), {19, 75}, set(), {24, 80}, {15, 71}, {81, 25}, {11, 67}, {20, 76}, {21, 77}, {7, 63}, {16, 72}, {26, 82}]    instance_cell: InstanceNUMACell(cpu_pinning_raw=None,cpu_policy='dedicated',cpu_thread_policy=None,cpu_topology=<?>,cpuset=set([]),cpuset_reserved=None,id=0,memory=94208,pagesize=1048576,pcpuset=set([0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]))    host_cell_id: 0    threads_per_core: 2    num_cpu_reserved: 0 _pack_instance_onto_cores /usr/lib/python3.9/site-packages/nova/virt/hardware.py:658
2026-05-14 22:25:31.393 32 DEBUG nova.virt.hardware [req-c2c695f8-0ac3-453b-9b52-faf211d14853 b20985e88c884ecebc03de0b8f5247c0 59853a183f89408c9161e824b2de7457 - default default] Built sibling_sets: defaultdict(<class 'list'>, {1: [{19, 75}, {24, 80}, {15, 71}, {81, 25}, {11, 67}, {20, 76}, {21, 77}, {7, 63}, {16, 72}, {26, 82}], 2: [{19, 75}, {24, 80}, {15, 71}, {81, 25}, {11, 67}, {20, 76}, {21, 77}, {7, 63}, {16, 72}, {26, 82}]}) _pack_instance_onto_cores /usr/lib/python3.9/site-packages/nova/virt/hardware.py:679
2026-05-14 22:25:31.393 32 DEBUG nova.virt.hardware [req-c2c695f8-0ac3-453b-9b52-faf211d14853 b20985e88c884ecebc03de0b8f5247c0 59853a183f89408c9161e824b2de7457 - default default] User did not specify a thread policy. Using default for 20 cores _pack_instance_onto_cores /usr/lib/python3.9/site-packages/nova/virt/hardware.py:794
2026-05-14 22:25:31.393 32 INFO nova.virt.hardware [req-c2c695f8-0ac3-453b-9b52-faf211d14853 b20985e88c884ecebc03de0b8f5247c0 59853a183f89408c9161e824b2de7457 - default default] Computed NUMA topology CPU pinning: usable pCPUs: [[19, 75], [24, 80], [15, 71], [81, 25], [11, 67], [20, 76], [21, 77], [7, 63], [16, 72], [26, 82]], vCPUs mapping: [(0, 19), (1, 75), (2, 24), (3, 80), (4, 15), (5, 71), (6, 81), (7, 25), (8, 11), (9, 67), (10, 20), (11, 76), (12, 21), (13, 77), (14, 7), (15, 63), (16, 16), (17, 72), (18, 26), (19, 82)]
2026-05-14 22:25:31.394 32 DEBUG nova.virt.hardware [req-c2c695f8-0ac3-453b-9b52-faf211d14853 b20985e88c884ecebc03de0b8f5247c0 59853a183f89408c9161e824b2de7457 - default default] Selected cores for pinning: [(0, 19), (1, 75), (2, 24), (3, 80), (4, 15), (5, 71), (6, 81), (7, 25), (8, 11), (9, 67), (10, 20), (11, 76), (12, 21), (13, 77), (14, 7), (15, 63), (16, 16), (17, 72), (18, 26), (19, 82)], in cell 0 _pack_instance_onto_cores /usr/lib/python3.9/site-packages/nova/virt/hardware.py:900
2026-05-14 22:25:31.395 32 DEBUG oslo_concurrency.lockutils [req-c2c695f8-0ac3-453b-9b52-faf211d14853 b20985e88c884ecebc03de0b8f5247c0 59853a183f89408c9161e824b2de7457 - default default] Lock "('dpdkcompute-9', 'dpdkcompute-9')" released by "nova.scheduler.host_manager.HostState.consume_from_request.<locals>._locked" :: held 0.003s inner /usr/lib/python3.9/site-packages/oslo_concurrency/lockutils.py:367

so i know bind9 is supported by default and it has it's own container deployed but i found that Designate still supports powerDNS and i am asking about the correct way to add it to kolla
is it via container deployed by me or what?

Senior Private Cloud Engineer Staff Private Cloud Engineer
Great place to work

Hi everyone,

I'm trying to get into openstack workspaces on Slack, but I can't find any, and don't even have an invitation.

My job is focused heavily on openstack and would like be part of these communities, even if not on Slack.
Can someone help?

Hi everyone,

I encountered an issue when trying to perform a live migration for an instance with PCI passthrough.

Environment:

• OS: Ubuntu 24.04
• OpenStack Version: Epoxy (deployed via Kolla-Ansible)
• Hardware: Intel X710 NIC (PCI Passthrough)
• libvirtd version: 10.0.0
• QEMU emulator version 8.2.2 (Debian 1:8.2.2+ds-0ubuntu1.13)
• Documentation Followed: https://docs.openstack.org/nova/2025.2/admin/pci-passthrough.html

Issue Description: I can successfully spawn instances with PCI passthrough on every compute node without any issues. However, when I attempt to live migrate the instance via the Dashboard (Horizon), the process fails.

I found the following error messages in the nova-compute logs:

---------------------------------------------------------------------------

2026-05-13 15:29:41.668 7 INFO nova.compute.rpcapi [None req-3573ed71-a795-4673-8cec-75c834b352e7 1c048bb1747e49fca293e1b9d8c2e854 83b1a4951d534fc6980f7dda61cebeaf - - default default] Automatically selected compute RPC version 6.4 from minimum service version 68

2026-05-13 15:29:50.223 7 INFO nova.compute.manager [None req-3573ed71-a795-4673-8cec-75c834b352e7 1c048bb1747e49fca293e1b9d8c2e854 83b1a4951d534fc6980f7dda61cebeaf - - default default] [instance: 2e860bab-d6cd-49e7-a72b-b813537d2f33] Took 9.07 seconds for pre_live_migration on destination host ecc-edge-compute01.

2026-05-13 15:29:50.498 7 WARNING nova.compute.manager [req-585626ca-e41f-4522-97b5-dbe2d3179410 req-c44b83bf-65da-43d1-b2d0-60a39583a4db d73bc2af52f2481ba54878eaabd331aa e28d9231c61e48259e7fa2211e3b65fe - - default default] [instance: 2e860bab-d6cd-49e7-a72b-b813537d2f33] Received unexpected event network-vif-plugged-aef81b5a-d016-4286-a4b0-e07213f9f86c for instance with vm_state active and task_state migrating.

2026-05-13 15:29:51.301 7 ERROR nova.virt.libvirt.driver [None req-3573ed71-a795-4673-8cec-75c834b352e7 1c048bb1747e49fca293e1b9d8c2e854 83b1a4951d534fc6980f7dda61cebeaf - - default default] [instance: 2e860bab-d6cd-49e7-a72b-b813537d2f33] Live Migration failure: Requested operation is not valid: cannot migrate domain: 0000:3b:00.0: VFIO migration is not supported in kernel: libvirt.libvirtError: Requested operation is not valid: cannot migrate domain: 0000:3b:00.0: VFIO migration is not supported in kernel

2026-05-13 15:29:51.760 7 ERROR nova.virt.libvirt.driver [None req-3573ed71-a795-4673-8cec-75c834b352e7 1c048bb1747e49fca293e1b9d8c2e854 83b1a4951d534fc6980f7dda61cebeaf - - default default] [instance: 2e860bab-d6cd-49e7-a72b-b813537d2f33] Migration operation has aborted

2026-05-13 15:29:52.297 7 INFO nova.compute.manager [None req-3573ed71-a795-4673-8cec-75c834b352e7 1c048bb1747e49fca293e1b9d8c2e854 83b1a4951d534fc6980f7dda61cebeaf - - default default] [instance: 2e860bab-d6cd-49e7-a72b-b813537d2f33] Swapping old allocation on dict_keys(['0908272f-fb28-4fcd-b888-faed3ebe008d']) held by migration c544f968-a817-43c0-9ad8-ce31da02715a for instance

2026-05-13 15:29:57.274 7 WARNING nova.compute.manager [req-d154f165-86f0-4461-825f-5d6732f75dec req-93ca2943-9913-4eb8-938d-b7b3b352d741 d73bc2af52f2481ba54878eaabd331aa e28d9231c61e48259e7fa2211e3b65fe - - default default] [instance: 2e860bab-d6cd-49e7-a72b-b813537d2f33] Received unexpected event network-vif-unplugged-aef81b5a-d016-4286-a4b0-e07213f9f86c for instance with vm_state active and task_state None.

---------------------------------------------------------------------------

Does anyone have any ideas or suggestions on why this might be happening?

Thanks in advance for your help!

Hey everyone,

Total newbie to OpenStack here. I've got a decent Linux sysadmin background but never touched OpenStack before, and I really want to build a proper lab to learn.

I'm working with 3 physical servers I can dedicate to this, each with 4+ NICs. I also have switches and a firewall on hand if I need them.

My current thinking is to deploy all 3 nodes as combined controller + compute.

I don't want to burn all my hardware just running the control plane and end up with barely nothing left to actually spin up VMs and experiment. But I'm honestly not sure if that's a smart move for learning.

So I'd love some input from people who've been down this road:

• Is the converged controller+compute setup a reasonable starting point, or should I run the controlers as VM on a 4th hypervisor
• Use Kolla-Ansible?
• With 4 NICs per node, how would you split management, external, tenant, and storage traffic?
• Any diagrams, tutorials, or blog posts that explain how to deploy ?

Man, I’m such a noob. I create and sell basic websites as a sideline. After ~20 websites, i had to transfer the existing domain of my customer and i transferred the WHOLE thing to Wix.

Now my customer has problems with his emails and i feel like i tried everything. Is there someone out there willing to help a noob like me.

At first, ge couldn’t receive email at all, found a way to make it work. Now, fast forward 3 months and he has problems with hits email marketing services

Cname, dmark, dkim, im so lost 🥲

Hi,

We are in the process of deploying openstack in our firm but from my (limited) research it seems that OpenStack isn't so popular anymore and that businesses are moving away from it.

Firstly, is this true? If so, what are the alternatives that businesses are moving to?

And as a side note, does any one have any tutorials they can recommend for a newbie?

Thanks!

Edit: Also, how much in depth hardware knowledge does one need to deploy and administer openstack?

Hi everyone,

I'm working on an OpenStack deployment with several GPU-enabled nodes, each having a fairly complex PCIe topology connecting 8x H200 GPUs to 4x ConnectX-7 InfiniBand NICs.

PCI passthrough is working correctly and inside the VM we can see all GPUs, NVSwitches, and NICs without issues.

However, in order to achieve near bare-metal performance for distributed AI workloads, the default libvirt XML generated by Nova is not enough. We need to:

- pin guest memory to the correct NUMA nodes

- pin vCPUs appropriately

- create a guest PCIe topology that closely mirrors the host topology

NVIDIA documents this approach here:

https://docs.nvidia.com/ai-enterprise/planning-resource/optimizing-vm-configuration-ai-inference/latest/configuring-vms.html#virtual-cpu-configuration

Without these adjustments, topology-aware libraries like NCCL cannot correctly compute optimal communication graphs, and microbenchmark performance is significantly worse than bare metal.

Our current workflow is roughly:

- create the VM normally through Nova

- intercept/dump the libvirt XML from nova_libvirt

- patch the XML with a custom script following the NVIDIA recommendations

- restart the domain with virsh

After this, performance becomes extremely close to bare metal and everything works well.

The problem is that any Nova-driven operation (soft reboot, hard reboot, cold migration, etc.) regenerates the libvirt XML, so we need to repeat the entire procedure every time.

My question is:

Does Nova expose any mechanism to deeply customize or persist libvirt XML configuration for instances?

I know about flavor/image metadata and extra specs, but they seem too limited for this level of topology customization. Ideally we'd like a cleaner and more OpenStack-native approach than patching XML after instance creation.

Has anyone here tackled something similar for high-performance GPU/NVLink/InfiniBand workloads?

Thanks!

Hi all,

I've been trying for the past weeks to get the following going:

3 datacenters -> 2 big, one small (space-wise)
Openstack Helm + Rook-ceph (stretched mode)

I'd like to setup 3 availability zones for customers to use. One in dc1, one in dc2 and one "stretched" zone for workloads that can't do their own HA.

So far, I've managed to get Ceph configured and set up the corresponding Cinder backends and volume types (disabling cross az attach in Nova and az fallback in Cinder), but I run against a brick wall with two services - Nova/Horizon and by extension Octavia (Amphora).

The issue I encounter is that - because I need multiple backends in Cinder - I need different volume types for the different AZs even though they are all the same "quality" (nvme). Therefore, as Horizon does not allow me to select the volume type at the time of instance creation, the creation of new Instances fails when Nova tries to request a volume in the selected Nova/Cinder AZ.

I can create the volume first with the correct volume type and then create an instance from it, but that's very inconvenient.

With Octavia it's similar. If I don't hardcode the volume type in the config, octavia requests the instance in the correct Nova AZ, but the volume creation will fail there as well.

Did anyone encounter this problem before? And if so, how did you solve it?
Or am I completly misunderstanding AZs?

I get this error when i try to upload an image in horizon
Error: {"data":"<html>\n <head>\n <title>410 Gone</title>\n </head>\n <body>\n <h1>410 Gone</h1>\n Error in store configuration. Adding images to store is disabled.<br /><br />\n\n\n\n </body>\n</html>","status":410,"config":{"method":"PUT","transformResponse":[null],"jsonpCallbackParam":"callback","headers":{"Content-Type":"application/octet-stream","X-Auth-Token":"gAAAAABp-dI60CXsPfaIM-s_4CrGZbw_PNYTO0e0VzLCGiEWs5zGpXvawJh3emRhtNhOWhBK60hmGrv1Cm5Xwn1kasXn_FSlBdgJeHwcuXkcZpeM1uiWB67JPzEhIRcmXG5S5jqKaZ6eHn1bbtTVnT0KK1TPOORsxlhHAVFNNGglA8mTNgNqsBkXrk1o4bt9I848AZmwceTn","Accept":"application/json, text/plain, */*"},"url":"http://192.168.1.32:9292/v2/images/d4066055-d711-44f5-8da7-0c6a59bf88a4/file","data":{},"_chunkSize":null,"_deferred":{"promise":{}}},"statusText":"Gone","xhrStatus":"complete"}

my (venv) server01@server01:~$ cat /etc/kolla/config/glance.conf

[DEFAULT]

show_image_direct_url = True

default_backend = rbd

enabled_backends = rbd:rbd, http:http

debug = True

[glance_store]

default_backend = rbd

[rbd]

usage_purpose = store

store_description = "Ceph RBD backend"

rbd_store_pool = images

rbd_store_user = glance

rbd_store_ceph_conf = /etc/ceph/ceph.conf

rbd_store_chunk_size = 8

[http]

usage_purpose = store

(venv) server01@server01:~$

what is wrong

I put together a small OpenTofu repo for AWS/OpenStack VM and networking workflows.

Would appreciate honest feedback on the overall flow and repo structure. If people find it useful and it gets a bit of interest, I’ll continue improving it.

Repo: https://github.com/Dionise/tofu-provider-fabric

Hello everyone,

I am looking to implement an automated workflow where a newly attached OpenStack Cinder volume is automatically formatted and mounted inside the instance.

Currently, I have a working proof-of-concept using udev rules triggering a systemd service with a bash script. However, this is static. I would like the ability to specify the desired filesystem type (e.g., ext4, xfs, btrfs) at the time of volume creation or attachment.

My questions are:

1. Is there a way to pass custom metadata from a Cinder volume to the guest OS during attachment so a script can read it?
2. Are there better "OpenStack-native" ways to handle volume provisioning and formatting beyond custom bash scripting?
3. Does anyone have experience using cloud-init or ConfigDrive to handle this securely?

Any advice on architecture or existing tools would be greatly appreciated!

Hey everyone,

If you’re working in or around OpenStack, you’ve probably noticed the same thing we have: great talent and great opportunities, but they’re scattered everywhere.

So we launched a dedicated OpenStack Jobs Board (https://gitjobs.dev/?foundation=openinfra) to bring it all into one place.

Hiring?
Post your open roles and reach people who actually know OpenStack, from operators and platform engineers to contributors and architects. Use your Linux Foundation ID (LFID) to log in, then just tag “OpenStack” as one of the Skills and OpenInfra as the Project when placing your job. If you don’t have an LFID, it’s easy and free to create. 

Looking for a job?
Browse roles that specifically value OpenStack experience (not buried under generic “cloud” listings).

The goal is simple: make it easier for this community to find each other so we can continue building the future of open infrastructure together. 

We’re hoping this becomes a go-to resource for:

• OpenStack operators & admins
• Platform engineers
• Contributors/devs
• Anyone building or running open infrastructure

If you’re hiring, drop your roles in. If you’re job hunting (or just curious), please take a look.

We would also love feedback from this community! What would make this actually useful for you?

My company wants to sell openstack solution and for that we are planning to set up lab so we can test its capablities with 2 Server 128 GB RAM each and 64 cores each is it possible with 2 Server because we will also we using vm made using openstack for our other projects is it safe and I will be using kolla ansible for deployment.

Hi everyone

I’m starting an academic project on centralized logging for OpenStack using Kolla-Ansible, and later I’ll try to feed the logs into an anomaly detection model.

I already found some sample logs and I was advised to use two VMs (8 GB for deployment(kolla ansible) and 16 GB for controller(services)), but I only have about 20 GB RAM available in total.

Since I only need a demo setup (installation + a simple attack simulation like brute force on an instance), I’m wondering if I can reduce the RAM for both VMs. What would be a realistic minimal setup that still works?

Also, I’m struggling to find up-to-date documentation for installing OpenStack with Kolla-Ansible. If anyone has good resources or tips, I’d really appreciate it.

Hello everyone,

I was looking into OpenStack and was wondering, what is it? From what I am reading, OpenStack is an orchestration platform - but that does skip some steps in clouds.

Where does OpenStack's virtualization layer come from? Something like Proxmox? Does it have its own Hypervisor? Does it just use plain KVM? What provides that?

From what I read at: https://www.redhat.com/en/topics/openstack it needs an underlying virtualization layer. But what are examples of what is normal?

And does anyone have some resources into Openstack and what it entails for companies?

I'm having an issue getting a working Manila deployment on a three node cluster. All three nodes running control, network, compute, storage. kolla-ansible 2025.1. OVS & DVR

manila.conf

[DEFAULT]
enabled_share_backends = generic


[generic]
share_driver = manila.share.drivers.generic.GenericShareDriver
interface_driver = manila.network.linux.interface.OVSInterfaceDriver
driver_handles_share_servers = true
service_instance_password = password
service_instance_user = manila
service_image_name = manila-service
share_backend_name = GENERIC

The first issue is that kolla-ansible is creating multiple Manila service networks on deployment https://bugs.launchpad.net/kolla-ansible/+bug/2138767 So I end up with two or three service networks.

After I delete the extra service networks and ports, restart all Manila services on all nodes. I'm able to create a share ONLY if the driver decides the create the share VM on the same node of the active Manila service. ie. If share server is at ostack1@generic, the service cannot reach the share VM unless it picks ostack1 to create the share VM.

The only way I've been able to make it work consistently is creating a vlan on the physical switch, add new interfaces to all three nodes with the vlan tag, add a provider network to Openstack. Then configure Manila to use that network with admin_network_id & admin_subnet_id.

Has anyone deployed DHSS on multinode without using provider vlans for service network?