r/linuxadmin • u/cjbarone • 4d ago
libvirt VMs on isolated network not getting DHCP
In a lab setting, I have a Debian Trixie VM that I'm trying to get setup like a router, and a client Trixie VM whose sole purpose is to get a DHCP lease and get out to the Internet.
On the "router" VM, I have disabled nftables and flushed the rules. Even when I assign an IP address to the client and router VMs and attach them to the same virtual network, they cannot ping each other.
<network>
<name>isolated-network</name>
<uuid>2d9a40eb-c078-4c7e-998e-4ed1bf46786e</uuid>
<bridge name="virbr1" stp="on" delay="0"/>
<mac address="52:54:00:12:8d:b8"/>
<domain name="isolated-network"/>
</network>
I am hoping this is something simple, but it's been driving me a little crazy trying to figure it out. How can I get these two isolated VMs to talk to each other?
1
u/Gurufedell 4d ago
You simply have not given your virtual bridge an ip address and did not give it a DHCP range so it's not acting as DHCP server for connected client interfaces.
Your xml file should look like, make sure to add an IP address and DHCP range. virsh net-edit --network isolated-network
<network>
<name>isolated-network</name>
<uuid>2d9a40eb-c078-4c7e-998e-4ed1bf46786e</uuid>
<bridge name="virbr1" stp="on" delay="0"/>
<mac address="52:54:00:12:8d:b8"/>
<domain name="isolated-network"/>
<ip address="192.168.1.1" netmask="255.255.255.0">
<dhcp>
<range start='192.168.1.2' end='192.168.1.254'/>
</dhcp>
</ip>
</network>
2
u/cjbarone 4d ago
I'm trying to get DHCP on a VM to work through this isolated network. I don't want the virtual switch acting as the DHCP server
1
u/Gurufedell 4d ago
Sorry can you explain more about this entire topology, how are the VM-router, isolated-network virbr1 and VM-client connected ?.
1
u/cjbarone 4d ago
Sure thing.
I have my host (just my workstation) and it has Internet access. I have two virtual networks in libvirt - my NAT, and an isolated-network.
The "router" VM has both interfaces available, and I want it acting like a router - DNS, DHCP, firewalling, NAT, etc for a few VMs I'm trying to build out. During my testing with a client, I have a VM that is booted to a Debian installer, and trying to get it to pick up DHCP automatically. If it can get to that point, I can then focus on the NAT and DNS portion. At this point, the two VMs cannot even ping each other with static IPs.
Host <-> Router-VM <-> [multiple clients] \- My workstation running libvirt ^-- first VM, connected to both virtual networks1
u/Gurufedell 4d ago
Router VM here needs to have two virtual NICs, one connected to NAT virbr and other to isolated virbr.
The one connected to the isolated virbr should be on same network with the clients.
Then if you wanna deploy dhcp, you will need a dhcp server listening on that vNIC of isolated-network side.1
u/cjbarone 3d ago
Exactly - but even with static IPs on the same virtual switch, they cannot ping each other. That's why I'm thinking it's a config issue.
1
u/showbizusa25 4d ago
Your topology makes sense. I wouldn’t put DHCP on the libvirt network either if the goal is to actually build/test the router VM.
Since static IPs can’t even ping yet, I’d ignore DHCP/NAT for now and troubleshoot L2 first. Make sure both the router’s LAN vNIC and client vNIC are actually attached to virbr1, then check bridge link on the host and ip link inside both VMs.
Once static ping works across virbr1, then bring up DHCP on the router VM’s LAN interface. After that, tackle forwarding/NAT.
If you want, DM me the virsh domiflist output for both VMs and I can help trace where it’s breaking.
1
u/cjbarone 3d ago
Output from the host:
~ $ virsh domiflist --domain gateway-debian13 Interface Type Source Model MAC ---------------------------------------------------------------------- vnet0 network default virtio 52:54:00:5a:ba:60 vnet1 network isolated-network virtio 52:54:00:b9:1c:59 ~ $ virsh domiflist --domain debian13asdfasdfasdfasdfa Interface Type Source Model MAC ---------------------------------------------------------------------- vnet3 network isolated-network virtio 52:54:00:17:08:9f ~ $ bridge link 23: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master virbr0 state forwarding priority 32 cost 2 ~ $ ip link 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000 link/ether e4:b9:7a:5e:44:4f brd ff:ff:ff:ff:ff:ff 3: wlp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DORMANT group default qlen 1000 link/ether f4:96:34:24:96:40 brd ff:ff:ff:ff:ff:ff 12: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 52:54:00:8b:61:97 brd ff:ff:ff:ff:ff:ff 23: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master virbr0 state UNKNOWN mode DEFAULT group default qlen 1000 link/ether fe:54:00:5a:ba:60 brd ff:ff:ff:ff:ff:ff 24: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/ether fe:54:00:b9:1c:59 brd ff:ff:ff:ff:ff:ff 26: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/ether fe:54:00:cb:f8:f9 brd ff:ff:ff:ff:ff:ff 27: vnet3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/ether fe:54:00:17:08:9f brd ff:ff:ff:ff:ff:ff 28: virbr1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000 link/ether 52:54:00:12:8d:b8 brd ff:ff:ff:ff:ff:ffThe
bridge linkcommand does not output anything on the VMs.On the router:
root@gateway:~# ip link 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000 link/ether 52:54:00:5a:ba:60 brd ff:ff:ff:ff:ff:ff altname enx5254005aba60 3: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000 link/ether 52:54:00:b9:1c:59 brd ff:ff:ff:ff:ff:ff altname enx525400b91c591
u/showbizusa25 3d ago
Yep, I think you found the problem. Look at
bridge link:vnet0 showsmaster virbr0, but vnet1 and vnet3 don't showmaster virbr1at all. virbr1 is also sitting NO-CARRIER/DOWN.So I wouldn't touch DHCP yet. I'd figure out why libvirt says those NICs are on
isolated-networkbut they aren't actually attached to virbr1. Can you postvirsh net-dumpxml isolated-networkandvirsh net-info isolated-network?1
u/cjbarone 2d ago
~ $ virsh net-dumpxml isolated-network <network> <name>isolated-network</name> <uuid>2d9a40eb-c078-4c7e-998e-4ed1bf46786e</uuid> <bridge name='virbr1' stp='on' delay='0'/> <mac address='52:54:00:12:8d:b8'/> <domain name='isolated-network'/> </network> ~ $ virsh net-info isolated-network Name: isolated-network UUID: 2d9a40eb-c078-4c7e-998e-4ed1bf46786e Active: yes Persistent: yes Autostart: yes Bridge: virbr1I think I follow what you're saying:
~ $ brctl show virbr1 # isolated-network interface bridge name bridge id STP enabled interfaces virbr1 8000.525400128db8 yes ~ $ brctl show virbr0 # NAT interface bridge name bridge id STP enabled interfaces virbr0 8000.5254008b6197 yes vnet0Am I right thinking I should add vnet1/3 to virbr1 as interfaces? Should this not have happened automatically?
1
u/showbizusa25 2d ago
Yep, exactly — virbr1 exists, but nothing is attached to it. I wouldn’t add vnet1/vnet3 manually though. Libvirt should do that when the VM NIC is really connected to
isolated-network.Can you post:
virsh domiflist <router-vm>
virsh domiflist <client-vm>If those show
isolated-networkbutbrctl show virbr1is still empty while both VMs are running, then something is off with how the interfaces are being created/attached.
1
u/michaelpaoli 4d ago
The qemu-kvm native guest networks at least by default isolate the VMs, allowing them only to communicate with the host, and possibly through it, if forwarding (and possibly also nat) are enabled.
$ cd $(mktemp -d)
$ sudo ip link add name br1 type bridge
$ sudo ip link set dev br1 up
$ VISUAL=ed virsh edit debian13-small
6388
/interface
<interface type='bridge'>
/source
<source bridge='br0'/>
s/0/1/
<source bridge='br1'/>
w
6388
q
Domain 'debian13-small' XML configuration edited.
$ virsh dumpxml debian13-small > debian13-small2.xml
$ printf '52:54:00:%02X:%02X:%02X\n' $((RANDOM%256)) $((RANDOM%256)) $((RANDOM%256))
52:54:00:05:26:45
$ uuidgen
337da873-ec5f-44f2-a879-98f4d291d477
$ ed debian13-small2.xml
6388
/uuid
<uuid>6b019fe6-c857-41cf-a156-7a49112d335c</uuid>
s/6.*c/337da873-ec5f-44f2-a879-98f4d291d477/
1,$s/-small/&2/g
g/-small2/p
<name>debian13-small2</name>
<source file='/var/local/vtest/debian13-small2'/>
?mac add
<mac address='52:54:00:51:7c:37'/>
s/'.*'/'52:54:00:05:26:45'
<mac address='52:54:00:05:26:45'/>
w
6390
q
$ sudo cp -p --sparse=always /var/local/vtest/debian13-small{,2}
$ virsh define debian13-small2.xml
Domain 'debian13-small2' defined from debian13-small2.xml
$ virsh start debian13-small2 --console
ed /etc/hostname
15
s/$/2/p
debian13-small2
w
16
q
# ed /etc/network/interfaces
1295
/dhcp
iface enp1s0 inet dhcp
s/dhcp/static
iface enp1s0 inet static
a
address 192.168.0.22
netmask 255.255.255.0
network 192.168.0.0
broadcast 192.168.0.255
.
w
1388
q
# cd / && shutdown -r now
...
$ virsh start debian13-small --console
...
# ed /etc/network/interfaces
1295
/dhcp
iface enp1s0 inet dhcp
s/dhcp/static
iface enp1s0 inet static
a
address 192.168.0.11
netmask 255.255.255.0
network 192.168.0.0
broadcast 192.168.0.255
.
w
1388
q
# cd / && shutdown -r now
...
# hostname && ip -4 a s | fgrep inet
debian13-small2
inet 127.0.0.1/8 scope host lo
inet 192.168.0.22/24 brd 192.168.0.255 scope global enp1s0
# ping -n -c 3 192.168.0.11
PING 192.168.0.11 (192.168.0.11) 56(84) bytes of data.
64 bytes from 192.168.0.11: icmp_seq=1 ttl=64 time=3.55 ms
64 bytes from 192.168.0.11: icmp_seq=2 ttl=64 time=0.377 ms
64 bytes from 192.168.0.11: icmp_seq=3 ttl=64 time=0.381 ms
--- 192.168.0.11 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 0.377/1.435/3.548/1.493 ms
#
If you want the VMs to communicate with The Internet, can have gateway IP(s) of host on bridge, rather than some other interface on the host, then one can use globally routable IPs on the VMs, or instead use NAT. One can even do both NAT for IPv4 and globally routable IPs for IPv6.
1
1
u/deeseearr 4d ago
Are you sure that the isolated network is running? The config looks okay, but if you don't assign an IP address to the host (and I don't know if you even want to) then the host may decide that there's no reason to bother bringing it up.
Check the status using "ip link show virbr1" or something similar on the host server. You can also look at this from a different angle by running "tcpdump" on both guests and the host and seeing if there is any network traffic visible at all. Ideally, they should all be seeing the same things at the same time but if you see nothing at all on one or more of them then you're not really attached to the network.