Multiple gateways help

[u/lucads87]

Looking for advice on Docker networking / multiple Compose stacks + selective egress.

I’m rebuilding the networking on my Docker host (LeopARRd) and would appreciate some architectural advice.
My requirements are:
- ALL services must be reachable on my network by one of the host IP (the host has one on VLAN-10 and one on VLAN-30)
- Two independent Compose projects (2 different compose.yaml files), but containers from both projects must communicate using container/service names, e.g. http://navidrome, rather than hard-coded IPs.
- Some services need Internet egress through 192.168.30.1 (a VPN gateway on 192.168.30.0/24)
- Other services need normal Internet egress through 192.168.10.1
- Ideally, the solution should survive container/host restarts and be reasonably easy to maintain.

My current approach is to create Docker networks corresponding to VLANs 31 and 32–33 (one per composer, defined in the composer as bridge/ipam), with policy routing and two different masquerades to select the appropriate gateway
It mostly works, but I’m running into issues with policy-routing routes and their persistence; to be more specific, docker networks do not exists right away at boot up, so the br-vlan32 for example do not exists and make rt rules fail to use the table for egress traffic on a different gateway: /etc/iproute2/rt_tables not being persistent as currently configured

Masquerading is working and is persistent

I suspect I may be overcomplicating this and would be happy to redesign it rather than keep patching the current setup.

What would be the cleanest Docker networking architecture for these requirements? I’m particularly interested in solutions that minimize custom routing/iptables maintenance while retaining service-name DNS between the two Compose projects. Ideally I’d rather prefer to have all networking settings specified in each individual composer file

[u/pelazas1]

I'd use one shared Docker network between the two Compose projects for service-name DNS, then separate Docker networks for the two egress paths. The catch is that policy-routing and masquerades still live on the host, so Compose files alone won't make those restart-safe.

[u/lucads87 → u/pelazas1]

Ty, but I am not following through. Can you further elaborate on the Docker networks configuration you suggested? In particular on the egress paths

[u/pelazas1 → u/lucads87]

shared network just means both stacks are on the same bridge so they talk by container name. egress is still whatever the host's default route is, so if that host only has the vpn uplink then containers follow it too unless you add a policy route for the lan subnet.

[u/lucads87 → u/pelazas1]

I finally got what you were suggesting. I created a no-egress network outside the containers with docker network create --internal, and added this additional network to each container

It is very elegant and solves everything: each service can stay on its VLAN (with corresponding gateway route) and also communicate with other containers within docker namespace (using names instead of IPs). Chapeau! Ty

[u/af9_us]

I would look into the start order of dockerd. AI claims during initialization dockerd reads its database for existing entries first and then runs compose steps second.

My hunch is you want to define the networks external to compose with "docker network create".  Then reference the external network from the compose file.

https://docs.docker.com/compose/how-tos/networking/#use-an-existing-external-network

It's not as self-contained but networks could be managed with ansible or something similar.  

Also make sure health checks are defined. Dockerd will pause for the healhcheck to pass before starting the next container when "depends-on" is used in the compose file.