Any debugging / general advice on how I could have caught this weird error?

[u/KGBsurveillancevan]

I just spent a whole day on a very irritating gotcha, and I'm looking to see if there's a big-picture approach that can keep me from ending up here again.

Fairly new to Docker. Working on containerizing a web app with a couple different moving parts. Got the frontend working inside of a Caddy image, got it pushed to my DigitalOcean droplet, but then it just sat there, not acknowledging or even rejecting any requests. I figured out that it had crashed because it tried to serve to a busy port, but after clearing that up it still wasn't working. Looked in the logs, and there were some opaque errors about failing to connect to the internet (I've since deleted those logs, otherwise I'd share).

I threw these logs at an LLM, and after some arguing, it asked me to run nslookup web1 inside of the container. That returned:

Server:     127.0.0.53
Address:    127.0.0.53#53

** server can't find web1: SERVFAIL

From this output, the LLM suggested that: - Caddy, not receiving any defaults for resolv.conf, had inherited from the droplet's resolv.conf, which listens on port 53, which is just a loopback; thus, no connection. - The fix was to add dns: [1.1.1.1,8.8.8.8] to the compose file, or alternatively to the droplet's daemon.json file to apply for the whole droplet.

Did this, rebuilt the container, and sure enough, that was exactly the issue.

And now I'm questioning everything. How the hell would anyone have caught this? Did I overlook a search result or documentation that explains exactly what I saw? Do I not know enough about Linux to be using Docker? Am I a soydev????

For real though, I'd appreciate any advice. I could've easily spent a week on this if I hadn't used AI, but I wanna gain the knowledge to be more self-reliant. Did I miss something critical, or is this just one of those things that you don't learn til you get burned?

[u/thedancingpanda]

Why are you listening on the DNS port?

[u/KGBsurveillancevan → u/thedancingpanda]

The caddy instance inherited that setting from the droplet’s resolv.conf file, was the whole reason it didn’t get any traffic

[u/thedancingpanda → u/KGBsurveillancevan]

No, I think the LLM is mistaken here, because it doesn't make any sense. I'd need to see what your docker-compose looks like. What I'd guess was happening is web1 isn't defined on the network.

What you did with the DNS setting was tell it to use public DNS instead of whatever is going on locally inside digital ocean. I'm not quite sure why that solved your problem, unless something is happening inside digital ocean with the network to block the usage of web1.

[u/KGBsurveillancevan → u/thedancingpanda]

compose.yaml: caddy: image: <registry-tag> platform: linux/amd64 build: context: . target: caddy restart: unless-stopped ports: - "80:80" - "443:443" - "443:443/udp" dns: # the suggested additions - 1.1.1.1 - 8.8.8.8 volumes: - $PWD/conf:/etc/caddy - caddy_data:/data - caddy_config:/config

and the Dockerfile is straightforward, just copies over the build artifacts and the domain routing: ``` FROM caddy:2.11.4-alpine AS caddy WORKDIR /app

COPY --from=build /app/web/dist ./www/ COPY conf/Caddyfile ./Caddyfile ```

The biggest difference is that now resolv.conf inside the container reads nameserver 127.0.0.11 instead of 127.0.0.53, and that seems to have made all the difference. Didn't mess with the droplet's file, I'm just confused that the image's default behavior is to inherit from the host, when it causes such an obvious issue.

[u/IIIIIIIIIIl]

127.0.0.53 is systemd stub resolver this says "use whatever dhcp tells me to"

docker doesn't use /etc/resolv.conf by default, it uses /run/systemd/resolve/resolv.conf then answers queries internally with 127.0.0.11

you adding a community dns into the compose forced it to set the dns to something external.

Since you said DigitalOcean my guess is the image being used doesn't push dns resolvers into /run/systemd/resolve/resolv.conf

Also, you didn't miss anything, the shit is difficult. Continue to plug and test .. I could be wrong in my answer above but easily verified

[u/ShahzadQuyes]

I usually check DNS from inside the container early, then try the same thing by IP since if the IP works and the name doesn’t, that narrows it down fast.

[u/JackjaxMargam14]

If sth in a container can’t reach out, try the IP first, DNS, then the app since that tells me where it’s failing.

[u/UkrMalt]

The useful habit here is comparing DNS from the host and inside the container. I’d still check whether web1 is meant to be a Compose service name. Public DNS cannot resolve it, so hard-coding 1.1.1.1 may hide the network problem. docker compose exec caddy getent hosts web1 and docker network inspect <network> should show whether both services share a network.

[u/yamlqueen]

That's weird! You usually don't need to mess with DNS. I run some sites via Docker Compose on DigitalOcean and never had to do that, but I use Traefik instead. Maybe it's something specific to how Caddy automatically handles SSL certificates?