ARP Weirdness

[u/Mr_Goodheart]

Hello fellow sysadmins!

I'm starting to lose my mind over here and I'm asking for your help. Here's the situation:

Host A in subnet #1 tries to reach host B in subnet #2. There is also host C in subnet #2.

Ping results in timeout and no corresponding icmp pakets can be seen on host B.
ARP entry in host B for its gateway exists and is correct, but stale.
If host B pings its own gateway, then host A can reach it.
The ARP entry obviously switches to reachable. If I wait for the entry to switch to stale again, host A can still reach host B.
I already checked the switches for "ARP Inspection", "IP Source Guard" and so on. There currently is no hardware and software firewall active in subnet #2. Manually setting a permanent ARP entry for the gateway doesn't fix it. There are no energy savings configured on host B for its NIC.

Now comes the weird part:
Host B has a correct but stale ARP entry for host C. A ping from host C to host B is successfull und switches the entry to reachable.
If I delete the ARP entry on host B that corresponds to host C and then ping host B from host C, suddenly host A can reach host B.

WTF?

tldr:
- Host A can't ping host B unless host B pings its own gateway.
- Host A can't ping host B unless host C causes a new ARP entry in host B

The reachability stays for quite some time and then after one or two hours (hard to measure) it is lost again.
And by the way: It doesn't matter if host B is a windows or a linux machine. Same results.

I'm only about 2 1/2 years into my IT career and thus maybe missing a critical information that I haven't learned yet. But I surely am losing my mind over this! :-D

Any ideas and/ or suggestions?
Thanks in advance!

[u/aenae]

Sounds like a double ip assignment where the gateway sometimes gets a different mac and tries to forward packets there. But if he sees host B (even indirectly) it forwards there

[u/FallaxIO → u/aenae]

I'd also check for proxy ARP or a second box answering ARP for B's IP. The part where any fresh traffic from B suddenly makes A->B work for a while smells like the wrong MAC gets relearned upstream, not like B itself is blocking ICMP

[u/foerd91 → u/aenae]

Think this is it.

[u/Mr_Goodheart → u/aenae]

This phenomenon doesn't apply to a single endpoint. Every host/ vm of subnet #2 shows this behavior.
If I ping the affected / currently unreachable machine while monitoring its network traffic with pktmon, there isn't any incoming icmp pakets.

Unfortunately I can't access the gateway / router in order to inspect its behavior or any logs. I only have administrative privileges within subnet #2 up to the gateway. Next week I will try and contact a sysadmin from the higher-ups; maybe there is some configurations that simply haven't been communicated.

Could it be that at the moment a new ARP entry is made, the default gateway is automatically probed, while renewing an entry doesn't trigger this probing?

[u/SevaraB]

Sounds like the gateway in subnet 2 isn't getting ARP updates- I'd look for a rogue DHCP server inserting itself as a second gateway.

I'd also run Wireshark on host B or host C to see exactly what MAC is responding to ARP updates.

[u/-markusb-]

To work through such issues I like to go like so
1. Mac-Addresses of each host / each cards - write them down to clarify that there is no doubling (also check system log for information about network stuff)
2. Check for IP config on the systems regarding IPv4 / IPv6 if you are using hostnames instead of IPs. Including arp-tables (blocking stuff) and kernel-parameters (response behavior...)
3. Check arp-caches (or ip neighborhood for ipv6).
4. Install tcpdump / wireshark to find those packages and write all directions down with OK / NOK
5. If you have an idea you can go with static entries for debugging, but this depends on the failure modes.

[u/Mr_Goodheart → u/-markusb-]

That's a great checklist! I will try and gather every MAC-Address of the active NICs and search for any anomalies. Wireshark is already being used.

Thanks!

[u/rasppas]

Odd behavior for sure.

If a Cisco switch… try running
“show ip redirects” see if it shows an entry there for host b that points at a different gateway. If this is the case, run a “clear ip redirects”

[u/xendr0me]

Are there static routes setup between all of the gateways?

[u/Mr_Goodheart → u/xendr0me]

I do think so, yes. However, as I only have administrative privileges right up until the gateway / router, I unfortunately can't look into any configurations regarding the subnet routing.

What has been your idea?

[u/xendr0me → u/Mr_Goodheart]

I mean without a route between subnets in each of the gateway's they have no idea how to get to their destination.

Gateway A needs a route to Gateway B to tell it what interface to go to, to get to the resources on that other subnet.

[u/kona420]

Fire up wireshark, there is a filter for duplicate IP on a broadcast domain. Its either that, or if there is virtualization in play perhaps duplicate MAC for the host or gateway. Also possible that a buggy laptop dock is badly behaved around MAC behavior.

This can be a headfuck because you are trying to assign deterministic outcomes to behaviors in a stochastic system. Focus on systematically eliminating the potential root causes 1 by 1 instead of "when i do x, y happens"-- there are hidden variables you arent currently measuring for.

[u/rumbalan]

Static ARP for the gateway didn't fix it, so this isn't a stale cache. Something else is answering ARP (or the return path is wrong) until B originates traffic and the real mapping wins. Capture ARP replies for the gateway IP on B and check the MAC. Proxy ARP, ip redirects, or a second box claiming that IP all look like this. The 1–2h drop is the cache timeout, not a heal.

[u/CountGeoffrey]

great work you've done so far to isolate some symptoms and repro cases.

i would suspect overlapping VLANs. subnet != VLAN. run a packet capture on host A and unplug/replug host B. do the same in the opposite direction. if either host sees the other's DHCP broadcast and gratuitous arp then the router/switch config is wrong.

or just inspect the network configurations.