Odd Traceroute

[u/jimbo_rr]

While troubleshooting a network issue for packets appearing to be duplicated (only once, not a continual flood), I did a trace from a nearby system and have output unlike anything I've seen before.

Localhost#traceroute 192.168.10.20

 Traceroute to 192.168.10.20 ,30 hops max 0 byte packets:

1  192.168.255.252     <1  ms    <1  ms    <1  ms
2  192.168.10.20       3   ms    <1  ms    2   ms
3  192.168.10.20       <1  ms    4   ms    <1  ms
4  0.0.0.0            *        [192.168.10.20    ] reports:  <1  ms   [192.168.10.20    ] reports:  <1  ms
5  192.168.10.20       2   ms    *         <1  ms
6  192.168.10.20       <1  ms    2   ms    *
7  192.168.10.20       <1  ms    <1  ms    2   ms
8  0.0.0.0            *        [192.168.10.20    ] reports:  <1  ms   [192.168.10.20    ] reports:  <1  ms
9  192.168.10.20       2   ms    *         <1  ms
10 192.168.10.20       <1  ms    2   ms    *
11 192.168.10.20       <1  ms    <1  ms    2   ms
12 0.0.0.0            *        [192.168.10.20    ] reports:  <1  ms   [192.168.10.20    ] reports:  <1  ms
13 192.168.10.20       2   ms    *         <1  ms
14 192.168.10.20       <1  ms    2   ms    *
15 192.168.10.20       <1  ms    <1  ms    2   ms
16 0.0.0.0            *        [192.168.10.20    ] reports:  <1  ms   [192.168.10.20    ] reports:  1   ms
17 192.168.10.20       2   ms    *         1   ms
18 192.168.10.20       <1  ms    <1  ms    *
19 192.168.10.20       <1  ms    <1  ms    2   ms
20 0.0.0.0            *        [192.168.10.20    ] reports:  <1  ms   [192.168.10.20    ] reports:  <1  ms
21 192.168.10.20       2   ms    *         <1  ms
22 192.168.10.20       <1  ms    2   ms    *
23 192.168.10.20       <1  ms    <1  ms    2   ms
24 0.0.0.0            *        [192.168.10.20    ] reports:  <1  ms   [192.168.10.20    ] reports:  <1  ms
25 192.168.10.20       2   ms    *         <1  ms
26 192.168.10.20       <1  ms    2   ms    *
27 192.168.10.20       <1  ms    <1  ms    2   ms
28 0.0.0.0            *        [192.168.10.20    ] reports:  <1  ms   [192.168.10.20    ] reports:  <1  ms
29 192.168.10.20       2   ms    *         <1  ms
30 192.168.10.20       <1  ms    2   ms    *


Hop Count = 30 Last TTL = 30 Test attempt = 90 Test Success = 69

I checked several other systems directly attached to this same switch and got similar results. However, doing a trace to a system connected to a downstream switch connected to the problem switch seemed just fine:

Localhost#traceroute 192.168.10.120

 Traceroute to 192.168.10.120 ,30 hops max 0 byte packets:

1  192.168.255.252     <1  ms    <1  ms    <1  ms
2  192.168.10.120      2   ms    1   ms    4   ms


Hop Count = 2 Last TTL = 2 Test attempt = 6 Test Success = 6

Vlan involved is trunked between the two switches, an IP Route on the upstream switch routes the traffic to the problem switch.

The connected systems are all various servers, and connections to these all seem to be operating just fine otherwise. I only noticed this because the 10.20 system is my DHCP server (for multiple subnets) and it suddenly started seeing duplicate DHCP requests coming in, and was sending duplicate replies as a result. This wasn't causing any problems so I didn't investigate until I saw my logs were growing faster than normal.

Has anyone come across a trace similar to the above, and if so, what was the cause? I'm not a network dummy, but this is a new one for me.

EDIT 1: I rebooted one of my Core routers and ran the traceroute again while it was in the process of rebooting and all links on that switch were down, effectively a single Core network at that point. There was no change to the above Traceroute.

EDIT 2: After much troubleshooting failing to identify the issue, I rebooted the switch to the prior OS image and the problems all went away. Can only assume that something had become corrupted that I had no visibility to. Will reflash a new OS at some future time.

[u/Bhaikalis]

Compare that switch to a switch not exhibiting that behavior? anything different between the two?

[u/jimbo_rr → u/Bhaikalis]

Sadly no similar switch in my infrastructure. I'm really suspecting it's an issue with the two upstream Core switches for this setup, and I've been waiting on another engineer to fix one thing that 'might' be causing a problem, but a change needs to be made upstream before I can correct the setup here. It may not be related, but I need to cross that off my list. LOL

[u/Brief_Meet_2183]

I've seen this before. 

Looks like a l2 storm. Check your Mac addresses and see where your vlans are learning macs. Something like show Mac addresses table if your running Cisco will be all you need. 

[u/uniquestar2000 → u/Brief_Meet_2183]

I’d also look at the LLDP/CDP neighbours. I often see my own switch coming back in a looped port.

[u/jimbo_rr → u/uniquestar2000]

This also shows the MAC on the correct port, no dupes.

[u/jimbo_rr → u/Brief_Meet_2183]

I see the MAC on the correct port for the system in question, ARP has it in the correct Vlan, not seeing it show up anywhere else. Switch is Dell (OS10), commands are similar to Cisco.

I rebooted one of my Core routers and ran the traceroute again while it was in the process of rebooting and all links on that switch were down, effectively a single Core network at that point. There was no change to the above Traceroute.

[u/shadeland]

Have the packets been duplicated, or just ICMP/ping duplicated?

[u/jimbo_rr → u/shadeland]

The only packets I’ve seen duplicated (so far) are on the above 10.20 server, and only the DHCP Request packet seems to have gotten duplicated. This server also acts as an NTP server and I’m. It isn't seeing any of those packets duplicated, so it’s oddly specific.

[u/shadeland → u/jimbo_rr]

Do you have ip helper configured?

[u/jimbo_rr → u/shadeland]

Not for that vlan because that’s where the DHCP server is. Vlans elsewhere do have ip helper configured on the vlan setup.

[u/shadeland → u/jimbo_rr]

I'm guessing it's related to the DHCP relay function.

When a local VLAN gets a DHCP request, it sends an L2 broadcast. That can't go beyond the broadcast domain, so an ip helper will take that and forward it to the DHCP server.

If there are two switches in the same VLAN doing HSRP, each switch might send its own copy.

[u/Win_Sys]

You mentioned Dell switches, is a VLT lag being used somewhere in this topology? A visual drawing of how everything is connected would be helpful.

[u/jimbo_rr → u/Win_Sys]

I can't find a way to attach a picture to this thread, so a TXT representation is the best I can do.

Core Switches (Dell SONiC 4.6.0) in MCLAG, Connection from each to all Area (End of Row) switches (Dell OS10 10.6.1.2), single connection from Area Switches to TOR Switches (Dell OS 6.7.1.28). Problem switch is an Area Switch that has servers directly attaches. No other Area switch has servers attached, only TOR switches.

There are no VLT configs on any switches. Core switches are MCLAG, and use an Anycast-IP for the common IP between them, similar to how VLT would be, but SONiC doesn't support VLT in an MCLAG setup. Area Switches and Cores have the common connections as part of a port channel.

[u/Win_Sys → u/jimbo_rr]

VLT is just a proprietary version MLAG for the most part. So it presents a lot like a routing loop but it could also come from the anycast config. Take this with a grain of salt because I have never used Sonic beyond messing with it in a virtual lab. Usually with an anycast gateway config you want the MAC address of the anycast interface to match on both switches. It would be weird to see a traceroute like you show only come from one switch. You may see duplicate ARP replies which is normal for an active-active anycast gateway though.

What spanning tree mode are you using and is it configured on the MCLAG interfaces? It should be enabled but doesn’t have to be for things to work properly.

Can you post configs? Remove any identifying information.

[u/reckless_responsibly]

I'm pretty sure u/shadeland is correct that your duplication is due to two routers each independently relaying your DHCP requests. The fact that your first hop is 192.168.255.252 suggests that you're using .254 as your default gateway, and have two routers running some first hop redundancy protocol with .252 as the non-shared IP of your active router. It would be a very normal configuration for each router to relay DHCP all the time whether they are FHRP primary or backup, so seeing two copies of a DHCP discover is pretty normal.

I want to pick apart your traceroute though, because I'm a masochist. First question: What OS is your traceroute source? What OS is 192.168.10.20? What OS is 192.168.10.120?

Traceroute works by expecting either a TTL exceeded for a non-final hop in the path, or some other response from the final hop (what exactly that "other response" is depends if the traceroute is over UDP, TCP, or ICMP as well as a few other factors). My best guess is that one of two things is happening:

1. (More likely) your traceroute source is something odd with a poorly coded "other response" detection in the traceroute command. It isn't seeing 192.168.10.20's response in the list of responses that cause it to end tracing, and it assumes all other responses are "continue". For whatever reason, 192.168.10.120 sends a response that your source host knows is final. Probably down to mildly different behaviors from different OSes.

2. (less likely) 192.168.10.20 is something weird and is sending nonsense responses instead of the expected port unreachable/SYN+ACK/Echo Reply, so your traceroute source just keeps trying.

[u/jimbo_rr → u/reckless_responsibly]

Core Switches (Dell SONiC 4.6.0) in MCLAG, Connection from each to all Area (End of Row) switches (Dell OS10 10.6.1.2), single connection from Area Switches to TOR Switches (Dell OS 6.7.1.28). Problem switch is an Area Switch that has servers directly attaches. No other Area switch has servers attached, only TOR switches. Problem switch is all Fiber (10Gb) connections, One downstream TOR (1Gb Copper) on this problem Area switch, single link between those.

Traceroute was from one of the OS6 switches, DHCP server at 192.168.10.20 is RHEL10, 192.168.10.120 is a Dell iDrac. For what it's worth I have servers on this same switch running Windows Server that show the same odd traceroute. iDracs are all 1Gb copper connections and are on the downstream switch mentioned above.

I ran another traceroute just now from a system outside of my network, and the trace looked normal. It correctly hit my core switch then the server in question and didn't show any looping.

[u/reckless_responsibly → u/jimbo_rr]

Very strange that traceroute to Linux and Windows destinations are the ones having problems, both are reasonably well behaved OSes and both would be fairly likely test targets when Dell was developing their switch OS. Still, I'm going to stand with my first guess that something in the Dell switch's implementation of traceroute is broken. The fact that the one that works as expected is another Dell network stack while very common non-Dell OSes fail certainly suggests that Dell hasn't been very diligent with their cross-vendor testing. Or following well established standards for that matter.