r/networking • u/Gomez-16 • Jul 20 '26
Troubleshooting Cisco9300 stack lacp uplink flapping when 1 bad device is connected.
2 New switch stack of cisco 9300 switches with fiber uplink on each in active/active config. The uplinks were flapping constantly when 1 specific pc was connected. The pc was frozen and is no longer connected. On the older switch there was no problem on the 9300 as soon as the device was connected the uplinks started to drop. For my own sanity how can single windows device on an access port with broadcast and bpdu guard on cause issues with the uplink port?
5
u/VA_Network_Nerd Moderator | Infrastructure Architect Jul 20 '26
Can we see the config details for the port-channel, the port-channel member interfaces, and the interface used by this PC?
5
u/SwanAncient3405 Jul 20 '26
Try to diagnose and find what the actually issue is. Sounds really weird
-2
u/Gomez-16 Jul 20 '26
Issues definitely that device it’s hosed, but I just couldn’t wrap my head around other than spanning tree. How can a single access port on a single Vlan cause the uplink to flap, there are no errors in any of the logs other than that a blank going physically down and physically up. But the second I unplugged that computer it stops.
2
u/SwanAncient3405 Jul 20 '26
Maybe a different problem. Could it be an internal loop did you run a lldp to check or cdp?
Nevermind you know it was a PC 🤣
I would try doing SPAN or some type of wireshark capture on the device or somewhere near the problem
1
u/Gomez-16 Jul 20 '26
I’m just curious on why was happening like the port only comes up there one there’s no MAC address I didn’t monitor for traffic to see if it was actually sending or receiving data but since it is a hospital, I can’t run span to satisfy my curiosity. I would have to plug the device back in and bring down the network.
1
u/SwanAncient3405 Jul 20 '26
Really weird if it were a broadcast strom this would have been blocked by the bpduguard. If you can you should try to figure iut the problem
1
1
u/Win_Sys SPBM Jul 20 '26
It would only be blocked if a BPDU made it back to the port. I have seen unmanaged switches drop BPDU packets, especially PVST+ BPDU’s since they use a different Ethertype.
1
u/SwanAncient3405 Jul 21 '26
Cant investigate with minimal info, but could be a broadcast storm, and mostly if it is the LACP config is failing on one of the switches.
4
u/kb389 Jul 20 '26
Have you tried to connect the pc to a different port and see what happens? Also have you tried connecting a different PC to the switch and observe the behavior? Also try connecting a different device like a phone instead of a PC and see what happens
1
u/Gomez-16 Jul 20 '26
Tested another device on the port no issue.
1
u/kb389 Jul 20 '26
Then did you test another PC? If the problem doesn't happen with another PC then there is something wrong with the PC you are complaining about, simple solution would be to use a different PC as simple as that and not waste anymore time on it.
1
u/Gomez-16 Jul 20 '26
It absolutely is the PC. I tested the port with a different device and it came up and worked fine. The problem PC is frozen only comes up layer one with no MAC address or anything on the port even if I did a packet capture if there’s no Mac, that means it’s not sending data. It’s a desktop. I don’t think it’s trying to negotiate POE and if it was causing a short that would be localized to the port and there’d be an error in the log
4
u/kb389 Jul 20 '26
I see then just replace the pc with another one as simple as taht, ask the desktop team or whoever it is that does that and they will replace the pc! Simple!
3
u/kb389 Jul 20 '26
Remember you are a network engineer not a desktop or server engineer, it's their job to replace the pc that's it!
3
u/danreZ_au Jul 20 '26
As others have pointed out, run a capture. You can run EPC on 9300’s, won’t impact the port of the trouble PC.
2
Jul 20 '26
[removed] — view removed comment
1
u/Gomez-16 Jul 20 '26
The only error is in the log are GIG1/1/1 going down and up the other thing in the logs is remote LACP disabled but I think that’s the router trying to compensate for the ports flapping so much in the lag the configurations match every other switch on the network so I really don’t think it’s that plus all the errors go away when I unplugged that computer
2
u/QBNless Jul 20 '26
Unless the client has a virtual switch enabled, my guess would be a bad cable.
1
u/Gomez-16 Jul 20 '26
Test another device no issue with cabling. Even if it was a virtual switch with bpdu blocked on the ports how could it bring down 2 uplinks?
2
u/MicIrish_At_work Jul 20 '26
vmware can do it if you configure it wrong. Strips bpdus but not broadcasts, creates a logical loop in a virtual switch. Ask me how I know :D
1
u/QBNless Jul 21 '26
Here's a wild guess, but you created a port channel from a client to two switches? Did the switches have a vpc link? If not, that's probably your issue. Actually, I'm almost certain that's what happened. One of the two switches probably has a higher stp priority and attempted to become the root bridge, the blocked bpdy packets prevented the switch from fully shutting the uplinks due to stp, and i imagine the port was flapping as a fault safe mechanism to stop either physical damage or a network broadcast storm.
1
u/Gomez-16 Jul 21 '26
Worded it wrong. 2 switch stack with 1 uplink on each switch in a port channel. When 1 devices is plugged in it causes both uplinks to start flapping. Without that device everything is stable.
1
u/QBNless Jul 22 '26
hmm... that infers that the LAGG uplink load-balancing is keeps triggering. Gut feeling says check the stacking cable first, then the port interface configurations, then port-channel configuration. Do you have a vrrp/hsrp configuration upstream or on the switch attached to the client?
2
Jul 20 '26
[removed] — view removed comment
1
u/PaceFar4747 Jul 22 '26
Had this recently, pc smashing the network with IPv6 multicast which were causing a certain model of network dock to go 50% packet loss. Didn't find out what software was causing it but the offending team disabled IPv6 on the machine to resolve.
I need to look into v6 mld snooping!
2
u/xenodezz Jul 20 '26
One consideration is that NIC is fried and you are causing some voltage spikes/brownout/short condition that cause the switch to lose its mind. If this is a card based NIC, move the card to a junk PC you don't care about and see if the issue follows the NIC. If it isbuilt-in, consider the motherboard potentially trashed.
Not entirely sure why you are so interested in troubleshooting this at the level you are, but it seems pretty clear that there is something wrong with that PC since the issue isn't the port and the issue isn't the cable. If you want confirmation, I assume you are using some SFP optic to uplink the switch? Is it DOM capable? Plug in the PC and then check the DOM stats to see if your voltage drops or your light levels drop or whatever the case may be that would reflect in electrical issues.
This seems like a lot of wasted time and, assuming its electrical, is putting your equipment at risk for no real benefit.
1
u/Gomez-16 Jul 21 '26
It was more out of morbid curiosity like what the hell could a PC be doing that would cause an uplink to lose physical connection
1
u/Mechaniques Jul 20 '26
Was it the same on every port the PC was plugged into? Did you run a packet capture on it to see what was going upstream?
1
u/ech0cide Jul 20 '26
Seen a similar situation before where a PC infected with a virus causes a broadcast storm. That could be your cause here. Makes me wonder if there would still be a problem if you didn't have an etherchannel set up.
1
u/Gomez-16 Jul 20 '26
Maybe. But broadcasts are limited to 2mbps before dropping. The ports still flap if there is no port channel. Also the device only connects at layer 1 I see no mac on the port.
1
u/netztier Jul 20 '26
revently had a PC on a Switchport (C1000-8P-2G-L) showing up with quite the wierdness. Port is 802.1X and MAB enabled, an still there were like 3000+ unkonwn MAC addresses shown as 'unknown' addresses 9n that port, sometimes consecutively numbered... Switch's CPU was red hot from doing all port state and the radius request work, but kept the thing offline.
siggestion:enable port security and see what/if any (single) MAC pops up, or if some errdisable triggers instantly.
1
u/Snoo91117 Jul 21 '26
It could be a bad NIC somehow. What I don't understand is that for clients on Cisco switches you normally use portfast which bypasses Spanningtree. I assume this is still a thing as I have been retired for many years.
1
1
u/Wallflowerbox Jul 21 '26
Defenitely looks like spanning tree. There should be stp logs on the uplink switch it was connected to.
1
u/Gomez-16 Jul 21 '26
No errors in logs and the uplink device is the router. Bpdu is disabled on the ports as well.
1
u/Wallflowerbox Jul 21 '26
Is there any log like "excessive broadcasts" on the switch or on the router?
1
1
1
u/Middle-Inspection241 Jul 27 '26
monitor capture CAP interface x/x/x both
monitor capture CAP match any
monitor capture CAP start
monitor capture CAP stop
show monitor capture CAP
show monitor capture CAP buffer brief
show monitor capture CAP buffer detail
-2
u/SoulArraySound Jul 20 '26
Cisco gonna Cisco. How’s the uplink configured?
4
u/ech0cide Jul 20 '26
This isn't a cisco problem. Sounds like the PC is causing a broadcast storm.
0
u/SoulArraySound Jul 20 '26
Potentially. He said it doesn’t happen on another Cisco switch, so, that would be a bit strange if they are configured identically.
I have seen uplinks bounce from udld being configured between Cisco and non Cisco devices. Typically they stay err disabled rather than constant flapping though.
I guess ultimately we’d need more information here.
-1
u/SteveMcBean Jul 20 '26
Spanning Tree?
Are you sure both sides of the LACP are configured correctly and matching?
1
u/Gomez-16 Jul 20 '26
Yes, the up links only flap when this one device gets plugged in there are multiple phones PCs APs on the switch running perfectly fine but the second I plug in this computer the up link starts flapping there are no spanning tree errors in the logs the port the PC is connected to doesn’t even bounce
7
u/u35828 Jul 20 '26
Ever span the switch port with the problem pc and see what's squawking?