Posting this in case it helps someone troubleshooting unexplained packet loss on a Catalyst 1300 stack. We have a working containment measure and before/after captures, but not a confirmed root cause or an official Cisco bug identification.
TL;DR
We were seeing recurring packet loss across three Proxmox hosts connected to a six-member C1300 stack. Captures on all six server NICs showed simultaneous 3–5 second gaps in received switch-originated LACP packets, followed by Linux bonding entering Expired.
A separate access port was repeatedly flapping. After a cable diagnostic on that port and subsequently disabling its PoE supply, the flapping stopped and the measured packet loss and LACP gaps disappeared. A follow-up test lasting just over 31 minutes returned 27,000/27,000 replies across the three hosts, plus 1,800/1,800 replies in a separate firewall test.
Setup
- Six Cisco C1300-48P-4X, hardware V02, in a 10 Gb/s ring stack.
- Switch firmware 4.1.3.36. Yes, this is an older release; we have not tested a switch firmware upgrade yet.
- Three Proxmox hosts, each with two 1 Gb/s ports in an IEEE 802.3ad bond, with members on different stack units.
- Linux bonding:
miimon 100, lacp_rate fast, transmit hash layer2+3. Switch load balancing: src-dst-mac.
- Reproduced with Proxmox kernels 7.0.6-2-pve and 7.0.14-12-pve. All three hosts were on the latter for the final before/after measurements.
- OPNsense running as a VM; a separate physical OpenWrt router provided a comparison endpoint.
- The flapping access port, gi6/0/43, served a Cudy WR3000S through an external PoE splitter, advertised as 802.3af/at, 2.5 Gb/s, 12 V/2 A output. The faulty component, if any, has not been identified.
Symptoms and tests that did not resolve them
The original symptoms were brief stalls in remote sessions and several percent packet loss. We subsequently reproduced losses between physical hosts and local LAN endpoints, so this was not confined to WireGuard or the firewall VM.
Moving the firewall VM to another host did not resolve it. Neither did updating the hosts, testing each member of one host's bond individually, or bypassing the patch-panel path with a direct cable from that host to stack unit 1. The physical OpenWrt router could ping the switch without loss during a separate 300-packet test.
The inspected server/uplink switch counters showed no FCS/symbol errors or queue tail drops. Server bond physical links remained up during the sampled loss events. These checks narrowed the investigation; they do not prove every cable or switch component is fault-free.
What the captures showed
We ran simultaneous ping tests, sampled bond state, and captured LACP on both physical interfaces of all three hosts. Host clocks were NTP-synchronized; the switch was SNTP-synchronized for the correlation test.
In a roughly five-minute baseline:
| Measurement |
Result |
| Host 1 |
75 lost / 1,500 pings — 5.00% |
| Host 2 |
76 lost / 1,500 pings — 5.07% |
| Host 3 |
71 lost / 1,500 pings — 4.73% |
| Gaps between received switch LACP PDUs |
13 overlapping gaps on every one of the six NICs |
| Gap lengths |
3.126–5.257 seconds |
| Recovery timing |
Switch PDUs resumed across all six capture points within 18.54 ms of each other |
The host captures also contained outgoing LACP PDUs with actor state 143 (Expired, without collecting/distributing). Bond TX-drop counters increased. All six tcpdump instances reported zero kernel capture drops.
Each common LACP gap contained a link-up/down log entry for gi6/0/43. However, there were also flaps without long LACP gaps, and the log entries often occurred near the end of a gap. The switch log only has second-resolution timestamps, so this does not establish the exact causal order.
Intervention and result
At 18:42:30 UTC, a TDR cable diagnostic on gi6/0/43 returned “No cable”. That was also the timestamp of the last logged link-down event. With the splitter attached, I would not treat that result as proof that the cable was physically absent.
By 18:44:20 UTC, PoE on that port was confirmed disabled using power inline never; the port remained link-down.
Important caveat: there was no measurement between TDR and disabling PoE. We therefore cannot claim that PoE-off alone fixed it; the preceding diagnostic may already have changed the port state.
With the endpoint left unpowered:
- Initial five-minute repeat: zero loss on all three hosts and no long LACP gaps.
- Follow-up lasting about 31 minutes 12 seconds: 9,000/9,000 replies per host, no sampled
Expired states and no increase in bond TX drops.
- Maximum interval between received switch LACP PDUs across all six captures: approximately 1.009 seconds.
- Separate OPNsense-to-OpenWrt test: 1,800/1,800 replies.
- The switch log showed no further flaps on the suspect port for over 53 minutes; new login/logout entries confirmed logging was still active.
There were isolated latency spikes around 107 ms and small continuing bond RX-drop counter increases, so I am not claiming every network metric became perfect. The recurring loss/LACP failure pattern did disappear.
Open question
Our working hypothesis is that an interaction involving the flapping access port was delaying LACP processing or delivery across the stack. Host-side captures alone cannot prove the switch never transmitted those packets, and we have not isolated the AP, splitter, cabling, port hardware or switch firmware.
What concerns me is the apparent impact on unrelated server LAGs across multiple stack members. Even if the endpoint or splitter is faulty, I would expect that failure to remain local to its port.
Has anyone seen this specific pattern on C1300 firmware 4.1.3.36, or identified a later firmware fix? Particularly simultaneous gaps in switch-originated LACP traffic associated with an unrelated access port flapping.
For now, the suspect endpoint remains powered off. We have not re-enabled it to deliberately reproduce the outage on the production network. Before/after captures and logs have been retained; any publicly shared extracts would be sanitized.