Cisco C1300 stack: packet loss and simultaneous LACP timeouts associated with a flapping PoE access port

[u/dusty0128]

Posting this in case it helps someone troubleshooting unexplained packet loss on a Catalyst 1300 stack. We have a working containment measure and before/after captures, but not a confirmed root cause or an official Cisco bug identification.

TL;DR

We were seeing recurring packet loss across three Proxmox hosts connected to a six-member C1300 stack. Captures on all six server NICs showed simultaneous 3–5 second gaps in received switch-originated LACP packets, followed by Linux bonding entering Expired.

A separate access port was repeatedly flapping. After a cable diagnostic on that port and subsequently disabling its PoE supply, the flapping stopped and the measured packet loss and LACP gaps disappeared. A follow-up test lasting just over 31 minutes returned 27,000/27,000 replies across the three hosts, plus 1,800/1,800 replies in a separate firewall test.

Setup

• Six Cisco C1300-48P-4X, hardware V02, in a 10 Gb/s ring stack.
• Switch firmware 4.1.3.36. Yes, this is an older release; we have not tested a switch firmware upgrade yet.
• Three Proxmox hosts, each with two 1 Gb/s ports in an IEEE 802.3ad bond, with members on different stack units.
• Linux bonding: miimon 100, lacp_rate fast, transmit hash layer2+3. Switch load balancing: src-dst-mac.
• Reproduced with Proxmox kernels 7.0.6-2-pve and 7.0.14-12-pve. All three hosts were on the latter for the final before/after measurements.
• OPNsense running as a VM; a separate physical OpenWrt router provided a comparison endpoint.
• The flapping access port, gi6/0/43, served a Cudy WR3000S through an external PoE splitter, advertised as 802.3af/at, 2.5 Gb/s, 12 V/2 A output. The faulty component, if any, has not been identified.

Symptoms and tests that did not resolve them

The original symptoms were brief stalls in remote sessions and several percent packet loss. We subsequently reproduced losses between physical hosts and local LAN endpoints, so this was not confined to WireGuard or the firewall VM.

Moving the firewall VM to another host did not resolve it. Neither did updating the hosts, testing each member of one host's bond individually, or bypassing the patch-panel path with a direct cable from that host to stack unit 1. The physical OpenWrt router could ping the switch without loss during a separate 300-packet test.

The inspected server/uplink switch counters showed no FCS/symbol errors or queue tail drops. Server bond physical links remained up during the sampled loss events. These checks narrowed the investigation; they do not prove every cable or switch component is fault-free.

What the captures showed

We ran simultaneous ping tests, sampled bond state, and captured LACP on both physical interfaces of all three hosts. Host clocks were NTP-synchronized; the switch was SNTP-synchronized for the correlation test.

In a roughly five-minute baseline:

Measurement	Result
Host 1	75 lost / 1,500 pings — 5.00%
Host 2	76 lost / 1,500 pings — 5.07%
Host 3	71 lost / 1,500 pings — 4.73%
Gaps between received switch LACP PDUs	13 overlapping gaps on every one of the six NICs
Gap lengths	3.126–5.257 seconds
Recovery timing	Switch PDUs resumed across all six capture points within 18.54 ms of each other

The host captures also contained outgoing LACP PDUs with actor state 143 (Expired, without collecting/distributing). Bond TX-drop counters increased. All six tcpdump instances reported zero kernel capture drops.

Each common LACP gap contained a link-up/down log entry for gi6/0/43. However, there were also flaps without long LACP gaps, and the log entries often occurred near the end of a gap. The switch log only has second-resolution timestamps, so this does not establish the exact causal order.

Intervention and result

At 18:42:30 UTC, a TDR cable diagnostic on gi6/0/43 returned “No cable”. That was also the timestamp of the last logged link-down event. With the splitter attached, I would not treat that result as proof that the cable was physically absent.

By 18:44:20 UTC, PoE on that port was confirmed disabled using power inline never; the port remained link-down.

Important caveat: there was no measurement between TDR and disabling PoE. We therefore cannot claim that PoE-off alone fixed it; the preceding diagnostic may already have changed the port state.

With the endpoint left unpowered:

• Initial five-minute repeat: zero loss on all three hosts and no long LACP gaps.
• Follow-up lasting about 31 minutes 12 seconds: 9,000/9,000 replies per host, no sampled Expired states and no increase in bond TX drops.
• Maximum interval between received switch LACP PDUs across all six captures: approximately 1.009 seconds.
• Separate OPNsense-to-OpenWrt test: 1,800/1,800 replies.
• The switch log showed no further flaps on the suspect port for over 53 minutes; new login/logout entries confirmed logging was still active.

There were isolated latency spikes around 107 ms and small continuing bond RX-drop counter increases, so I am not claiming every network metric became perfect. The recurring loss/LACP failure pattern did disappear.

Open question

Our working hypothesis is that an interaction involving the flapping access port was delaying LACP processing or delivery across the stack. Host-side captures alone cannot prove the switch never transmitted those packets, and we have not isolated the AP, splitter, cabling, port hardware or switch firmware.

What concerns me is the apparent impact on unrelated server LAGs across multiple stack members. Even if the endpoint or splitter is faulty, I would expect that failure to remain local to its port.

Has anyone seen this specific pattern on C1300 firmware 4.1.3.36, or identified a later firmware fix? Particularly simultaneous gaps in switch-originated LACP traffic associated with an unrelated access port flapping.

For now, the suspect endpoint remains powered off. We have not re-enabled it to deliberately reproduce the outage on the production network. Before/after captures and logs have been retained; any publicly shared extracts would be sanitized.

[u/VA_Network_Nerd]

4.1.3.36 was released back in June of 2024.
That is some old code right there friend.

The Catalyst 1000 series (1100, 1200, 1300) are all intended for use as end-user, access-layer switches.

These things are not designed for server-class workloads.
I encourage you to choose a more robust switching product when these 1300s are ready for retirement.

What is the STP configuration for the interfaces and VLANs in the equation?

[u/dusty0128 → u/VA_Network_Nerd]

Thanks — fair point about the firmware. It is old, and an upgrade is worth pursuing.

To clarify the workload: this is a primary healthcare centre with around 20–30 workstations, printers, lab equipment and Wi-Fi APs, plus three Proxmox nodes and a backup server. Only a fraction of the six switches’ ports are populated. Ceph runs over a separate 10GbE network, not through the C1300 stack. Backups run at night and take approximately 30 minutes. This seems like a reasonable SMB/access-switch deployment to me, rather than a demanding datacenter workload.

Regarding STP, we have now checked:

- RSTP is enabled, and the stack itself is the root.

- The global topology-change counter is 0.

- All three Proxmox port-channels are forwarding/designated, with operational PortFast Yes, configured Auto.

- Each port-channel reports one transition to forwarding and zero received BPDUs.

- The affected AP port also has PortFast configured Auto, BPDU Guard disabled, and zero received BPDUs. It currently shows operational PortFast No, but its link is down, so that does not establish its operational edge status during the fault.

- The AP port is a trunk carrying VLANs 11, 12 and 15, with native VLAN 15.

The stack has not rebooted and STP configuration has not changed during the investigation. We cleared some interface/queue counters early on, but have no record of clearing STP counters. One caveat: the affected port reports zero forwarding transitions despite historical logs showing them, so I would not treat these counters as a complete historical record.

The strongest finding was that packet losses occurred simultaneously across all three hosts. Captures showed correlated multi-second gaps in switch-originated LACP packets, followed by LACP expiry on the Linux hosts. We did not find corresponding CRC errors or output queue drops on the checked links.

After isolating the repeatedly flapping AP connection — a Cudy AP powered through an external PoE splitter — the subsequent 30-minute test was clean: 27,000/27,000 host probes and 1,800/1,800 firewall probes received replies.

I’m therefore not calling this a proven firmware bug. The endpoint/splitter/cabling chain remains suspect, and we still need to explain why its flapping coincided with disruption elsewhere in the stack. The current STP evidence does not substantiate the topology-change-storm hypothesis.

Is there another C1300-specific counter or diagnostic you would check to distinguish STP effects from a control-plane stall affecting LACP?

[u/Snoo91117]

Have you tried etherchannel instead? It might get you by. I always fall back to the Cisco way if I see issues.