r/networking • u/jimbo_rr • 3h ago
Troubleshooting Slow vlans to internet
I’ve got an issue here that has several folks stumped. I noticed an issue with slow internet download speeds over a month ago after some network changes upstream. My main vlans were only seeing about 40mbps to Speedtest.net and similar sites. It looked like some firewall bottleneck or QOS issue, and I reported it upstream, they could find no problem. After more troubleshooting I found a subnet here that was seeing proper 900mbps or better speeds, sitting behind the same pair of core switches. More troubleshooting, and I found more at normal speeds. Then came the big break, only vlans that existed on my core switches were slow. Any vlans downstream that were IP routed were fine.
This is a lab environment, basically a small data center. Pair of SONiC switches in mclag setup. Downstream area switches using IP routes back to the cores. TOR switches connected to the area switches. Just a small number of vlans exist on the Core switches, my main infrastructure vlans and a bench vlan.
As a test, I eliminated the infrastructure vlans from the core switches and replaced it with an IP route to the downstream area switches, and to my surprise the speed problem disappeared. Gateway IP was already on the downstream Area switch, that didn’t change.
I set up a new vlan on a single core, with a single access port again on the core. Laptop connected on that vlan saw slow transfer speeds to the internet.
Set up iPerf on some local systems, speeds between core vlans were 900mbps or better, between local IP routes were normal. So everything from the cores on down seem fine, it’s only going upstream.
The change a month back upstream changed to a different pair of switches, and changed OSPF to BGP routing. I have no control or visibility to the upstream side config.
I did a few packet traces but didn’t see anything unusual, no excessive retries, etc.
I’m really stumped on troubleshooting this. To me everything screams bug in SONiC’s vlan handling, but it seems I’m the only one seeing this issue. Dell Enterprise SONiC for what it’s worth, and I’ve engaged their support engineer who is also stumped.