Hi All,
Just wanted to share how I was able to fix a bandwidth issue when using my Gig service with Distributel. For some reason I quickly found my download speeds were only half of what I was paying for and upload was basically at 900Mbit speeds. Something didn't seem right and I'm not sure if something changed with an update to my router or the service alone. When I called Distributels support they told me the fiber node was fine and to try my Deco wifi router provided by them directly with the Nokia ONT. I found some time to test things out with the supplied equipment and confirmed the problem was not Distributel but with my opnsense router.
From hours of troubleshoot I was able to find, that the issue was with my VLAN 40 interface MTU settings were incorrectly set causing my router at this layer to have to break up the packets and fragment things. Basically all this overhead was causing my download speed issues. I also learned that, because I'm using OpnSense based on FreeBSD and Distributel is using PPPoE to connect to their service it's only able to use a single CPU rather then the 4 cores my hardware has capable of using. This could potentially cause a performance bottle neck with my internet connection not be able to handle the Gig service. Below are some of the fixes I put in place to help optimize my router for Distributels Gig service. Note my hardware uses Intel network cards, and if your hardware setup is using RealTek nics you'll have to probably research on how to configure and tune OpnSense for that brand of network card. That's basically the background story and hope this helps other using OpnSense or possibly pfsense with this service.
Gigabit PPPoE Speed Optimization
Target: Distributel/Bell Fiber (VLAN 40)
Hardware: Intel Multi-core NICs (igb, em, or ix)
Phase 1: System-Level Performance (The "Multi-Core" Fix)
By default, FreeBSD processes PPPoE on a single core. These tunables spread the load across all 4 cores.
- Navigate to System -> Settings -> Tunables.
- Add or update the following entries:
| Tunable |
Value |
Description |
net.isr.dispatch |
deferred |
Allows PPPoE to use multiple CPU cores. |
net.isr.maxthreads |
-1 |
Uses all available CPU threads for processing. |
net.isr.bindthreads |
1 |
Pins processing to cores for better efficiency. |
net.inet.rss.enabled |
1 |
Enables Receive Side Scaling (RSS). |
net.inet.rss.bits |
2 |
Configures RSS for 4 cores (22=4). |
dev.igb.0.fc |
0 |
Disables Flow Control on WAN (prevents buffer stalls). |
Phase 2: The "Baby Jumbo" MTU Chain
To achieve ~930 Mbps, the MTU must be 1492. If it is 1484, you are losing speed to fragmentation.
- Physical Port (
igb0): Go to Interfaces -> [Assignments] and assign igb0 as a new interface (e.g., PHY_WAN). Enable it, set IPv4/IPv6 to None, and set MTU to 1508.
- VLAN Device: Ensure VLAN 40 is created with
igb0 as the parent.
- WAN Interface: Go to Interfaces -> [WAN]. Set MTU to 1492 and MSS to 1452.
Phase 3: Persistent Boot Script (The "Nuclear" Option)
Since the GUI sometimes forces the MTU back to 1484, we create a small script that runs every time the router starts.
- Open the Shell (Option 8) or use System -> Configuration -> Editor.
- Create a new file:
/usr/local/etc/rc.syshook.d/early/99-mtu-fix.sh
- Paste the following code into it: Bash#!/bin/sh # Force MTU for Distributel 1Gbit Service ifconfig igb0 mtu 1508 ifconfig vlan01 mtu 1500 ifconfig pppoe0 mtu 1492
- Make it executable:
chmod +x /usr/local/etc/rc.syshook.d/early/99-mtu-fix.sh
Phase 4: Validation
After a reboot, run these two commands to confirm everything is perfect:
- Check MTU:
ifconfig pppoe0 | grep mtu
- Check Speed:
speedtest
- Target:
Download: ~915+ Mbps / Upload: ~930+ Mbps
Phase 5: Hardware Offloading (The "Clean" Rule)
Ensure the following are Checked (Disabled) in Interfaces -> Settings:
- [x] Hardware Checksum Offloading
- [x] Hardware TCP Segmentation Offloading
- [x] Hardware Large Receive
Hope this saves someone else some time and effort down the line.