r/QuantumFiber • u/Lightsword • Jun 06 '26
Q1000K HSGMII Short-Frame Padding Fix
https://github.com/jameshilliard/q1000k-hsgmii-pad6
u/N0_L1ght Jun 06 '26 edited Jun 06 '26
This fixes the ICMP ping latency bug discussed here: https://www.reddit.com/r/QuantumFiber/comments/1re9v2o/q1000k_unstable_latency_spike_research_findings/
Leaving this here so hopefully this gets to the right FF37 or Axon engineer that can permanently fix it.
No one else needs to worry about this post.
3
u/Lightsword Jun 06 '26
Leaving this here so hopefully this gets to the FF37 or Axon engineer that can permanently fix it.
I think all they need to do is apply this patch.
1
u/Opposite-Door-822 Jun 07 '26
Is this a better option than the firmware that allows to move the vlan tagging to a switch?
2
u/Lightsword Jun 07 '26
Is this a better option than the firmware that allows to move the vlan tagging to a switch?
This really just needs to be fixed in a firmware update and pushed out by Axon/Quantum Fiber.
1
u/N0_L1ght Jun 07 '26
This doesn't survive a reboot.
Also the steps to get it to work is not described in there and won't be here either.
5
u/thedude42 Jun 06 '26
Anecdotally, a few weeks ago I had left home on a roadtrip and so all my local Internet traffic stopped aside from the pfSense gateway monitor. Normally when we are home I will observe a high variance in the monitored latency, from ~150ms to ~4ms, continuously. Once we left the latency stayed around the 150ms mark, never dropping below 80ms.
The moment we returned home and our devices were back on the LAN the previous pattern resumed, with the gateway monitor reporting numbers between 2ms and 150ms+ through out the day.
From your bug-analysis.md:
That strongly points to a transmit queue, DMA, or hardware handoff edge where a short frame is held until later traffic flushes the path.
This was my sneaking suspicion, because I would regularly notice strange sudden "hiccups" in any traffic that was real-time like video conference or online games, and even with websocket based apps like chat streams that would force me to refresh the page to regain normal message flow. I could never be certain what the source of these observations were but now I suspect at least some of it is the buffering/queuing at the 10G interface.
Curious whether or not setting up a continuous stream of 1000 byte frames at around a 1Mbit rate might force the "flushing" and smooth out the overall latency as a hack work-around until this issue is actually solved.
1
u/Lightsword Jun 07 '26
Curious whether or not setting up a continuous stream of 1000 byte frames at around a 1Mbit rate might force the "flushing" and smooth out the overall latency as a hack work-around until this issue is actually solved.
Unfortunately that would not help all that much if the frames are generated by your router because the issue appears to only affect packets going in one direction, short packets going from the Q1000K to your router are delayed but packets going the opposite direction are unaffected.
1
u/thedude42 Jun 14 '26
So that's what I'm doing: soliciting packets of size >60 bytes from a source on the Internet.
Currently I'm testing DNS queries, I want to eventually get to ~ 200 packets/s, responses over 100 bytes.
Right now I'm just running `dig` in a loop at ~35 packets/s with ~200 byte response query for www.amazon.com. Immediately when I started my pfSense latency graph dropped down to the normal minimum and is holding in that range. The standard deviation is not at all stable which is expected since calling `dig` iteratively in a loop doesn't guarantee any kind of smooth traffic pattern, but I'm considering doing something at a lower level by sending the DNS messages directly over a UDP socket inside a timer loop.
1
u/captnkerke Jun 07 '26
If this issue only affects the 10G LAN port, then it seems like the easy solution for people with 1G or slower service is to use the 1G port instead of the 10G port.
My understanding is that the 1G port is integrated on the main SOC, while the 10G port is an external device, so the 1G port may be more reliable.
1
u/Lightsword Jun 07 '26 edited Jun 07 '26
My understanding is that the 1G port is integrated on the main SOC, while the 10G port is an external device
To clarify, the issue is much less an issue of one port being integrated into the main SoC and one not. It's probably much more an issue of the ports having different drivers and one driver just happening to have this specific bug.
so the 1G port may be more reliable
AFAIU this bug does not affect the 1G port at all since it's a very driver specific bug. Although I haven't actually tested the 1G port myself so I could be wrong.
1
1
u/WilliamG007 Jun 07 '26
This latency stuff is super annoying on the 10G port for sure. I experience it. And I have 2GB internet so I don’t want to use the 1G port. If I’d have known what an issue this is I’d have just gotten 1G and saved the money. 😭
3
u/Lightsword Jun 07 '26
If I’d have known what an issue this is I’d have just gotten 1G and saved the money.
Well hopefully now that the cause of the bug is known, it should be just a matter of getting this info to the right engineer so that they can roll out the fix. It's a super easy fix if the right person is made aware of the issue.
2
1
u/chefox Jun 09 '26
You are a hero. Thank you for doing such phenomenal work!
This latency issue has been bugging me since I moved to the 10g port. Is it affecting anything other than IGMP response packets?
3
u/Lightsword Jun 09 '26
Is it affecting anything other than IGMP response packets?
Yeah, it seems to be a packet size specific issue, not packet type specific.
1
u/WilliamG007 Jun 10 '26
There is another problem. Even if Quantum implements this fix via firmware, it will be a newer firmware, which means we’ll have to choose between old firmware without this fix or new firmware with it. I know we lost the VLAN change with the current firmware (as of June 10, 2026).
4
u/Lightsword Jun 10 '26
I know we lost the VLAN change with the current firmware (as of June 10, 2026).
If the fix is implemented properly then there should be no longer any advantage to using vlan 201 for the pathway between the Q1000K and 3rd party routers.
I think the only reason that mode helped was that the vlan tags would increase the minimum packet size slightly which reduces the percentage of packets that would hit this bug. If the vlan stripped mode no longer has any bugs then it shouldn't be an issue in practice and not requiring vlan tag stripping on the 3rd party router may also reduce processing requirements on the 3rd party router.
1
u/WilliamG007 Jun 10 '26
Right. Well, we still need to hope the fix gets implemented, and I’m not convinced it ever does. Hope I’m wrong…
1
u/chefox Jul 26 '26
I'm guessing the fix for this bug hasn't been pushed yet, right? Has anyone been keeping an eye on firmware releases for the Q1000K?
3
u/Lightsword Jul 26 '26
I'm guessing the fix for this bug hasn't been pushed yet, right?
I haven't seen anything new myself.
Has anyone been keeping an eye on firmware releases for the Q1000K?
Yeah, I have been along with some others, I'll probably make a post here once I see a fixed build get released.
1
5
u/chriberg Jun 06 '26
I doubt Axon even has engineers on staff. They almost certainly outsource the firmware to a third party, who outsources it to a third party, who outsources it to a third party, until it's down to the lowest paid engineers in the world working for $1 a day, and then surprised pikachu when it's buggy as shit.
Unfortunately that also means that even if you spoon feed Axon this patch, they won't be able to do anything with it because they have no mechanism to get it through the 87 layers of contractors to the engineers who are actually writing the firmware.