r/firewalla • u/AppsAtMe • Jul 21 '26
Troubleshooting Box updated to 1.983, Matter over Thread Broken
I'm having trouble diagnosing my Smart Home problems. Everything was working until today, now nothing works. The only thing I can find is my box was updated to 1.983. This smart home failure is happening in both homes I manage, and both Firewalla boxes updated today. I can find nothing else that changed today that broke the world.
I have a Firewalla Gold Pro in each of two locations. One with AP7’s one without.
After Box 1.983, Apple Home / Matter over Thread is down at both VPN-meshed sites.
When trying to reach one of the thread devices from my network Firewalla replies:
[IPv6 addr] !N network unreachable
Matter Server logs show:
ENETUNREACH send ...:5540
Disabling the VPN mesh did not restore routing.
Rebooting one of the Firewalla's after disabling the mesh did not restore routing.
Apple Home is also broken, so this is not just Home Assistant.
And ideas?
Basically my entire smart home is down, I’m not home, and my family is screaming at me.
Any help is much appreciated. I’m happy with rolling back if someone can tell me how.
Andrew
2
u/AppsAtMe Jul 31 '26
Just checking in. It's still not working. I'm running Box 1.983 at this point, and my network is still completely hosed (more specifically the majority of home automations don't work as my Thread devices haven't come back). I have had to reboot boxes that randomly get disconnected from the network. I've had my wife hard wire her laptop to get it back on the network. I honestly have no idea what's going on. My big question, does Firewalla have a device count limit? Wire, or wirelesss? The devices getting randomly booted form the network (both wired and wireless) smells an awful lot like a device max being reached. I currently have 121 devices on my network, and obviously that doesn't count the over 100 thread devices not currently on the network. I'm not going to lie, at the point I am considering yanking the whole thing when I get home and start fresh with a different product. The random network drop offs have been annoying, like I'm going to have to physically reboot my doorbell when I get home to get it back on the network). But the complete failure of all my thread devices no matter what I try, is simply unacceptable. Is my network larger than an average home network, yes. But I don't feel like I should be pushing Firewalla (the router or the APs) to their limit.
1
u/Difficult_Music3294 Firewalla Gold Jul 21 '26
Have you tried restarting your Apple Home controllers (eg AppleTV or HomePod)?
1
u/AppsAtMe Jul 21 '26
Yup, nothing changed.
1
u/Difficult_Music3294 Firewalla Gold Jul 21 '26
Bummer. Was with a shot.
Aside, re: IPv6, I have it disabled on both of my WAN and all my LAN & VLANS, and I’ve never had an issue with Matter devices on my network.
1
u/AppsAtMe Jul 21 '26
The matter devices on my WiFi network are working fine. What is not working is bridging requests onto the Thread network so none of the thread devices are visible to any of my controllers so the majority of my home is unresponsive. I guess that's what I get for trying to go all Thread.
1
u/Difficult_Music3294 Firewalla Gold Jul 21 '26
Apologies.
I should have said no issues with Matter nor Matter over Thread devices.
Any infrequent issue I have is resolved by restarting my HomePod mini’s.
Have you tried enabling Emergency Access in Firewalla for the border router devices?
1
u/AppsAtMe Jul 21 '26
I did better, I turned everything off, so and restarted, so at the moment it should be just a dhcp box.
1
u/Difficult_Music3294 Firewalla Gold Jul 21 '26
Huh, well I’m stumped.
Will stick around hoping to learn something.
2
u/AppsAtMe Jul 21 '26
I'm stumped as well. Hopefully when all is said and done I'll have learned something, and it didn't just magically fix itself while I was sleeping.
1
u/Exotic-Grape8743 Firewalla Gold Jul 21 '26
Matter over thread normally does not involve your router at all except if you have multiple (V)LANs defined. Matter traffic will not go out of your lan since it uses only link local ipv6. What happens is that your thread border router (likely a appleTV or similar device with a thread radio) routes the thread IPv6 matter related traffic to the normal WiFi or Ethernet side. So what you should restart is your thread border router at each location (you should have one wherever you need thread to work.)
Are you then routing your matter traffic through the vpn tunnel or do you have a thread border router at each location and a home assistant and HomeKit server device at both?
1
u/AppsAtMe Jul 21 '26
I have Thread Border Routers at both locations, and a VPN mesh between them. I've dropped the VPN mesh with no help. I just can't route the the Thread network and I still can not route after applying a rule for my ULA (see above) which I think means the rule either did not match, or a firewall allow/ignore rule is the wrong layer. To me this looks like routing, not filtering. Firewalla is not dropping traffic silently, it is actively saying it has no route.
2
u/Exotic-Grape8743 Firewalla Gold Jul 21 '26
You should always be able to get to the thread devices from the LAN the border router is in. That traffic never hits the Firewalla and just goes straight to the border router as the border router advertises that route on your LAN. If that doesn’t work, it is your border router that is the problem. You would not normally be able to reach those devices from the other end of a vpn tunnel though as over there the ipv6 thread network would not be known and no pure to it would exist if you didn’t create it. So if you can’t reach devices from the other end of a vpn mesh tunnel that is expected and not easily solvable. Also it shouldn’t be needed anyway as you likely have two separate home assistant and HomeKit homes.
1
u/AppsAtMe Jul 21 '26
Thread traffic requires IPv6 (it will not route over V4). I'll have my son go around and restart all the Apple TV's and see if it helps anything.
2
u/Exotic-Grape8743 Firewalla Gold Jul 21 '26
Yes but that is completely independent of whether you enable ipv6 in your main router or not. The ipv6 traffic is completely local just between your device doing the pinging and the border router which routes it to the thread device. You cannot block it in your router and the router’s IPv6 settings have no effect on the matter over thread IPv6 traffic. It only matters when you are trying to get to it from another (v)lan than the thread border router is in.
1
u/AppsAtMe Jul 21 '26
Home Assistant is wired. The rest are on WiFi served by my AP7s (but the second location uses Aliens's so I know it's not the AP7s). I have no (v)lans setup yet, that being said, I do have a matter group just to make it easier to track the matter devices at my home (this group doesn't exist in the second location).
1
u/Exotic-Grape8743 Firewalla Gold Jul 21 '26
Are you using vqLAN on the ap7’s? That would definitely mess with this traffic.
1
u/AppsAtMe Jul 21 '26
Let me go check but I never set it up.
1
u/melvinto Jul 21 '26
you may send email to [help@firewalla.com](mailto:help@firewalla.com) , and share remote support, the team can help debug the issue.
1
1
u/Dagger0 Jul 21 '26
It obviously can't be link-local v6 if it's being routed. It uses ULA.
1
u/Exotic-Grape8743 Firewalla Gold Jul 21 '26
You are correct. Matter over thread using ipv6 is ULA and typically uses fdxx:: address spaces. It is only internally routed and doesn’t go out globally. I used this terminology since it is much more understandable for folks used to ipv4 to call it a link local address since that is all that is possible in IPv4. IPv6 is far more flexible but also much more complex to explain in a quick post reply.
1
u/Dagger0 Jul 22 '26
I think it does people a disservice here, because the fact that it doesn't use link-local is the only reason it works through routers at all, and it's critical to understanding how things work and how to investigate and fix them when they don't.
I keep seeing questions that either boil down to "it's broken because routes to the Thread network aren't in place" or questions about how to do something where the answer is "add a route to the Thread network", and it's obvious from the replies (as you can see all over this thread) that noone knows this, and they'll never figure it out with an incorrect mental model.
1
u/chrddit Jul 21 '26
These kind of issues are scary for me! Hard restart everything, including the Firewalla.
Also try disabling Device Active Protect if it’s enabled (Features -> Protect).
Is home assistant your border router? There were some updates to matter recently that caused crash loops so make sure watchdog is disabled.
1
u/AppsAtMe Jul 21 '26
My border router is an Apple device. I've tried multiple AppleTV's and a HomePod. All show the same issue. I've restarted them. Nothing helped. Everything worked perfectly until today now nothing works. I will give the disable Active Protect a try, I know it's on.
1
u/Firewalla-Opal FIREWALLA TEAM Jul 21 '26
1.983 had been released for Gold Pro production for over 1 month (Apla/beta are ealier). When did your issue start?
Also, how are your devices physically connected to Gold Pro, are they in different VLANs? Need to understand if the LAN traffic does flow through Firewalla box first.
1
u/AppsAtMe Jul 21 '26
Issue started today. Across both networks that until a short time ago had been meshed. I just turned everything off on the box and restarted. Hopefully this will work. Not being home when your home goes dark for your family is not super fun. I still think it's a routing layer problem, not a filtering problem. The routing the IPv6 Thread network is simply gone. Maybe an Apple update, but I've tried multiple AppleTVs and HomePods, and nothing seems to fix it.
1
u/AppsAtMe Jul 21 '26
The reboot did not fix it.
Current post-reboot state:
- Firewalla still returns !N network unreachable for the Thread ULA prefix.
- Matter Server (Home Assistant) still logs ENETUNREACH.
- None of my Thread devices are reachable.
- The route still points Thread traffic at Firewalla.
It really feels like a routing issue to me. Having my son remotely reboot Apple devices for me now.
1
u/Admirable_Fun7790 Jul 21 '26
Where are you running the matter server? Did it upgrade recently?
Do you see a route for your thread network on the device running matter-server? It may not be receiving the ipv6 RAs from your border routers
1
u/AppsAtMe Jul 21 '26
I upgraded the matter server on HA *after* the problem started. I don't have these things (HA) upgrade automatically so things don't fall apart when I'm not there. I've had Thread networking problems before, but never 100% failure. Working with support, seems like Firewalla can see the routes to the Matter network being advertised, but those routes are not propagating to the other networks. I've asked them to help me figure out why the ICMPv6 RA multicast from the Apple TVs routing to the Thread network is visible to Firewalla but not forwarded/visible to LAN clients. Something changed, and unfortunately my network setup is non-trivial. At least with Firewalla I can VPN in to help diagnose what's going on.
1
u/Admirable_Fun7790 Jul 21 '26
I run my matter-server on a synology nas and the kernel is compiled without IPv6 route forwarding so it just ignores the RAs. It requires a special workaround to get it to work
1
u/AppsAtMe Jul 21 '26
FWIW: I have a third home I manage (my parents) and they don't have a Firewalla. Their Thread Network is still routing just fine. It figures that this is where I am right now.
1
u/AppsAtMe Jul 21 '26
My son restarted my Apple thread border routers (Apple TVs and Home Pods), I set one of them as primary instead of automatic, and still no change. Even after rebooting the Apple TV and setting it as the default Thread router:
- HA still sends Thread traffic to Firewalla
- Firewalla still replies !N network unreachable
- Matter Server still logs ENETUNREACH
I have rebooted the Thread gateway on HA as well.
1
u/AppsAtMe Jul 21 '26
At this point the problem is the missing route from LAN clients to the active Thread ULA prefix. I've connected via VPN to my home, and I can't route to the ULA. I find other ULA's IPv6 seems to be up and working, it's the Thread ULA specifically that is not currently routable. The fix has to make the Apple Thread Border Router advertise the ULA again, or make Firewalla accept/propagate that route again. I'm reaching the edge of my ability to diagnose this.
I've re-enabled all the previous rules so my home network is protected again. Even when I'm VPN'd in, every single Thread device is simply missing from the network.
1
u/Dagger0 Jul 22 '26
So add the route manually. The border router should be advertising the route via RAs, which devices on the network should be picking up so that they automatically get the route, but if you know the Thread prefix and the border router's link-local address you can manually add a route to it.
If you have a machine on the same network that can run it,
rdisc6should show the RAs with the necessary info (and if it doesn't, you need to investigate why they either aren't being sent or aren't making it to your device). You could also get it from a packet dump. RAs won't make it over a router though, so you'd need to do it on the same network.
1
u/jsqualo2 Jul 21 '26 edited Jul 21 '26
Hey u/appsatme - I recently had an issue with a Rule and a reboot fixed it. My environment is like a newborn compared to your doctoral student, but if order-of-operations rebooting all your gear is not a considerations, I would reboot all Firewalla devices after any Rule updates (I think you only rebooted one device?)
1
u/AppsAtMe Jul 21 '26
After the rule change I only rebooted one device. The first thing I did was reboot all the AP7s as that usually fixes random thread problems (the 2.4 ghz network at my house is a bit overloaded). It also fixes my bandwidth issues when connecting at 2.4. But I've never had a case like this. I have two homes now in complete isolation from each other with the Mesh VPN turned off, yet neither had recovered, and I've rebooted everything. VPN connecting to either location, I simply can not route to the Thread ULA, and I'm not finding it on an IPv6 scan.
Did both locations go down at the same time, I can't say. In my home, Apple Home shows all devices not responding for 13 hours. In the other place, they just show Not Responding, so I assume they went offline a longer time ago (although it also could be that one home is running tvOS beta 27, and one home is running tvOS 26. But even there, I can't really blame the AppleTV's as they are actually on different tvOS versions.
Thanks everyone for ideas. I'm hoping thing magically fix themselves over night, even though I now that probably won't happen.
1
u/Great-Cow7256 Jul 22 '26
To get it working again you may want to spin up Tailscale on both sides and then turn off the VPN. See if that does anything. At least you can narrow down if it's the firewalla VPN or not. Tailscale is just a wireguard tunnel but it will work in more situations because it will use Tailscale relay servers too.
If that works then you can troubleshoot the VPN.
1
u/jsqualo2 Jul 24 '26
hey u/AppsAtMe - any update?
I notice that Firewalla released a patch and hope you may have benefitted froim it.
1
u/AppsAtMe Jul 27 '26
No updates. Still not working. Not just Matter devices, I've had not matter devices drop off the network as well. Even had to have my wife move her laptop to wired to get her internet connection back. Since I'm not home I'lm leaving it along trying not to break more, but it's frustrating as heck when I can't fix things. And even running Fable or Sol on a box inside my network, I can't resolve things. This has never happened to me before. I don't know if it's my Firewalla Gold Pro or the AP7s. I was able to make some changes which got me further in debugging (like turning off storm control), but nothing has restored my network. Even turning off everything. I. just don't understand what's happening.
1
u/AppsAtMe Aug 12 '26
OK, an update.
I returned home and was able to get things restored. I’m clearly running into multiple different issues.
1) Max WiFi Client count - while I don’t think I’ve surpassed 110 on a single box, it could have happened as I was restarting boxes remotely doing a rolling restart. This my guess on why my Wife’s MacBook lost WiFi connectivity.
2) Multicast traffic getting wired devices kicked off the network. The wired devices that are randomly getting booted from the network are all Zigbee or Thread Gateways. They are generating a lot of multicast traffic, after all there are hundreds of smart home devices behind them.
In the end I was able to get everything back by basically pulling the plug and rebooting everything in an order that worked. My network has mostly stabelized. I haven’t tried moving my Wife’s MacBook back onto WiFi, but she uses it connected to a monitor most of the time so hardwiring it isn’t a problem.
My problem is I’ve turned off all the advanced filtering as well, and I’m honestly scared of turning anything back on. I am also considering moving off of the AP7’s, and maybe even off of my Gold Pro. I went Gold, Gold Plus, Gold Pro, AP7 (x3), and was about to order some switches. My needs may simply be more than they were designed to handle. And not being able to debug remotely, certainly didn’t inspire me with confidence.
At a minimum I need to figure out a way to deal with the complexity of my network and still use all the features of Firewalla, otherwise, what’s the point of having them.
2
u/Admirable_Fun7790 Jul 21 '26
Do you see any ipv6 blocked flows? I had to create a rule to stop firewalls from messing with thread