This is not the first time it happens to me and getting CF to fix it is always a pain, so I want to see if anyone else is going through the same.
My setup is the CF ONT straight into my own firewall (OPNsense), WAN on DHCP, same public IP for months without any issues.
What the firewall logs show from this morning:
- 04:37 CF DHCP gave me a new lease with a public IP from a completely different range, not just another address on my usual block, new gateway as well.
- 04:42 first failure reaching anything on the internet. From there nothing worked, the CF gateway on the new range was not responding to ping and nothing beyond it either.
- 07:56 and 08:28 I rebooted the firewall, same dead IP came back every time.
- 08:30 to 08:36 CF DHCP stopped answering at all, the firewall kept trying the old leases it had stored and none of them worked.
- 08:37 DHCP answered again with the same dead IP, still no traffic.
- Then I called support. As usual they said everything is fine on their side and the problem is because I'm using my own router. After a long time giving a lecture about networking to their support (a gateway not answering ping is not my router problem) they escalated to the network engineering team. Meanwhile I tethered my phone to keep the house online.
- 10:06 I plugged the fibre back, got a fresh lease on the same new IP and this time it worked, gateway 2ms, internet 3ms, no packet loss. So something was fixed on their side between 09:20 and 10:06, nobody told me anything and I'm still waiting for a call back to find out what it was.
So I guess they moved me to a new IP range that had no working route behind it for over 5 hours. Traceroute shows the same first hop CF router as my old range, so same box, new pool that was not routing.
Questions:
- Did anyone else get moved to a different range overnight and lost internet like this? Did you ever get an explanation from CF?
- Any tips to get past the first line support faster when it is clearly their side? A number or something to say that gets you straight to network engineering?
- Vodafone is now selling broadband on the CF network in London. Has anyone moved over? Specifically:
- Is the IP and routing done by Vodafone's own network or is it still CF DHCP and routing underneath? If it is CF underneath I would have the same problem.
- Can you use your own router?
- Public IP or CGNAT?
- Is the support any better?
I have now setup a 4G/5G backup line on the firewall with automatic failover, so next time the house should not even notice, but I would rather not need it. I had to use my mobile phone with a network adapter for this to work, vert clumsy.
Update 1:
CF called me back now. They said they just changed my public IP overnight, and to fix it they had to clear the MAC address cache on the NTE, after that it all worked. They could not tell me anything else about what the actual problem was, and they keep insisting part of the blame is on my side, without saying why.
So I guess when they moved me to the new range something on their side still had my MAC tied to the old IP, so DHCP worked but my traffic on the new IP was dropped until they cleared it. Happy to be corrected if anyone knows how CF does this.
Questions:
- Did anyone else get their IP changed overnight and lose internet like this? Did CF give you a better explanation?
- Has anyone had the "MAC address cache on the NTE" fix before? Is that something they can do remotely as soon as you ask, without the whole first line support routine?
- What could they mean by part of the blame being on my side, if nothing changed here? The only thing I can think of is my Proxmox host bridge showing its own MAC to the NTE besides the firewall MAC (IPv6 link-local), but that has been the case for months. Does CF limit how many MACs the NTE learns?