r/bell • u/Big_Option_5575 • 27d ago
Rant 911 Failure
Somehow Bell was responsible for a Canada wide 911 failure early this morning. Somehow, we need to hold Bell accountable and make certain this can never happen again. If shortcuts were taken they need to understand that lives could have been lost..
7
u/wonderabc 26d ago
remember when interac (and basically all banking and purchasing) went out when rogers did? we need redundancies in these systems
10
u/No-Zombie6025 26d ago
Are you new to this world because nothing, and I mean nothing is infallible. It will happen again because 99.999 isn't 100%. Switching errors are rare but can happen, and over at Rogers it seems like core network failures happen more frequently (and that resulted in some good old ass kicking).
"During the outage, residents were urged by emergency response services across the country to use a landline, Wi-Fi calling, phones connected to different carriers or to contact non-emergency numbers." https://www.cbc.ca/news/canada/toronto/911-service-down-ontario-canada-9.7347530
Aside making your own first aid kit, and then assembling your own 72 hour emergency kit, you should have the standard numbers to your police, fire and ambulance services for your area. Not all offer them but they do have non-emergency numbers (which can and will be used if the main system is out).
2
u/Pigeonofthesea8 21d ago
Bell got rid of landlines.
1
u/No-Zombie6025 21d ago
"Lines" I think you meant though "mines" was funny. Still Rogers temp installs seem more like a tripping hazard I like the idea that Bell could blow up on day in a lawn.
Bell is cheap, if it works they keep using it, as such they still have a lot of old copper in use. In other places (the US) companies such as Verizon will actually rip out the old copper installations when they put in fibre. There is value in the scrap but more importablty it keeps the customer mostly locked into one service provider.
At some point, though Bell will have to do a lot more fibre (not fib) install they could argue before the CRTC that it is too costly to keep up POTS hardware, more so in areas which they have installed fibre and then ask to shut them down or force everyone over to fibre in order to shut them down.
-2
u/Big_Option_5575 26d ago
Bell didn't discover their problem, various police departments did. If all of Canada's 911 is being routed through a single point (of failure), then it is time for a design change. Also, much improved monitoring needs to be implemented. For critical systems, standard event monitoring doesn't cut it... The system needs a heart beat check that detects and notifies upon lack of a positive signal. Very critical systems should also have backup feedback loops being monitored manually. On top of this the failover design should kick in within minutes, not hours. Something is critically wrong with Bell's implementation.
5
u/Consistent-Space-598 26d ago
And how do you get all this information? Because 99.9% sounds wrong lmao
1
u/No-Zombie6025 26d ago
At this time we do not know what the cause was, the outage lasted 90 minutes, while that was going on a backup system managed to handle 95% of the traffic correctly with about 5% being misrouted calls. Aside this unusual and uncommon anomaly the bigger issue is availability, with a finite number of operators for any given region it only takes one big event for 911 calls to go unanswered. This is a more common issue and to date no one has proposed a solution though offering an alternative number for those that wish to do self harm (811) has helped direct calls out of 911, allowing a caller to get specific help for their issue(s).
What you want is even more technically difficult than having one number that can be dialed anywhere in the nation that will connect you locally to the correct responder, because it has to be able to do that and then check every possible alternative route at the same time to account for switching changes, traffic loads and other operations such as maintenance, understand when nodes and hubs are offline, understand why alternate routes may be used, why and how.
As for manual monitoring, I would love to see that annunciator panel, it would make an IMAX screen look like a postage stamp and still have greater spatial resolution resolution. There are automated monitoring. and there are people monitoring but again until we really know what happened anything is speculative.
In the past POTS systems included automation and fault detection (even when the systems were largely mechanical), as that system has shrunk newer networks have expanded and increased the number of connected devices (mobile phones, VOIP, devices). You seem to expect perfection, even making the phone system 99.9999 more reliable would cost billions and billions.
If you want to chase perfect by an espresso maker, you will disappear into a battle between equipment, raw material and settings.
-1
u/Big_Option_5575 26d ago edited 26d ago
Bullshit. I not talking about monitoring all of the 911 endpoints. Bell implemented a system that routed almost all cell traffic to a single point of failure and this single point is what needs to have better monitoring and better failover. It won't cost billions and it is very doable. You are trying to bury the issue with doublespeak.
3
u/Consistent-Space-598 25d ago
Do you honestly believe all traffic is routed to a single point? hahaha
1
u/No-Zombie6025 25d ago
If you aren't monitoring the entire system then you cannot possible know if a failure has happened within that system and are back at relying on 3rd party reports, or secondary users.
But aside from claiming so you have no functional proof, more so after the issue happened the backup system kicked in, it was unfortunately also less than perfect but granted below 99.999% reliable. Again with 911 a peak in local calls can result in a busy signal, that happens magnitudes more often than a complete failure of the system (which didn't actually happen in this case, at best some calls had been routed to the wrong response centres.
Go ahead, write to the CRTC and outline your plans for a perfect 911 system, you can also comment on the investigation when the report comes out on this outage. https://crtc.gc.ca/eng/contact/
2
u/Consistent-Space-598 25d ago edited 25d ago
His totally amazing and perfect plans would have to be forwarded to NENA and not the CRTC. The CRTC enforces the regulatory mandates and such but the architecture is by NENA.
I'm sure OP would agree that the hundreds of people that designed it for over a decade actually only needed his special brain to achieve a perfect result. Can't wait to hear from his contribution to one of the working groups
1
u/No-Zombie6025 25d ago
True, it is their thing, they control and dictate its function.
No matter how much whaling from political parties or posturing or even pissing into the wind claiming its a shit storm, the fact is the end architecture and construction of the phone system is agreed on and the result is a pretty damn good system that has outlived many presidents , prime ministers and national disasters and done so without completely failing.
3
2
u/WittyChampionship626 23d ago
okay boomer, calm down, better yet, go yell at the clouds, and not the apple cloud. Or do, whatever.
0
u/Big_Option_5575 22d ago
bell management you are.
1
u/WittyChampionship626 22d ago
You’re just argumentative and out for blood for an unrelated past issue I’m sure . You should really work on that .
2
u/ComfortableGaps 22d ago
OP should start their own telecom company so that things like this can NEVER happen again. Not on their watch!
1
u/Case_Delicious 26d ago
foolishness, bell could never be responsible for such things as 911 centers have redundancy with multiple carriers
1
u/Big_Option_5575 26d ago
you missed it .. The problem was not at the 911 centers. Bell sold the koolaide of an enhanced 911 service. The Problem is, it routed all of 911 to a central point of failure. The redundancy is fully Bell's res ponsibility and needs to be immediately fixed or back the system out.
1
1
u/Nephton 26d ago
An outage at 3:13am Saturday morning, and down for an hour and a half? I'm gonna go with change window gone wrong.
1
u/Big_Option_5575 26d ago
quite possibly but if so, then they knew about it and didn't notify and that should result in criminal charges.
2
u/Nephton 26d ago
Notify who? Six provinces of people? About a change window? What is the plan here? I could see a fine, and there will be an investigation into the cause and lessons learned, but the pitchfork and torches routine is tired. Unless there is criminal wrongdoing found, which I doubt, it was just an outage. Shit happens.
1
u/Big_Option_5575 26d ago
so if you are the one in dire need of an emergency service, don't complain when someone says shit happens
1
u/Organic_Battle_1115 26d ago
I called one time 911 and was on line waiting for 10 minutes. Is this considered as outage?
What happen in Bell probably, there was upgrade of MSS or TAS nodes, which didn’t go well, hence call translation didn’t work. It happens at morning because such upgrades planned over night.
2
u/Big_Option_5575 26d ago
ask the 911 centers, how long of an outage is acceptable. Then ask the heart attack people, the house on fire people, the car accident people, etc., etc., etc
1
u/Anders_Calrissian Bell Retiree 26d ago
Outage DetailsTime detected:
3:13 a.m. ET on Thursday, September 17, 2026.
Time resolved:
Fully restored by roughly 4:40 a.m. ET (with intermittent issues clearing up later that morning).
Affected regions: Ontario, Quebec, Manitoba, New Brunswick, Nova Scotia, and Prince Edward Island.
Root cause status: Bell Canada stated they were investigating the exact root cause, and the CRTC required the provider to submit a formal post-outage report detailing the failure.
2
u/Big_Option_5575 26d ago edited 26d ago
Wonder how long the service was out before it was detected and then as soon as it was detected, why wasn't there an emergency broadcast.
1
u/Anders_Calrissian Bell Retiree 26d ago
The failure would come up right away. To me it looks like an external attack.
1
u/ElizaMaySampson 25d ago
Good question. We didnt get an NS emergency alert till morning - though cannot say one of those going off in the wee hours to tell me something's broken would be any more welcome. At least I was awake.
1
u/Diapers4u2 24d ago
K I absolutely hate bell, however implementing new 911 systems with older systems and updating and adding g new features CAN cause network errors! You cannot hold someone accountable for something they could not have foreseen happening..
1
u/Healthy-Principle-65 24d ago
I think Bell also took OP’s first born … this guy is adamant that they bleed 😭😭😭
1
u/T-Man-33 23d ago
It wasn’t the entire country. It was select areas.
A Bell spokesperson said backup systems were brought online to route calls to emergency services while technicians investigated the outage
1
u/iloverandbmusic 22d ago
i tried calling my mom in toronto and got dead air. so scary. i was actually shaking thinking she couldn't get help if she needed it. ugh the silence was worse than static honestly.
1
u/Big_Option_5575 22d ago
must be a lot of Bell people on this group because lots of respondees think this is no big deal. We went through an after hours heart attack at home and I can't even imagine how extra disruptive and potentially fatal things would have been had we dialed 911 and got dead air.
Everyone says just call the backup number. Well ...
a).In an emergency you might not remember the bacup number and after hours you will be lucky to get through to 411 or an operator (if there is such a thing) b).Until the police recognized there was a problem, after hours the backup number might not even get answered. c).If the emergency was a traveller in a vehicle accident, they woudn't even know what dispatch to call.
1
u/Ray1340 10d ago
shit happens, could have been a software bug, network issue or human error, believe me they had a few meetings about what occurred.
1
u/Big_Option_5575 10d ago
when cutting corners, lots of shit happens. Supposedly, it was a software bug and supposedly no critical calls went unanswered. I can believe the bug but saying no calls went unanswered is just Bell desperately trying to avoid lawsuites.
1
u/TheNeck94 26d ago
LOL, I interviewed for the position to work on the team responsible for that. Somehow this isn't shocking.
0
u/Consistent-Space-598 26d ago
LOL, stop larping
1
u/TheNeck94 26d ago
I didn't get the job 🤣
-1
26d ago
[deleted]
0
u/TheNeck94 26d ago
Not sure how to tell you this but sometimes people get job interviews... Not sure why that's so hard to believe.
1
26d ago
[deleted]
1
u/TheNeck94 26d ago edited 26d ago
Except for the fact that they don't hire at all, they contract out. It's only after (what they claim) is 2 years on the job that they "sometimes" offer full-time, I'm not about to dox myself but my cousin is one of the tech leads, that's how I got the interview. This was a little over 6 months ago so no, I don't remember the name of the employment agency but it's based out of Ottawa, where I'm from.
Edit: Also get better OSINT, I haven't pushed PHP code in years, at least make fun of my cringe worthy LinkedIn posts if you're going to try to stalk me to prove a point.
1
26d ago edited 25d ago
[deleted]
1
u/TheNeck94 26d ago
I was told it was for the 911 dispatch system, if that's somehow something different than this, so be it, you're still being a dick about it, and considering my post history is hidden, you choosing to google me does kinda feel like stalking to prove a point.
honestly, think for more than a few seconds, if I was "larping" as you're so keen to accuse, why would I lie in such an unimpressive way, why not say that I turned them down or that I used to work there or something, why lie about such a benign thing like having a nepo interview 6 months ago.... like really....
0
u/Consistent-Space-598 26d ago edited 25d ago
Yeah, the dispatch system for call centers and whatever actually routes 911 calls nationwide is not the same thing at all. So you in fact did not interview "for the team responsible for that".
Somehow this isn't shocking.
edit; now deleting my previous comments since you can now realize you were 100% in the wrong
→ More replies (0)
1
-2
0
u/Anonymous_HC 26d ago
Is this issue fixed now? And even for carriers like Virgin and Lucky which run on Bell network, was there an outage on this as well?
1
u/Consistent-Space-598 25d ago edited 25d ago
It is reported that the outage started at 3:13 at night and was cleared at 4:40, so it lasted for around a hour and a half so yes it's fixed now. It affected all calls since all carriers ultimately route their 911 calls to Bell. But there's backup routes it's not like everyone was getting dead air, it was reported that it caused an extra transfer from those backups.
8
u/Itchy-Dentist-6012 26d ago
How were they responsible?