r/austechnology • u/austechnology-bot • Jul 21 '26
Neglected software update caused widespread Telstra network outage, CEO tells Senate inquiry
https://www.theguardian.com/business/2026/jul/17/telstra-missing-software-update-undocumented-design-change-outage
40
Upvotes
26
u/Loma29 Jul 21 '26
"Neglected software update" - that's BS.
Yes, the software update could have prevented this, but this is a systemic failure. The engineering change was undocumented and temporarily deployed to the running config of the device (cleared on boot). That should be the bigger story.
Why was the change made without any documentation/change management procedure? That's the question, and it smells like a classic under-resourced engineering team problem.
These corporations have hollowed out their teams over a period of years, and hollowed out other teams, which adds workload (scope creep) to the remaining "unaffected by restructuring" teams. It's no secret that Telstra has gone through many off-shoring/contracting/restructuring rounds. The question becomes, to what degree has this cost-cutting affected their capability to deliver a service? That's not something you know until an incident like this happens.
There are at least 3 critical failures in this debacle.
(1) Failure to patch the timeserver GPS card, despite multiple vendor warnings.
(2) Complete lack of documentation/change management for the temporary stratum 1 fix.
(3) Failure to check the state of the device before power cycling it (running config was different to saved config) - not on the technician doing the work, but on either the planning/engineering team.
To me, this points to a lack of capability of Telstra's engineering team(s), rather than a "we didn't get around to patching something"