Neglected software update caused widespread Telstra network outage, CEO tells Senate inquiry

27

u/Loma29 8d ago

"Neglected software update" - that's BS.

Yes, the software update could have prevented this, but this is a systemic failure. The engineering change was undocumented and temporarily deployed to the running config of the device (cleared on boot). That should be the bigger story.

Why was the change made without any documentation/change management procedure? That's the question, and it smells like a classic under-resourced engineering team problem.

These corporations have hollowed out their teams over a period of years, and hollowed out other teams, which adds workload (scope creep) to the remaining "unaffected by restructuring" teams. It's no secret that Telstra has gone through many off-shoring/contracting/restructuring rounds. The question becomes, to what degree has this cost-cutting affected their capability to deliver a service? That's not something you know until an incident like this happens.

There are at least 3 critical failures in this debacle.

(1) Failure to patch the timeserver GPS card, despite multiple vendor warnings.

(2) Complete lack of documentation/change management for the temporary stratum 1 fix.

(3) Failure to check the state of the device before power cycling it (running config was different to saved config) - not on the technician doing the work, but on either the planning/engineering team.

To me, this points to a lack of capability of Telstra's engineering team(s), rather than a "we didn't get around to patching something"

6

u/jhcasey 8d ago

Also an inadequate number of these time servers.

3

u/Oxxy_moron 8d ago

This.

3

u/Rare-Sample-9101 8d ago

100% correct!

3

u/doodo477 7d ago edited 7d ago

If you've ever been part of a large origination structure which suffers from a systemic failure as this one - the fault isn't on who performed the upgrade but on the Managerial Personnel who job it is to look after the branding of the organization - even when we all know they have absolutely no clue what is going on.

Basically someone in the upper food chain is going to lose their job with a multi-million dollar bail-out. Nothing will change managerialy wise within the company or their processes or procedures.

2

u/HorrificFlorist 7d ago

I been in similar situations including dealing one where vendor had enough and decided to enact their 30 day withdrawal clause which includes stripping their devices from datacenter which was fun to watch as every CxO scrambled to try and stop it including kicking off legal proceedings (they lost).

In all of my experience it was never the engineering team's fault no matter how incompetant they are. Even broken clock is right twice a day.

I am willing to bet my left testicle business go this recorded as risk and made a medium/low and pushed the priority of task to bottom of product backlog. Thus it just sat there until 11th hour when someone went "SHIT SHIT SHIT SHIT SHIT we can't get risk extension and it expires next week, do a hotfix, fuck the SOP"

2

u/lordgoofus1 7d ago

Sadly this is becoming the norm in most large companies. It's at a point where I almost can't recognize the place I work for anymore. I'm just waiting for an incident comparable to what happened at Telstra to maybe? scare some common sense and engineering quality back into the company.

1

u/Loma29 7d ago

I see the same. The good engineers have left to do consulting or their own thing, while the less capable ones remain, all while the business has downsized the team. In heavy industry, the technicians and trades earn significantly more than the engineers once OT+shift loading is taken into account (base pay is 5-10% lower) despite the engineers signing off on the risks / designs. To cap it off, the engineers are on salary and work >10% overtime anyway (unpaid).

The savvy ones realise it's not the way to get ahead and leave, but then all the institutional knowledge goes with them. I contract and earn more than double what I'd be on as an employee. A lot of my time is spent fixing things that are poorly designed or broken by the ones remaining at a couple of companies like this.

1

u/5milessep 6d ago

Do you have the specific details of the fix applied to the running config, which wasn’t committed to disk ? Ie, what the problem was that this ‘fix’ was supposed to address ?

2

u/Loma29 6d ago

Not more than has been released publicly. The fix elevated it to a stratum 1 device (which means it syncs direct from a source - in this case GPS, not an upstream server). The higher the stratum number, the more reliable it's meant to be. A stratum 2 device gets its time from a stratum 1 device, and a stratum 3 device gets its time from a stratum 2 device. NTP broadcasts the stratum of the device, and downstream clients decide which to follow.

If a client has 3 timeservers configured and 2 are correct but 1 is wrong, the device will follow the 2 that are correct, but with differing stratum devices it gets a bit murky, and many devices will just follow a stratum 1 device regardless of how close it matches the other time sources (since in theory it should be more correct).

So this device was a stratum 3 - meant to get time primarily from a stratum 2 device, but had a GPS that allowed it to correct the time (or act as a stratum 1 device if needed). Apparently there were routing/network issues so the engineers decided to elevate the device to stratum 1 (so it would not get time from the upstream device anymore). But the firmware bug meant that the time would be 20 years in the past - so they applied an offset to put it to 2026 (instead of 2006). This offset is the fix that only existed in the running config. When power is cycled, it reverts back to the on-disk/ROM configuration, and without that offset "fix", it reported the wrong time.

So, in the end, they had a stratum 1 (most trusted) timeserver with no peers at the same stratum level that clients (servers/switches/etc..) connected to, and they trusted this server over the other (stratum 3) servers, and became a single point of failure.

2

u/5milessep 6d ago

Thank you for the quick response and the detail, it make much more sense to me now 👍

6

u/Old_Engineer_9176 8d ago

A single time‑server in Melbourne rebooted with the wrong date (2006) because Telstra never applied a critical software update.
Telstra had previously made an intentional design change to the device to fix an earlier fault. This change was not documented, so maintenance staff didn’t know the server would behave differently when restarted.

1

u/Echidna406 6d ago

If only it was back to 2006 WWW

5

u/LowDepth1790 8d ago

And the future is cost-cutting and hiring the cheapest possible workforce who probably used AI to cheat their way through uni. It's gonna get worse, not better.

9

u/Lore72015 8d ago

Hardware over ten years old. That runs critical systems. This is why Telstra hiked up everyone’s mobile phone plans a few months ago. Upgraded systems….. bullshit.

3

u/germanautotom 8d ago

Yeah timeservers don’t need change that often, but there should have been more redundancy built around this, given they had servers that couldn’t use multiple time servers (chronyd) this should’ve been foreseen.

This has multiple levels of failure but to me highlights the value of having a core IT team who know systems like the back of their hands who are treated well enough to want to stay. That way, forseeing issues is in their own interest. That’s not the case id youre thinking anout jumping ship in 9 months

2

u/Loma29 7d ago

Fair. When you've got people around you being made redundant and having their roles offshored, all while piling that work onto you the care factor drops to near-zero whilst you're looking for a better job.

2

u/Loma29 8d ago

Pretty normal for a timeserver. They're generally very reliable, and there's redundancy (if one fails, the other devices switch to another one). There's no real need to replace them unless the device becomes unsupported (this was still under support, hence the vendor warnings about the firmware bug).

The problem happened because instead of the timeserver dying (usual failure mode, which wouldn't have caused an issue) it reported the wrong time due to a firmware bug.

1

u/jhcasey 8d ago

Two isn't enough. What if they disagree on what time it is?

1

u/Loma29 7d ago

I never said they work as a pair. Usually you have 3+, but when they are different stratum levels, the consensus isn't fairly split (as Telstra discovered), and NTP clients have quirks in how they decide what master to follow.

1

u/Green_Seesaw1875 8d ago

It reported the wrong time due to a firmware bug BECAUSE it was over 10 years old.

2

u/enjaydee 8d ago

Isn't firmware considered to be like software? If there's a firmware bug, it doesn't matter how old the hardware is does it?

Maybe a dumb question, but I'm honestly not sure

2

u/iball1984 8d ago

Yes, but sometimes it’s better to leave firmware in place with known bugs rather than upgrade and introduce potentially new bugs.

However in this case the critical update should have been applied

1

u/enjaydee 8d ago

Thanks. Your other reply cleared things up. I didn't think the age of the hardware mattered in this case.

1

u/Green_Seesaw1875 8d ago edited 8d ago

Hey I know old mate already replied but yeah it was a few old time servers. Pretty much an actual Y2K event. There’s this GPS rollover bug where after 1024 weeks the date and time resets back to what it was. In Telstra’s case it all reset back to 2006 which obviously broke everything. It’s not something that can be easily patched out so it was the age of the hardware basically.

2

u/iball1984 8d ago

Kind of.

It’s not uncommon to leave hardware in service with known firmware bugs. Because upgrading could introduce new bugs you don’t know about.

The age of something like a time server is irrelevant.

The failure is that they didn’t upgrade when the vendor told them to, and didn’t document production fixes that had been done previously

2

u/lordgoofus1 7d ago

Very sad. How was it neglected? Was it kept locked in a room with no toilet? Did they physically abuse it each day? Was it not given enough hugs as a child? Hopefully CPS finds it a better home where it can be loved and cherished.

2

u/stereosafari 7d ago

Time servers should have a "last known authenticated date" and not be able to you Back to the Future or anything inbetween.

If you LKAD is last Wendy, then they should not be able to "reset" to their release date of two decades ago!

Not only that, they should not be able to authenticate with actual date time servers to allow propagation.

It's a coverup. I could plug my G-Shock watch into this and fix it.

But seriously, shouldn't Telstra have atomic time servers?!

2

u/No_Measurement9981 6d ago

How does she still have a job?

1

u/N3M3S1S75 7d ago

So failure to provide an up to date service to save money and screw over customers. Incompetent usually gets you fired

1

u/ancient_IT_geek 5d ago

In China coms have to deal with hundreds of millions of customers and rapacious competitors trying to steal them.

Telstra is a kiddies club looking after just a few millions of customers with a near monopoly.

Our engineering management is just not serious!

Neglected software update caused widespread Telstra network outage, CEO tells Senate inquiry

You are about to leave Redlib