一次“穿越”回2006年的故障:Telstra全国电话网是如何崩溃的
John Howard was still the prime minister, the first iPhone was months away, and Facebook had only just become available to the world. When Telstra’s mobile network fell over on Wednesday morning, it was because a software defect was telling the network it was back in November 2006.
Telstra’s chief financial officer, Michael Ackland, initially explained that the software defect hit the company’s GPS node, which tells the organisation’s systems what the most accurate time is to the nanosecond.
The reset changed the time and synchronisation, which was then passed on to the rest of the Telstra network. On Thursday, Ackland confirmed that the update pushed out to the rest of the network dialled back the clock internally to November 2006.
Why did this cause the outage – and how were people affected?
Sign up for the Breaking News Australia email (https://www.theguardian.com/email-newsletters?CMP=copyembed&CMP=emailbutton)
•
How a time-telling error brought down network
Unlike a grandfather clock, or an old-school alarm clock, people generally don’t need to adjust their smartphones when daylight savings kicks in.
That is in part because of the network time protocol (NTP). A system can send an NTP request to receive accurate time information from a server dedicated to telling the time.
Phones then use this accurate time to convert to local time based on location.
On Wednesday, the NTP within Telstra’s GPS node was suddenly telling the rest of the network it was nearly 20 years ago.
Networks rely on accurate time information for synchronising and authenticating, Ackland explained, “so that one part of the network knows what the other part is doing and the time is an important part of that”.
“Otherwise you could have messages or instructions that are from a different time period being authenticated on the network,” he said.
Telstra CFO Michael Ackland rejected comparisons to the Y2K bug. Photograph: Christopher Hopkins/AAP
Khurram Shahzad, a senior analyst at the research and advisory firm Gartner, said asynchronous time could have a wide range of effects on mobile networks.
“When the root clock fails, the entire digital domino chain falls within minutes,” he said.
Telstra has yet to detail how the dominoes fell, but Shahzad said network nodes from cell towers authenticate their identity with the central data hub using security certificates that have strict validity windows that are checked against the current time to ensure they haven’t expired or been compromised.
“If Telstra’s central time servers began broadcasting the wrong time – or stopped broadcasting entirely – core network switches suddenly saw valid security certificates as expired or invalid,” Shahzad said.
“The system assumed it was under a security threat and automatically terminated the connections, locking devices out of the network and throwing phones into ‘SOS only’ mode.”
He said 4G and 5G also need time measures to maximise speeds.
Ackland rejected comparisons to the Y2K bug (https://www.theguardian.com/commentisfree/2019/dec/31/millennium-bug-face-fears-y2k-it-systems), caused by systems not being able to deal with the date format as the year clicked over to 2000, instead describing it as a software glitch. He said Telstra would do a root cause analysis, working with vendors, partners and external support to work out what exactly happened.
•
Triple zero
While Telstra’s network was down, some customers who attempted to dial triple zero could not be connected (https://www.theguardian.com/business/2026/jul/09/telstra-outage-triple-zero-calls-angus-taylor-sarah-henderson), despite failsafe systems that require a customer to be connected to emergency services through one of the two other mobile network providers in the event of a network failure.
Telstra has yet to explain why this handover did not take place, but on Thursday said it had resolved the issue.
A second issue identified on Wednesday night, which Ackland said the company believed was driven by the same time-keeping issue, caused some customers to receive an error message when attempting to call other numbers or triple zero. This had been resolved by Thursday morning.
Telstra said as of 1pm AEST on Thursday, there had been 639 welfare checks completed on people who had made unsuccessful calls to triple zero. Of those, 230 advised they didn’t need assistance. The company said 402 cases required a follow-up call, with 170 of those referred to police for a welfare check, and seven referred to emergency service organisations.
•
Why did some trains stay offline longer?
The Australia Rail Track Corporation was forced to suspend services on Wednesday after the Telstra outage disrupted the national train communications system that allows drivers to communicate directly with the network control centre over Telstra’s 4G.
ARTC owns infrastructure for rail services across the country, including for the Ghan between Adelaide and Darwin, XPT services from Sydney, V/Line in Victoria, services in Queensland, and the Indian Pacific.
V/Line indicated Telstra’s 4G network had interfered with the satellite backup system used by train drivers to communicate with the network control centre.
“Overnight, Telstra were able to undertake some repair programming to stabilise the network, and that involved making sure that the 4G train radio system was talking correctly and integrated correctly with the satellite phones, which are our backup system in the train,” the chief executive of V/Line, William Tieppo, said on Thursday morning.
Ackland said Telstra was working with all affected businesses.
“While the broader outage was resolved yesterday, we continue to work closely with a small number of enterprise customers on residual flow-on impacts in their services,” he said.
“We will keep working with our customers until every remaining issue is resolved and as they complete all restoration and verification processes that they need to complete.”
Given one glitch had taken out the mobile network, Ackland was asked if that showed Telstra’s network was fragile.
Ackland said it was not. “I would say it is a complex system. I think it is robust,” he said.
“There is a lot of redundancy, but this was an unfortunate incident.”
At this dangerous time
We hope you appreciated this article. Before you close this tab, we want to ask if you could support the Guardian at this dangerous time for journalism in the US.
According to a leading global watchdog, American democracy is now more imperiled than at any point since the 1960s, marked by a precipitous decline in press freedom – driven by mounting pressure from the Trump administration in the form of threats, criminal investigations, politicized regulation, frivolous lawsuits and, for public media, catastrophic funding cuts.
Meanwhile, organizations that are supposed to be independent like the FBI and the FCC, our radio and television regulator, have also been targeting press freedom under Trump-aligned leadership, with the FBI raiding a reporter’s home and the FCC threatening ABC’s TV licenses after Jimmy Kimmel made a joke about Melania Trump.
The response from some ultra-wealthy and corporate media owners, keen to appease the president, has been chilling: CBS News has been taken over by a Trump ally; CNN is poised to be taken over by the same billionaire; Jeff Bezos has continued to impose cuts and editorial interventions at the Washington Post; and multiple outlets have settled multimillion-dollar lawsuits from the administration to protect their business interests.
Democracy is best served by a robust, thriving free press. But when that freedom is under attack, it falls to a determined few news organizations to ensure the full truth still reaches the public. Owned neither by a billionaire nor a corporation, the Guardian remains dedicated to covering this administration with uncompromising moral and factual clarity – and to keeping trustworthy journalism paywall-free for the world.
• *Despite the risks of maintaining our fierce independence, what sustains us – and fills us with deep gratitude – is the unwavering support we’ve seen from readers. It is no exaggeration to say that we are here because of you: a majority of our funding comes directly from people like you responding to messages like this. Your support not only powers our work, but more importantly, it safeguards the financial independence that underpins our editorial freedom and courage.**
We know our requests for support are not as welcome as our reporting, but without them, it’s simple: our reporting wouldn’t exist. Of course, we understand that some readers are not in a position to support us, and if that is you, we value your readership no less.
• *But if you are able, please support us today. All gifts are gratefully received, but a recurring contribution is most impactful, helping sustain our work throughout the year ahead (and among the great benefits, we’ll show you far fewer fundraising requests like this). It takes just 37 seconds to give. Thank you for protecting the free press.**
Support $1.15/week
Back independent journalism on a monthly basis.
Recommended
Support $3.45/week
UnlockAll-access digitalbenefits:
• Far fewer asks for support
• Ad-free reading on all your devices
• Unlimited access to the premium Guardian app
• Regular dispatches from the newsroom to see the impact of your support
• Unlimited access to Feast, the Guardian recipe app
Support once from just $1
We welcome support of any size, any time - whether you choose to give $1 or more