Human made summary: A telecom company had time synchronisation issues in a location of their infrastructure. They eventually used a single hardware GPS clock that fixed the issues but became their only authoritative clock in this location.

One day, the clock was turn off and on and it went 1024 weeks backwards because the time GPS time protocol sucks and use a week counter with too few bits.

Apparently a GPS clock can keep track of the time if it’s up and running when the week counter overflows, otherwise it has to take a wild guess. Apparently their hardware GPS clock used a hardcoded start time from its firmware instead of trusting a less reliable existing clock.

The article finishes with some AI looking suggestions to prevent such an issue to happen again.

Mine would be to have bought one or two more GPS clocks and not from the same provider.

Lots of bloviating AI-prose. Here's the original report:

https://www.telstra.com.au/exchange/what-we-ve-learned-from-...

https://www.telstra.com.au/content/dam/tcom/dynamic-media-pr...

Coincidentally just bought myself a stratum 1 ntp/gps "clock" few weeks ago for £60.

Runs in my loft, what a time to be alive.

https://lectronz.com/products/gps-ntp-server

> Each decision solved the problem in front of it. Nobody was asked to look at the sum of all actions.

This is the perfect description of Telstra as a company.

Direct link to the analysis report: https://www.telstra.com.au/content/dam/tcom/dynamic-media-pr...

Author needs to decide which end of the stratum stack they want as the top.

> This design from 2010 had at the top stratum 1 sources at Australia's National Measurement Institute (NMI)

> Among otherwise comparable candidates, the lower stratum carries more weight.

ai;dr

All I could think of while reading this was “has no one heard of ntptrace?”

Is it really so that Time Division Multiplex timing in cellular networks is based on NTP time? I stopped reading after the article implied this - it doesn't sound plausible at all.

Shout out to the Jeff Geerling video about this

It's funny, but I had the same thing happen to me once. One day, overnight, my computer decided it was 2006.

It happened twenty years ago...

;-)