Oops...

[Infrasonic]

https://arstechnica.com/gadgets/2024/05 ... r-account/

Earlier this month, Google Cloud experienced one of its biggest blunders ever when UniSuper, a $135 billion Australian pension fund, had its Google Cloud account wiped out due to some kind of mistake on Google's end. At the time, UniSuper indicated it had lost everything it had stored with Google, even its backups, and that caused two weeks of downtime for its 647,000 members. There were joint statements from the Google Cloud CEO and UniSuper CEO on the matter, a lot of apologies, and presumably a lot of worried customers who wondered if their retirement fund had disappeared.

In the immediate aftermath, the explanation we got was that "the disruption arose from an unprecedented sequence of events whereby an inadvertent misconfiguration during provisioning of UniSuper’s Private Cloud services ultimately resulted in the deletion of UniSuper’s Private Cloud subscription." Two weeks later, Google Cloud's internal review of the problem is finished, and the company has a blog post up detailing what happened.

Google has a "TL;DR" at the top of the post, and it sounds like a Google employee got an input wrong...

Cont.

[bungeejumper]

You mean there's nobody backing up the backups? That's a reassuring thought

BJ

[Infrasonic]

You mean there's nobody backing up the backups? That's a reassuring thought

BJ

There were multiple backups - including a third party data company that had nothing to do with Google. So from that perspective it was textbook.

The issue was it took two weeks to restore everything, working "24/7", which at that level is totally unacceptable.

Many companies are moving to co-location now, on prem servers + Cloud not just for redundancy + local admin control but also to reduce costs as if you do the numbers after a few years being exclusively cloud costs you more in the long run. The break on it is quite low - only a few TB.

[clissold345]

...
Google has a "TL;DR" at the top of the post, and it sounds like a Google employee got an input wrong...

"TL;DR" = Summary.

[Mike4]

You mean there's nobody backing up the backups? That's a reassuring thought

BJ

There were multiple backups - including a third party data company that had nothing to do with Google. So from that perspective it was textbook.

The issue was it took two weeks to restore everything, working "24/7", which at that level is totally unacceptable.

Many companies are moving to co-location now, on prem servers + Cloud not just for redundancy + local admin control but also to reduce costs as if you do the numbers after a few years being exclusively cloud costs you more in the long run. The break on it is quite low - only a few TB.

I find myself wondering if 10 or 12 of those days were Google first trying out their full repertoire of ways to blame the client!

[Infrasonic]

I find myself wondering if 10 or 12 of those days were Google first trying out their full repertoire of ways to blame the client!

I would imagine that as this was a big pension company they would be required to have a very high level of transparency as well as very strict data sovereignty rules that they have to adhere to. As such I wouldn't be surprised if they told Google 'we'll have to go public on this' at which point Google bumped it up to their Head of Cloud and joint statements were issued.

You can guarantee if you or I were in this situation we would have been lucky to get an acknowledgment email let alone any actual 'help' beyond links to their documentation and DIY fixes...

[stewamax]

Google's post commented "It is not a systemic issue" - the same words used by UK forensic accountants Second Sight in their interim report on the Horizon scandal to the Post Office.