SoylentNews is people
SoylentNews
Buried under the news from Google I/O this week is one of Google Cloud's biggest blunders ever: Google's Amazon Web Services competitor accidentally deleted a giant customer account for no reason. UniSuper, an Australian pension fund that manages $135 billion worth of funds and has 647,000 members, had its entire account wiped out at Google Cloud, including all its backups that were stored on the service. UniSuper thankfully had some backups with a different provider and was able to recover its data, but according to UniSuper's incident log, downtime started May 2, and a full restoration of services didn't happen until May 15.
UniSuper's website is now full of must-read admin nightmare fuel about how this all happened. First is a wild page posted on May 8 titled "A joint statement from UniSuper CEO Peter Chun, and Google Cloud CEO, Thomas Kurian." This statement reads, "Google Cloud CEO, Thomas Kurian has confirmed that the disruption arose from an unprecedented sequence of events whereby an inadvertent misconfiguration during provisioning of UniSuper's Private Cloud services ultimately resulted in the deletion of UniSuper's Private Cloud subscription. This is an isolated, 'one-of-a-kind occurrence' that has never before occurred with any of Google Cloud's clients globally. This should not have happened. Google Cloud has identified the events that led to this disruption and taken measures to ensure this does not happen again."
[...] A June 2023 press release touted UniSuper's big cloud migration to Google, with Sam Cooper, UniSuper's Head of Architecture, saying, "With Google Cloud VMware Engine, migrating to the cloud is streamlined and extremely easy. It's all about efficiencies that help us deliver highly competitive fees for our members."
[...] The second must-read document in this whole saga is the outage update page, which contains 12 statements as the cloud devs worked through this catastrophe. The first update is May 2 with the ominous statement, "You may be aware of a service disruption affecting UniSuper's systems." UniSuper immediately seemed to have the problem nailed down, saying, "The issue originated from one of our third-party service providers, and we're actively partnering with them to resolve this." On May 3, Google Cloud publicly entered the picture with a joint statement from UniSuper and Google Cloud saying that the outage was not the result of a cyberattack.
[...] The joint statement and the outage updates are still not a technical post-mortem of what happened, and it's unclear if we'll get one. Google PR confirmed in multiple places it signed off on the statement, but a great breakdown from software developer Daniel Compton points out that the statement is not just vague, it's also full of terminology that doesn't align with Google Cloud products. The imprecise language makes it seem like the statement was written entirely by UniSuper. It would be nice to see a real breakdown of what happened from Google Cloud's perspective, especially when other current or potential customers are going to keep a watchful eye on how Google handles the fallout from this.
Anyway, don't put all your eggs in one cloud basket.
(Score: 5, Insightful) by SomeGuy on Wednesday May 22 2024, @06:00PM (2 children)
I'm honestly surprised this sort of thing does not happen more often. It seems like everything is falling apart these days, nothing really works any more, and nobody wants to hire anyone to make sure things keep running - just half-assed patch up whatever little problem and begone.
Don't put all your eggs in one cloud basket? As far as anyone in charge is ever concerned, there is only ONE basket, and that IS the cloud, and everything must go in there, because it makes their dicks look big. No need to worry about where anything is located, the cloud takes care of that. No need to concern yourself with how much resources are needed, it's all unicorn magic in teh cloudz!.
(Score: 1, Insightful) by Anonymous Coward on Wednesday May 22 2024, @06:48PM
> It seems like everything is falling apart these days, nothing really works any more, ....
Wild ass guess: SomeGuy is younger than 40, perhaps quite a bit younger?
Looking from my late 60s, it's pretty clear that everything has been falling apart and only patched back together--for most of human history. With 24/7 internet nooze, you might hear about more examples these days?
(Score: 4, Touché) by stormreaver on Wednesday May 22 2024, @09:04PM
It doesn't. This is the only time anything like this has ever happened, and even then I'm supremely confident that it is nothing more than mass hysteria. After all, we were assured that cloud providers are way more qualified to handle our business than we are, have way more people to ensure our data is safe, have redundant backups that we could only dream of, etc.
Nothing to see here, citizen. Move along.