Buried under the news from Google I/O this week is one of Google Cloud's biggest blunders ever: Google's Amazon Web Services competitor accidentally deleted a giant customer account for no reason. UniSuper, an Australian pension fund that manages $135 billion worth of funds and has 647,000 members, had its entire account wiped out at Google Cloud, including all its backups that were stored on the service. UniSuper thankfully had some backups with a different provider and was able to recover its data, but according to UniSuper's incident log, downtime started May 2, and a full restoration of services didn't happen until May 15.
UniSuper's website is now full of must-read admin nightmare fuel about how this all happened. First is a wild page posted on May 8 titled "A joint statement from UniSuper CEO Peter Chun, and Google Cloud CEO, Thomas Kurian." This statement reads, "Google Cloud CEO, Thomas Kurian has confirmed that the disruption arose from an unprecedented sequence of events whereby an inadvertent misconfiguration during provisioning of UniSuper's Private Cloud services ultimately resulted in the deletion of UniSuper's Private Cloud subscription. This is an isolated, 'one-of-a-kind occurrence' that has never before occurred with any of Google Cloud's clients globally. This should not have happened. Google Cloud has identified the events that led to this disruption and taken measures to ensure this does not happen again."
[...] A June 2023 press release touted UniSuper's big cloud migration to Google, with Sam Cooper, UniSuper's Head of Architecture, saying, "With Google Cloud VMware Engine, migrating to the cloud is streamlined and extremely easy. It's all about efficiencies that help us deliver highly competitive fees for our members."
[...] The second must-read document in this whole saga is the outage update page, which contains 12 statements as the cloud devs worked through this catastrophe. The first update is May 2 with the ominous statement, "You may be aware of a service disruption affecting UniSuper's systems." UniSuper immediately seemed to have the problem nailed down, saying, "The issue originated from one of our third-party service providers, and we're actively partnering with them to resolve this." On May 3, Google Cloud publicly entered the picture with a joint statement from UniSuper and Google Cloud saying that the outage was not the result of a cyberattack.
[...] The joint statement and the outage updates are still not a technical post-mortem of what happened, and it's unclear if we'll get one. Google PR confirmed in multiple places it signed off on the statement, but a great breakdown from software developer Daniel Compton points out that the statement is not just vague, it's also full of terminology that doesn't align with Google Cloud products. The imprecise language makes it seem like the statement was written entirely by UniSuper. It would be nice to see a real breakdown of what happened from Google Cloud's perspective, especially when other current or potential customers are going to keep a watchful eye on how Google handles the fallout from this.
Anyway, don't put all your eggs in one cloud basket.
(Score: 2) by Joe Desertrat on Friday May 24 2024, @01:55AM (3 children)
There's no reason to print this stuff on paper. You can easily download statements from your bank (if you can't, get another bank). Of course, you will have to back this up yourself, but that is simple enough these days, and no matter how much space it takes on your hard drive, external drive and/or disks, it will still take up far less physical space than piles of paper will.
(Score: 2) by Nuke on Friday May 24 2024, @09:35AM (2 children)
Having a paper statement from the bank is forensic proof that you had that that money in the bank at least at that point in time. Of course it could be a few weeks old, except that most of my money is in longer term bonds or accounts that need some notice of withdrawal, so the statement will show the amount up to the present date.
This news item itself demonstrates a reason, one of several. Another is being able to annotate statements yourself.
How do you download statements from your bank if they have lost all their data in a cloud, like this pension fund did?
Bank statements don't take up much space, although it could be an issue if you live in a broom cupboard but I don't. I have far more space being taken up by books, bikes, furniture, PCs, mementos, tools, food, pet equipment, boxes of USB leads in every combination, tins of paint waiting for me to get round to doing some decorating, boxed of ceramic tiles ditto, I could go on and on. A 10mm thick stack of A4 bank statements in a pocket file is the last thing I worry about finding space for.
(Score: 2) by Joe Desertrat on Saturday June 01 2024, @12:33AM (1 child)
How do you get paper statements from them if they have lost all their data? You can download your statement each month (or however often you like), usually well before any paper statements have gone out in the mail. Save it on YOUR computer, with backups on reliable media of your choosing, and you will never have to worry.
(Score: 2) by Nuke on Saturday June 01 2024, @09:20AM
I would have all the paper statements* from before they lost all data, up to two weeks ago as I write this. Of course if they lost all data today they could claim I had withdrawn and spent everything in the last two weeks, like I was hiding a new car or yacht somewhere, but those statements are better than nothing and most of my money is in long-term bonds, or limited withdrawal saving accounts, anyway. With my having paper statements the only account the bank could argue with me about would be my current account (= "checking account" in USA?) which never has a large amount in it for very long.
I do that anyway.
Not forensically valid. Digital data can be tampered with and often is**. That's why you cannot open a bank account (in the UK anyway) without showing proof of residence etc on paper, digital version unacceptable - ironically by the very same banks that nag you to go paperless.
* Actually I keep most of them for up to 5 years ago, and the annual summaries for ever.
** As in the current UK Post Office Horizon scandal.