SoylentNews is people
SoylentNews
Buried under the news from Google I/O this week is one of Google Cloud's biggest blunders ever: Google's Amazon Web Services competitor accidentally deleted a giant customer account for no reason. UniSuper, an Australian pension fund that manages $135 billion worth of funds and has 647,000 members, had its entire account wiped out at Google Cloud, including all its backups that were stored on the service. UniSuper thankfully had some backups with a different provider and was able to recover its data, but according to UniSuper's incident log, downtime started May 2, and a full restoration of services didn't happen until May 15.
UniSuper's website is now full of must-read admin nightmare fuel about how this all happened. First is a wild page posted on May 8 titled "A joint statement from UniSuper CEO Peter Chun, and Google Cloud CEO, Thomas Kurian." This statement reads, "Google Cloud CEO, Thomas Kurian has confirmed that the disruption arose from an unprecedented sequence of events whereby an inadvertent misconfiguration during provisioning of UniSuper's Private Cloud services ultimately resulted in the deletion of UniSuper's Private Cloud subscription. This is an isolated, 'one-of-a-kind occurrence' that has never before occurred with any of Google Cloud's clients globally. This should not have happened. Google Cloud has identified the events that led to this disruption and taken measures to ensure this does not happen again."
[...] A June 2023 press release touted UniSuper's big cloud migration to Google, with Sam Cooper, UniSuper's Head of Architecture, saying, "With Google Cloud VMware Engine, migrating to the cloud is streamlined and extremely easy. It's all about efficiencies that help us deliver highly competitive fees for our members."
[...] The second must-read document in this whole saga is the outage update page, which contains 12 statements as the cloud devs worked through this catastrophe. The first update is May 2 with the ominous statement, "You may be aware of a service disruption affecting UniSuper's systems." UniSuper immediately seemed to have the problem nailed down, saying, "The issue originated from one of our third-party service providers, and we're actively partnering with them to resolve this." On May 3, Google Cloud publicly entered the picture with a joint statement from UniSuper and Google Cloud saying that the outage was not the result of a cyberattack.
[...] The joint statement and the outage updates are still not a technical post-mortem of what happened, and it's unclear if we'll get one. Google PR confirmed in multiple places it signed off on the statement, but a great breakdown from software developer Daniel Compton points out that the statement is not just vague, it's also full of terminology that doesn't align with Google Cloud products. The imprecise language makes it seem like the statement was written entirely by UniSuper. It would be nice to see a real breakdown of what happened from Google Cloud's perspective, especially when other current or potential customers are going to keep a watchful eye on how Google handles the fallout from this.
Anyway, don't put all your eggs in one cloud basket.
(Score: 2) by bzipitidoo on Friday May 24 2024, @02:20AM (3 children)
Yes, swapping the parameters is one of the tragic mistakes computers too easily empower. All the worse that commands aren't consistent on the ordering they expect. You've likely heard that "dd" stands for "disk destroyer". Ever meant to type in a capital A, and instead of hitting the shift key, you hit the ctrl key, which in some editors and applications selects everything, and then your next keypress replaces everything with the single character you just typed? If the editor can undo that mistake, no big deal, but occasionally, it can't be undone.
One of the worst systems for too easily losing text is the text boxes our browsers use to write these very comments. I've gotten into the habit of using that same dangerous ctrl-a followed by ctrl-c to copy my comments to the clipboard, in case something goes wrong when I mash that submit button, and then, when I go back, the browser presents me with a text box empty of the comment I had just made.
(Score: 2) by RS3 on Friday May 24 2024, @03:17AM (1 child)
Yes, I've done the ctrl-a ctrl-c when commenting here and other places. For a while this site would stop responding for a minute or many. Other times I've had browser crash. Sometimes I'm just not 'feeling it' and don't wish to post, maybe come back to it. However, every now and then the browser surprises me by remembering what I've written and filling in the text box with what I had written, like if I accidentally hit ctrl-w, which is a very common keystroke for me.
You mentioned 'dd', but I think dd is safer because you have to specify 'if' and 'of', which has the chance of hopefully making the person think a bit more. I'm pretty sure the aforementioned C/Unix guys used cp.
(Score: 2) by bzipitidoo on Friday May 24 2024, @02:15PM
The thing about having a "select all" function is the use cases. When would anyone want to "select all" in order to replace everything with a single character? If it's not "never", it's certainly rare enough that it ought to be changed. No one wants to do that when there are other options such as opening a new file. So why do editors facilitate such an action? After pressing ctrl-a, the main and almost only keypress that does something functional without erasing everything is ctrl-c. The only other keypresses that don't erase everything but instead "select none" are the arrow keys, and "select not quite all" are shift-left and up arrow keys. A change to that functionality could save a lot of pain. Maybe ctrl-a should be "copy everything to the clipboard and don't select anything", instead of "select all". Or maybe it should be "select all, for copying only", and the input of any visible character should deselect everything then just append that character to the end of the existing text, not replace all that text. If the user really does want it all deleted, let them use the backspace or the delete key for that.
In the early days of the MMORPG Everquest, 'a' was for "attack", and soon earned a reputation as the "'a' key of death". The player would be in a town facing a friendly NPC that was much more powerful than the player, and be chatting with other people, but missed prefacing their latest chat with the command to tell the system it was chat text, and the next time they used a word with the letter 'a', the system would take this as a wish to suicidally attack that friendly NPC, instantly turning it into an enemy that would squash the player in one mighty blow. Everquest eventually moved that "attack" command from 'a' to, IIRC, ctrl-a.
(Score: 2) by RS3 on Friday May 24 2024, @03:19AM
While I'm thinking of dd, if you're ever trying to recover a damaged hard drive, trying to dd sick drive to a good one or image file, dd will halt on sector errors. Check out ddrescue.