SoylentNews is people
SoylentNews
from the 5-backup-strategies-weren't-enough dept.
Ruby Paulson at BlogVault reports
GitLab, the online tech hub, is facing issues as a result of an accidental database deletion that happened in the wee hours of last night. A tired, frustrated system administrator thought that deleting a database would solve the lag-related issues that had cropped up... only to discover too late that he'd executed the command for the wrong database.
[...] It's certainly freaky that all the five backup solutions that GitLab had were ineffective, but this incident demonstrates that a number of things can go wrong with backups. The real aim for any backup solution, is to be able to restore data with ease... but simple oversights could render backup solutions useless.
Computer Business Review adds
The data loss took place when a system administrator accidentally deleted a directory on the wrong server during a database replication process. A folder containing 300GB of live production data was completely wiped.
[...] The last potentially useful backup was taken six hours before the issue occurred.
However, this is not seen to be of any help as snapshots are normally taken every 24 hours and the data loss occurred six hours after the previous snapshot which [resulted in] six hours of data loss.
David Mytton, founder and CEO [of] Server Density, said: "This unfortunate incident at GitLab highlights the urgent need for businesses to review and refresh their backup and incident handling processes to ensure data loss is recoverable, and teams know how to handle the procedure.
GitLab has been updating a Google Doc with info on the ongoing incident.
Additional coverage at:
TechCrunch
The Register
(Score: -1, Redundant) by Anonymous Coward on Friday February 03 2017, @06:43AM
The title, as submitted, was
GitLab Admin Deletes Database; All Backups Fail
...but, apparently, the number 6 is more significant that the number 5.
That would be the 5 redundant backup methods which ALL failed, AKA the reason that the story was submitted.
There's editing, then there's burying the lead.
-- OriginalOwner_ [soylentnews.org]
(Score: 2) by Scruffy Beard 2 on Friday February 03 2017, @08:24AM
I think the back-ups worked properly.
If it is live, it is not a backup (because the changes would then be replicated on the "backups").
(Score: 2) by bob_super on Friday February 03 2017, @06:28PM
> I think the back-ups worked properly.
Apparently, the server behind the couch worked properly, after all the official backup systems proved to be corrupted or misconfigured.
Reminds me of the almost identical story I heard, I believe about Toy Story:
- Someone does a bad rm -rf command, nuking the whole movie and models.
- Backups are all useless. Pixar panics as they may have just lost 3 years of work.
- Some lady with a newborn points out that she copied the whole database to work from home (wouldn't fit anymore these days).
- They end up driving her desktop computer back to the building praying the whole way that the hard drive doesn't decide today's a good day to die (why they didn't duplicate before transport instead wasn't in the interview).
(Score: 0) by Anonymous Coward on Friday February 03 2017, @10:03PM
https://www.techdirt.com/articles/20120514/17243918918/how-toy-story-2-almost-got-deleted-except-that-one-person-made-home-backup.shtml [techdirt.com]
(Score: 4, Touché) by janrinok on Friday February 03 2017, @08:25AM
Welcome to the editing team - You can start on Saturday morning!
Alternatively, you can stop complaining about editors in general and let us get on with our job. If you don't like the job we do, then step up to the plate and become an editor. We are working on this site for the benefit of the whole community - not just for your personal gratification.
The title that you submitted gave no indication about how much data was lost. It could have been an hour, day, week or month. The fact that all the backups failed is still in the story, which people can read and draw their own conclusions. However, GitLab has said: [gitlab.com] "Losing production data is unacceptable and in a few days we'll publish a post on why this happened and a list of measures we will implement to prevent it happening again."
That will be the time to publish the story concentrating on why their procedures failed and why 5 redundant backups didn't solve the problem. Any other discussion before we have the facts is simply guesswork and hot air.
JR