SoylentNews is people
SoylentNews
from the how-long-would-it-take-for-YOU-to-restore-a-backup? dept.
Link bookmarking service Instapaper came back online today following a critical database issue that forced it offline for 31 hours over the past two days. According to two blog posts [1, 2] detailing what happened, on February 8, 2016, at around 21:00 GMT, Instapaper's main database mindbogglingly filled up without anyone noticing, and stopped allowing users to save new links to their accounts.
Instapaper developers said that neither its staff or its cloud provider noticed that the database was nearing full capacity, so nobody took precautions to migrate the Instapaper database to a larger server beforehand. When it happened, the service was left with one option, and that was to export all Instapaper content and move it to a new server. Both operations were extremely slow, as most database migration processes generally are.
Instapaper came back online earlier today, on February 10, 2017, at around 3:00 GMT, after a massive and embarrassing 31-hour downtime. Nonetheless, the service isn't 100% yet. Instapaper staff says they only imported a small fraction of the user data into the new database. "In the interest of coming back up as soon as possible, this [database] instance only has the last six weeks of articles," Instapaper staff wrote. "For now, anything you've saved since December 20, 2016 is accessible."
The service expects to restore all data by February 17, next week, a whopping nine days after service went down.
(Score: 3, Insightful) by VLM on Monday February 13 2017, @06:12PM
Unfortunately, Instapaper was sold to Pinterest a few months ago
Ouch in my experience that changes the story quite a bit, some poor bastard who got downsized before the election as part of the merger probably has a really full mailbox of alert messages. Or more likely the mails all bounced into some unread root mailbox somewhere. This does change the tone of the story quite a bit.
I've worked in ops centers in the past, I can just imagine the conversation "So Instapaper is down, WTF do I care, I work at Pintrest?" "Whaddya mean we own Instapaper, nobody told operations nothing, WTF is an instspaper, some laser printer out there some wheres, IT supports laser printers not me." Unfortunately I distinctly remember having a conversation like that in the late 90s at a former employer I don't miss at all. I can see it taking 31 hours just to gain access to the servers, reverse engineer everything, then finally begin to fix it, etc.
(Score: 3, Funny) by SanityCheck on Monday February 13 2017, @10:49PM
Sounds like someone got what they deserved lol. Great analysis my friend.