SoylentNews

Earlier this morning (around 0520 UTC) we experienced a sequence of faults which we do not, yet, fully understand. Unfortunately this coincided with several key admins being asleep as they should be.

Several of the team started to try to recover the site but all the obvious efforts failed to have the desired results (e.g. restarting the failed software and switching it all off and back on again). Chromas took the lead in this but he was unable to resolve the problems. SemperOSS showed up next and made a mighty effort but things failed to cooperate. Martyb was next in and checked on status of our servers and a few other things but ultimately could not provide much assistance.

TheMightyBuzzard turned up at his usual time and he and SemperOSS set to work trying to recover the site. After a long slog it became apparent that the existing software had suffered some form of corruption and that a simple recovery was not going to be possible. The decision was finally made to restore to yesterday's snapshot, but this comes with a downside. All the stories, comments, and journal entries since that snapshot was taken (approximately 2019-08-14 22:02:36 UTC) have been lost. This is regrettable but under the circumstances unavoidable.

Some of you joined us on IRC and provided real or moral support to those carrying out the recovery task. Others sent us emails informing us of the outage and we also received offers of help. For this we thank you. We also thank our community at large for your patience while we carried out the work that needed to be done. The investigation into what happened has still to be done and will take some time to do. The site is now up again, but as we are not 100% sure what caused the problem we cannot guarantee that the site will function properly and we will have to wait a day or two to be confident that we have resolved the issue.

We tried to keep everyone updated on our progress on our IRC channel but, if you could not get there, then there was little else we could do to update you on our progress. Displaying a short 'Site Down' message actually exacerbated the problems so we decided that it was better to leave the database alone until the problems had been resolved.

Finally, I would like to say thank you to the guys who did all the work under difficult circumstances. Their efforts are appreciated. We used the usual piece of software on IRC to find out who should bear the blame for this debacle (~blame) — it responded with the only name it has in its choice of staff. Bytram!