from the slow-but-steady dept.
This is a follow-up to our site crash.
If you are just tuning in, SoylentNews experienced a database crash on 2021-05-20. We tried to restore from recent backups, but found they were corrupted and unusable. Thanks to heroic work by mechanicjay the site is back up and running!
Many thanks to mechanicjay for his 16-hour(!) day on Thursday to get us back up and then move us to a single back-end configuration. He didn't stop there, but has continued on gathering information and guiding work to get us to a more stable foundation. We'll keep you posted as to our progress.
Read on if interested... otherwise, another story will be along presently.
First off, as I understand it, we had previously been running with a cluster database (DB) configuration using ndb1 and MySQL on two servers; (fluorine and hydrogen). This has been troublesome in the past. Our options going forward appeared to be either add yet another server (more redundancy apparently makes ndb less cranky) or to slim down, eliminate the cluster, and run with just a single server. At least for the time being, we have decided to go with the latter under the K.I.S.S. way of thinking.
It appears our database backups had issues because we had some free space on our server... but not enough to permit full and clean backups.
We now have a daily report e-mailed out to staff that lists: server name, disk space total, disk space used, and disk space available. This should help prevent a repeat occurence of out-of-space happening on our servers.
Secondly, the DB restore basically forgot everything that happened from 2021-04-14 onward. (Insert pithy Monty Python Dead Parrot joke here.) Yeah, it stinks losing all those stories, journals, comments, and moderations. fnord666 and I had each lost ~150 stories we had edited and pushed out to the site.
So, one of the things we found out that disappeared was people's site subscriptions. By a happy coincidence, I just happened to have a screen up on the site listing the most recent subscriptions. We had a long discussion among staff as to how to proceed. First off, when the site is back and stable, we need a high priority code change: we need to log each subscription to someplace in addition to the DB.
Just to make things more interesting, since the site came back up, we have had some new subscriptions come in. The easiest and safest approach to restore these subscriptions came out to be straightforward albeit tedious.
I went through all known subs that got dropped, and gave a "gift" subscription to replace them. These gifts were based on the minimums listed on the Subscribe page. For example, if your subscription was for $20.00 or more, you were gifted with a 365-day subscription. At least $12.00 but less than $20.00 would get a 180-day subscription. Lastly, any subscription for less than $12.00 received a 30-day gift subscription.
There's one complication, we are still dealing with. Two gift subscriptions were made between 2021-04-14 and 2021-05-20. I know in whose name and UID the gift was made, but not that of the user actually making the gift. (We know the "giftee" but not the "gifter".) In one case we *do* have an email address; I've reached out to that person via e-mail for more information. Sadly, I have no other identifying information for the other person.
tl;dr: please check your subscription. If you find a discrepancy, please send an e-mail to admin (at) soylentnews (dot) org and I will personally look into it for you. Please provide whatever information you can: date, amount, Stripe vs PayPal, etc.
Lastly, I've seen a strong increase in story submissions among other very encouraging signs. It's as if there's a change in attitude from "what can I get from the site" to "what can I contribute". Thanks everybody for all you've done!
1What is an NDB Cluster?
As many of you noticed, we had a site crash today. From around 1300 until 2200 UTC (2021-05-20).
A HUGE thank you goes to mechanicjay who spent the whole time trying to get our ndb (cluster) working again. It's an uncommon configuration, which made recovery especially challenging... there's just not a lot of documentation about it on the web.
I reached out and got hold of The Mighty Buzzard on the phone. Then put him in touch with mechanicjay who got us back up and running using backups.
Unfortunately, we had to go way back until April 14 to get a working backup. (I don't know all the details, but it appears something went sideways on neon).
We're all wiped out right now. When we have rested and had a chance to discuss things, we'll post an update.
In the meantime, please join me in thanking mechanicjay and TMB for all they did to get us up and running again!