SoylentNews is people
SoylentNews
This is a follow-up to our site crash.
If you are just tuning in, SoylentNews experienced a database crash on 2021-05-20. We tried to restore from recent backups, but found they were corrupted and unusable. Thanks to heroic work by mechanicjay the site is back up and running!
Many thanks to mechanicjay for his 16-hour(!) day on Thursday to get us back up and then move us to a single back-end configuration. He didn't stop there, but has continued on gathering information and guiding work to get us to a more stable foundation. We'll keep you posted as to our progress.
Read on if interested... otherwise, another story will be along presently.
First off, as I understand it, we had previously been running with a cluster database (DB) configuration using ndb1 and MySQL on two servers; (fluorine and hydrogen). This has been troublesome in the past. Our options going forward appeared to be either add yet another server (more redundancy apparently makes ndb less cranky) or to slim down, eliminate the cluster, and run with just a single server. At least for the time being, we have decided to go with the latter under the K.I.S.S. way of thinking.
It appears our database backups had issues because we had some free space on our server... but not enough to permit full and clean backups.
We now have a daily report e-mailed out to staff that lists: server name, disk space total, disk space used, and disk space available. This should help prevent a repeat occurence of out-of-space happening on our servers.
Secondly, the DB restore basically forgot everything that happened from 2021-04-14 onward. (Insert pithy Monty Python Dead Parrot joke here.) Yeah, it stinks losing all those stories, journals, comments, and moderations. fnord666 and I had each lost ~150 stories we had edited and pushed out to the site.
So, one of the things we found out that disappeared was people's site subscriptions. By a happy coincidence, I just happened to have a screen up on the site listing the most recent subscriptions. We had a long discussion among staff as to how to proceed. First off, when the site is back and stable, we need a high priority code change: we need to log each subscription to someplace in addition to the DB.
Just to make things more interesting, since the site came back up, we have had some new subscriptions come in. The easiest and safest approach to restore these subscriptions came out to be straightforward albeit tedious.
I went through all known subs that got dropped, and gave a "gift" subscription to replace them. These gifts were based on the minimums listed on the Subscribe page. For example, if your subscription was for $20.00 or more, you were gifted with a 365-day subscription. At least $12.00 but less than $20.00 would get a 180-day subscription. Lastly, any subscription for less than $12.00 received a 30-day gift subscription.
There's one complication, we are still dealing with. Two gift subscriptions were made between 2021-04-14 and 2021-05-20. I know in whose name and UID the gift was made, but not that of the user actually making the gift. (We know the "giftee" but not the "gifter".) In one case we *do* have an email address; I've reached out to that person via e-mail for more information. Sadly, I have no other identifying information for the other person.
tl;dr: please check your subscription. If you find a discrepancy, please send an e-mail to admin (at) soylentnews (dot) org and I will personally look into it for you. Please provide whatever information you can: date, amount, Stripe vs PayPal, etc.
Lastly, I've seen a strong increase in story submissions among other very encouraging signs. It's as if there's a change in attitude from "what can I get from the site" to "what can I contribute". Thanks everybody for all you've done!
1What is an NDB Cluster?
(Score: 0) by Anonymous Coward on Sunday May 23 2021, @11:11PM
NDB and Galera are more similar than they are different, which makes since because they came from the same roots in MySQL. NDB and Galera increase both speed and redundancy. The problem is that they didn't set it up properly. I don't know exactly what they mean by "add yet another server" because I haven't read any of the IRC exchanges and because they are already in a fully redundant setup, which has five nines uptime guarantees when set up correctly. But they don't have it set up correctly, so they do not have said uptime, and I don't know which type of node(s) they are thinking they need on that new server so I can't really say if it even would help. Not that it matters anymore, since they are migrating away from it.