So, quick update here. The site was down for most of the night because the database cluster shot itself in the head. I had restarted a machine to install updates, and this caused the backend cluster to entire to entirely loose its mind. Unfortunately, I didn't have a manual dump of the database made, just a VM snapshot, since, well, I wasn't tinkering with it directly. I've mostly been trying to patch things to the point that I can sleep, and leaving things down like IRC and email which need to be seriously overhauled before they can go back up.
As far as damages go, it looks like we lost 10 or so days of messages, which uh, sucks for multiple reasons. We're currently on ##soylentnews on Libera.Chat while I pull bits of the site out of the flames, but I'm at the point that if I don't sleep, I will make things worse. Corruption in the production database is very much not what I wanted, and we're very much in limp mode for the moment. I'm going to let staff handle IRC and comments while I sleep, and then I'll post another update when I'm awake.
See you in a few hours
~ NCommander
(Score: 2) by RS3 on Saturday November 12 2022, @06:37PM (2 children)
Not knowing the details, I can't speak authoritatively, but moving an OS installation into a container requires some virtualization considerations, especially hardware, like Ethernet, USB (keyboard), graphics (for local terminals), etc. Theoretically a hypervisor or container should be able to emulate any hardware and run any OS, but of course, that never happens, so it's best to use ones that work well together.
It's possible that a docker image would work well on someone's local machine, but have problems when uploaded to the actual host.
But I doubt anything SN would be problematic, as it's pretty standard stuff (perl, mysql, nginx (or apache?)...
(Score: 2) by janrinok on Saturday November 12 2022, @06:46PM (1 child)
Understood
.
For me it is more that I can change the software locally, test it, and then put in a pull request to have my changes built into the next docker release. At present, having a full system configured with the same out-of-date software that our current servers use has proven impossible to achieve. And if I had have achieved it - it would be insecure and receiving no security updates.
We have been unable to do this for a long time now.
(Score: 2) by RS3 on Saturday November 12 2022, @08:14PM
Makes perfect sense.
At this moment I'm not sure how you could have a development server that's worked on by many developers / admins, without the file contention problem... I suppose you could run a git host on it, and then people could check out and own certain things. Just thinking out loud... setting up and adminning the git might be no fun... Your aforementioned more brute-force method is probably much better, just that someone has to play "traffic cop" and make sure to merge the updates, then when two or more people overlap efforts, communicate (irc?) and figure out how to merge. Thanks!! Major progress is being made!