SoylentNews is people
SoylentNews
So, quick update here. The site was down for most of the night because the database cluster shot itself in the head. I had restarted a machine to install updates, and this caused the backend cluster to entire to entirely loose its mind. Unfortunately, I didn't have a manual dump of the database made, just a VM snapshot, since, well, I wasn't tinkering with it directly. I've mostly been trying to patch things to the point that I can sleep, and leaving things down like IRC and email which need to be seriously overhauled before they can go back up.
As far as damages go, it looks like we lost 10 or so days of messages, which uh, sucks for multiple reasons. We're currently on ##soylentnews on Libera.Chat while I pull bits of the site out of the flames, but I'm at the point that if I don't sleep, I will make things worse. Corruption in the production database is very much not what I wanted, and we're very much in limp mode for the moment. I'm going to let staff handle IRC and comments while I sleep, and then I'll post another update when I'm awake.
See you in a few hours
~ NCommander
(Score: 2) by RS3 on Saturday November 12 2022, @06:24PM (6 children)
Wow, that's all very interesting, thanks for that. From your other postings, I had no idea this was your main gig.
It's a very interesting approach. Are you using any "automation" (puppet, chef, ansible, etc.)?
What hypervisor are you using?
(Score: 2) by JoeMerchant on Saturday November 12 2022, @06:57PM (2 children)
At one point we were using a bare metal hypervisor from some company in Switzerland, it preferred to work with CentOS so we were using Xfce over Cent over that hypervisor with a couple of cores devoted to running the GUI in Windows, trading control of the display back and forth between development and production modes...
Then we did a tech eval and decided that Virtual Box was good enough for our purposes, so the GUI moved in there and the host OS moved to Ubuntu.
The whole system communicates internally through a RabbitMQ server (Intel core based apps using AMQP, ST micro components using MQTT) and eventually that Rabbit MQ server found it's way into a docker container running on the generic dockerd community edition.
In other words it's a complicated mess, but I think a significant improvement over the mess it replaced.
🌻🌻 [google.com]
(Score: 2) by RS3 on Saturday November 12 2022, @07:54PM (1 child)
Ah, it's something special, not generic web / cloud hosting. Pretty cool! I do kind of remember you writing about it sometime long ago...
I'm deploying a new server, after keeping some old ones going perfectly well for years. Older CentOS (6), updates stopped, but I've had no problems. One has uptime of 390 days right now. Ain't broke, not fixing it... Well, new server is big CPU and RAM, so hypervisor is needed. I've run VMWare, Xen, messed with a few others, need to stay away from big $ software licensing / subscription costs. Most likely going with kvm. Trying to steer away from systemd- partly because I hear too much bad, and that dovetails with this IT gig not being full-time, nor any regular hours. The _last_ thing I need is emergencies 15+ miles from where I live or work. IE, I'm spoiled by systems that just run, and there have been time periods when I haven't visited the physical site in more than 1.5 years.
I've tried and like Devuan, and MX is pretty cool, but neither are really server-oriented. Love Alpine, wish for a better package manager. About to evaluate Void. Longtime Slackware user- I just worry that if I depart the situation, I don't want to leave something complicated for whoever takes over next...
(Score: 2) by JoeMerchant on Saturday November 12 2022, @08:52PM
Yeah, our system is single user but we have deployed thousands of systems around the world, different animal from a single server with thousands of users, but it's remarkable how many tools apply to both worlds.
We got a minor burn from our systemd service file behavior the other day, it had Rabbit / Docker as a dependancy for another service we run, never expected killing that service would also kill docker... Now that we know it's not a problem, but it was about 20 hours of developer investigation to deal with that bit of nonsense.
🌻🌻 [google.com]
(Score: 2) by JoeMerchant on Saturday November 12 2022, @07:33PM (2 children)
As for Ansible etc.... Nothing much along those lines yet, mostly just bash scripts for setup and Qt apps in the live system to take care of various system things as needed.
Dockerfiles are an interesting variation on the bash script approach, and we use them to (mildly) customize the Rabbit MQ server container.
The thing I like about bash scripts is that they easily encapsulate stuff you can try on the command line, and minor system image updates can be distributed to the team as a patch script that is usually identical to the modification of the system setup scripts that implements the same thing.
🌻🌻 [google.com]
(Score: 2) by RS3 on Saturday November 12 2022, @08:09PM (1 child)
Yeah, for what you're doing, esp. that you've been developing your scripts all along, it would probably be more effort to set up Ansible, et al. Those are more oriented to large stacks of duplicate servers.
That said, some of the "automation" packages make some things much easier, like samba, apache, mysql, and other configurations. I've occasionally used them on a test server just to get some information from them about their ideas for config files, in case I'm missing something, etc. But I already have a big head start on config files, so kind of like systemd, I'm not sure I want to trust someone else's idea of how my system's config files are written...
Since I don't have a stack of servers, just a few, I've done like you- written several simple scripts to automate simple things like updating WordPress plugins and themes. WordPress has built-in updating available, but it requires giving a site access password to the source for the update.
IIRC, Apache used to run per virtual hosts ownership / permissions. IE, each customer's /home directory is owned by that username, and each Apache process would only have access to those files. But unfortunately Apache runs as apache:apache, so as far as I can tell, giving that access password to one WordPress site would or could give access to all sites. Although it is difficult to get past the virtual root for Apache's files, still, it's a risk.
Some years ago there was a malicious WordPress plugin that used /tmp as a mechanism to do its dirtywork. I didn't study the thing- it was right around when I inherited the sites around 15 years ago. That was a pretty quick fix, including just some raw OS updates.
(Score: 2) by JoeMerchant on Saturday November 12 2022, @09:03PM
Just recently we updated Virtual Box to whatever installs in Ubuntu 22.04 with apt and they went and locked down (well, made less open) configuration of host only network addresses, now you have to create folder /etc/vbox and put a network.conf file in there to specify something other than their 192.168.56.1/20 default range...
It's always something.
For SN I would really try to avoid all the "cool" stuff that opens so many security holes like you describe. If you don't have fancy stuff, then you don't need to provide access for updating of the fancy stuff.
🌻🌻 [google.com]