SoylentNews is people
SoylentNews
Alright, I've literally been at it for seven hours. Here's what done, here's what needs to done, and if you find something broken, let me know.
- Upgraded all machines to Ubuntu 22.04 (from 14.04 in most cases)
- Successfully brought database cluster back up to full service
- Rebuild web frontend to modern nginx/Ubuntu from very old Gentoo
Here's what's going to happen sometime tonight or tomorrow
- Outbound SMTP service will be fixed
- Start fully decommissioning the first set of servers ...
There's likely a lot of things still broken. if you find something broken, leave it in the comments. I'll get to it in the next day. At least we're not on 8 year old software anymore. Current plan after taking care of the above, going to find a good host (probably Fastmail) for editor emails, I do have the backups of everyone inbox should they want it. This has been, very much an exercise in pain. If you want to support me directly: Patreon for recurring, or Ko-fi for one time.
I'll be back in a few hours to work on this more. We raised another $560 USD for charity, and I hope folks had a great time. The next parts I need to just do, so the next stream will likely involve dealing with getting rehash to run in Docker.
~ NCommander
Update - 2022/11/12: DNS resolved, and at least site outbound email should work now
Update - 2022/11/13: Site outbound email is actually working now
(Score: 3, Informative) by janrinok on Sunday November 13 2022, @12:15PM (8 children)
At the moment the site seems more responsive than it did 24 hours ago - that might just be that there is less traffic blocking the tubes, the French are not on strike (unlikely), or the system is dropping most of my work in a bit bucket somewhere. But it appears currently to be slightly better than it was.
From the editorial side everything seems to be working as expected but there might still be the odd bug lingering somewhere. Certainly no big stoppers, apart from email which you know about, found so far.
(Score: 0) by Anonymous Coward on Sunday November 13 2022, @12:23PM
Yes, I also noticed more responsive...and for a moment the green VT100 mode looked a bit sharper too(grin).
An apt fortune cookie just now?
> Take everything in stride. Trample anyone who gets in your way.
(Score: 2) by RS3 on Sunday November 13 2022, @04:34PM (5 children)
I'm sure most know that software updates usually contain bug fixes, often add security features and layers that actually slow systems down, but sometimes (happily) bring code / algorithm refinements that improve system speed and efficiency.
Without knowing true technical details, I'd guess the responsiveness improvement is mostly due to mysql cleanup, especially considering it's being referred to as a "cluster". Again, not knowing details, but if it's some kind of mirroring / rsync between 2 or more mysql instances, and one gets out of sync, who/what determines which is the correct one? Something, some process, anyway, and that process will greatly add to latency if it's trying to resolve a discrepancy.
(Score: 3, Informative) by NCommander on Sunday November 13 2022, @08:08PM (4 children)
No, we literally use Oracle's MySQL Cluster, although its an older community edition. That's on the overhaul list, but at least it's no longer in pieces of colorful confetti.
Still always moving
(Score: 2, Interesting) by fab23 on Sunday November 13 2022, @09:38PM (2 children)
Other options may be MariaDB Galera Cluster or then go to PostgreSQL (this may be a little bit more headache when doing major upgrades).
(Score: 2) by RS3 on Monday November 14 2022, @03:25AM (1 child)
Is a cluster necessary?
(Score: 2) by drussell on Monday November 14 2022, @12:10PM
Probably not...
(Score: 2) by RS3 on Monday November 14 2022, @03:20AM
What do you have against color? :-}
(Score: 2) by mrpg on Monday November 14 2022, @12:38AM
when I studied French in the book there was an episode about a strike. Years later I went to Paris and guess what, grève des trains. But the train arrived just one hour late, nothing real bad.