Alright, I've literally been at it for seven hours. Here's what done, here's what needs to done, and if you find something broken, let me know.
- Upgraded all machines to Ubuntu 22.04 (from 14.04 in most cases)
- Successfully brought database cluster back up to full service
- Rebuild web frontend to modern nginx/Ubuntu from very old Gentoo
Here's what's going to happen sometime tonight or tomorrow
- Outbound SMTP service will be fixed
- Start fully decommissioning the first set of servers ...
There's likely a lot of things still broken. if you find something broken, leave it in the comments. I'll get to it in the next day. At least we're not on 8 year old software anymore. Current plan after taking care of the above, going to find a good host (probably Fastmail) for editor emails, I do have the backups of everyone inbox should they want it. This has been, very much an exercise in pain. If you want to support me directly: Patreon for recurring, or Ko-fi for one time.
I'll be back in a few hours to work on this more. We raised another $560 USD for charity, and I hope folks had a great time. The next parts I need to just do, so the next stream will likely involve dealing with getting rehash to run in Docker.
~ NCommander
Update - 2022/11/12: DNS resolved, and at least site outbound email should work now
Update - 2022/11/13: Site outbound email is actually working now
(Score: 2) by SomeGuy on Sunday November 13 2022, @02:15AM (2 children)
Damn, since the last post I made I couple of hours ago it seems the minimum required TLS/encryption level has increased. No longer works in my SeaMonkey 1.1.20pre, although Retrozilla works for the moment.
(Score: 3, Interesting) by AlwaysNever on Sunday November 13 2022, @02:32AM (1 child)
The web front end was updated from Ubuntu 14.04 to 22.04. Probably TLS 1.0 is not supported by default by Ubuntu 22.04?
(Score: 2) by RS3 on Sunday November 13 2022, @05:27AM
I'm seeing:
(Score: 2) by drussell on Sunday November 13 2022, @03:13AM (1 child)
DNS is borked right now because the primary nameserver is offline, thus cache entries are expiring but NCommander is working on it...
(Score: 2) by drussell on Sunday November 13 2022, @04:55AM
Thanks to NCommander, DNS is back up... :)
(Score: 5, Informative) by Anonymous Coward on Sunday November 13 2022, @04:14AM (3 children)
Anyone interested in the titles of the recent articles (lost in database crash??) on the SN front page can find them here:
https://twitter.com/soylentnews/ [twitter.com]
(Score: 3, Informative) by captain normal on Sunday November 13 2022, @07:43PM (2 children)
You can't because once you start scrolling down, a pop-up that can't be blocked says you have to open a twitter account. That has just appeared in the last week.
The Musk/Trump interview appears to have been hacked, but not a DDOS hack...more like A Distributed Denial of Reality.
(Score: 2) by drussell on Sunday November 13 2022, @08:22PM (1 child)
Yeah, some masochist with a twitter account needs to pull that and repost it somewhere... :)
(Score: 4, Informative) by VanessaE on Sunday November 13 2022, @10:04PM
Ask and ye shall receive :)
https://i.imgur.com/9kmrjZz.jpg [imgur.com]
Fair warning: 1270x28800 in size, 7 MB
(Score: 0) by Anonymous Coward on Sunday November 13 2022, @05:19AM (1 child)
As of midnight EST (USA east coast) the IRC link isn't working -- near the bottom of the left column, home page.
Tries to go to chat.soylentnews.org but never gets there.
(Score: 5, Informative) by drussell on Sunday November 13 2022, @05:33AM
That box is currently offline...
Temporarily, for IRC use the ##soylentnews channel on on libera.chat instead.
(Score: 4, Informative) by janrinok on Sunday November 13 2022, @08:09AM (2 children)
I have not received any SN email for almost 2 days now - no stats info etc.
Cannot connect to SN email via Thunderbird - it just times out.
Cannot contact other staff without full SN email. It isn't a trivial side issue.
I am not interested in knowing who people are or where they live. My interest starts and stops at our servers.
(Score: 2) by drussell on Sunday November 13 2022, @11:03AM (1 child)
I think he tried to point outgoing mail at one of his own personal servers temporarily, hoping that stopgap would at least allow sigh-up e-mails and things like your automated stats to work, but there is a high probability that he was not successful in actually making it work.
Hopefully that will be unbroken relatively rapidly today...
(Score: 4, Interesting) by janrinok on Sunday November 13 2022, @01:07PM
Yes, thank you. My comment is so that we can gather all the bugs in one place as I don't appear to be able to connect to our usual site buglists. That might be a local problem and just a coincidence.
I am pleased the NCommander has returned and the site is getting some long overdue attention. Many people criticise him - particularly the way he operates - but nobody else stepped up to actually help.
The various services (IRC, email) which seem peripheral to our site are actually very important. We have lost all of the submission bots (upstart, Arthur, etc) so submissions have be done manually. Guess what? - we have a very small community and few people can be bothered to make submissions nowadays. I am sure that this comment will prompt a few more but it will be necessary to keep that up 7 days a week. To find 8 good stories usually takes at least twice that number of submissions.
Each URL takes many times longer to process as the tools that we have developed were designed to use the infrastructure that we had (RSS feeds, IRC, email, wiki etc). If I had more hours in the day I could rewrite my own software but it used information that was provided by the site - that has suddenly stopped or is inaccessible. The information that I need from wiki is not there. NC can write programs in minutes - it takes me a lot longer. IRC used to have several secure channels so that staff could discuss issues without compromising security or personal data. They are no longer available to us. The next best method is email - which is also not available to us. I am currently looking for 2 tin cans and a very long piece of string...
Basically, for the time being at least we are operating at about 30% of our former efficiency - but I accept that the systems have to be updated and we desperately need the additional security that they provide. With only 2 or 3 active editors it is a huge increase in individual workload. It would have been more sensible to have the alternatives in place before ripping what we had to pieces. Not very sexy things to work on - but essential nevertheless. I know that this is easier said than done - but that doesn't make it any less important. Perhaps we should slow down and think things through rather than try to do everything at pace. We have struggled with the existing system for several years - a few more days would not make any difference.
Having an updated and secure site with no stories to publish or being unable to discuss things with each other isn't a huge step forward. There will be real benefits in the months ahead, but we have a lot of catching up to do to even reach where we were a few days ago.
Just my €0.02 worth.
I am not interested in knowing who people are or where they live. My interest starts and stops at our servers.
(Score: 3, Informative) by janrinok on Sunday November 13 2022, @12:15PM (8 children)
At the moment the site seems more responsive than it did 24 hours ago - that might just be that there is less traffic blocking the tubes, the French are not on strike (unlikely), or the system is dropping most of my work in a bit bucket somewhere. But it appears currently to be slightly better than it was.
From the editorial side everything seems to be working as expected but there might still be the odd bug lingering somewhere. Certainly no big stoppers, apart from email which you know about, found so far.
I am not interested in knowing who people are or where they live. My interest starts and stops at our servers.
(Score: 0) by Anonymous Coward on Sunday November 13 2022, @12:23PM
Yes, I also noticed more responsive...and for a moment the green VT100 mode looked a bit sharper too(grin).
An apt fortune cookie just now?
> Take everything in stride. Trample anyone who gets in your way.
(Score: 2) by RS3 on Sunday November 13 2022, @04:34PM (5 children)
I'm sure most know that software updates usually contain bug fixes, often add security features and layers that actually slow systems down, but sometimes (happily) bring code / algorithm refinements that improve system speed and efficiency.
Without knowing true technical details, I'd guess the responsiveness improvement is mostly due to mysql cleanup, especially considering it's being referred to as a "cluster". Again, not knowing details, but if it's some kind of mirroring / rsync between 2 or more mysql instances, and one gets out of sync, who/what determines which is the correct one? Something, some process, anyway, and that process will greatly add to latency if it's trying to resolve a discrepancy.
(Score: 3, Informative) by NCommander on Sunday November 13 2022, @08:08PM (4 children)
No, we literally use Oracle's MySQL Cluster, although its an older community edition. That's on the overhaul list, but at least it's no longer in pieces of colorful confetti.
Still always moving
(Score: 2, Interesting) by fab23 on Sunday November 13 2022, @09:38PM (2 children)
Other options may be MariaDB Galera Cluster or then go to PostgreSQL (this may be a little bit more headache when doing major upgrades).
(Score: 2) by RS3 on Monday November 14 2022, @03:25AM (1 child)
Is a cluster necessary?
(Score: 2) by drussell on Monday November 14 2022, @12:10PM
Probably not...
(Score: 2) by RS3 on Monday November 14 2022, @03:20AM
What do you have against color? :-}
(Score: 2) by mrpg on Monday November 14 2022, @12:38AM
when I studied French in the book there was an episode about a strike. Years later I went to Paris and guess what, grève des trains. But the train arrived just one hour late, nothing real bad.
(Score: 2) by drussell on Sunday November 13 2022, @08:20PM (3 children)
Notes for bugsquishing:
More to come, I'm sure... 🙂
(Score: 4, Interesting) by NCommander on Sunday November 13 2022, @10:05PM (2 children)
Password function fixed; caused by broken mail setup. The comments page is utterly bizarre. I need to look more in-depth with that.
Still always moving
(Score: 2) by drussell on Monday November 14 2022, @03:00AM (1 child)
Sweet...
Additionally:
(Score: 2) by drussell on Monday November 14 2022, @04:33PM
Scratch that... The initial report of brokenness has been rescinded by mrpg, apparently it was working all along.
One less thing to worry about! :)
(Score: 1) by fab23 on Sunday November 13 2022, @09:33PM
This is just a test for email
(Score: 2) by acid andy on Sunday November 13 2022, @11:56PM (4 children)
I can't get the Search [soylentnews.org] to work at the moment. Not sure if you're already aware.
For example if I put in Microsoft, select Stories and sort by most recent, there are no results. None of the other searches I tried worked either. Maybe some indicies need rebuilding on the database? Or something, what do I know...
Welcome to Edgeways. Words should apply in advance as spaces are highly limite—
(Score: 2) by mrpg on Monday November 14 2022, @12:40AM (2 children)
It gives me No stories were found that match your query.
(Score: 2) by acid andy on Monday November 14 2022, @01:16AM (1 child)
Yes that's the actual message it gives me as well, even if I search Stories for Unix Time which is in the title of the most recent story.
Welcome to Edgeways. Words should apply in advance as spaces are highly limite—
(Score: 2) by nostyle on Monday November 14 2022, @04:28AM
I noticed this earlier myself - I was trying to search for a previous comment I'd written about unix time.
I tried again just now, and search fails for "comments" "users" "stories" and "journals". I'd guess nothing there works.
(Score: 1) by khallow on Friday November 18 2022, @02:21AM
(Score: 2) by DannyB on Monday November 14 2022, @03:29PM (1 child)
Thank you for doing a thankless job with little fanfare.
The Centauri traded Earth jump gate technology in exchange for our superior hair mousse formulas.
(Score: 4, Interesting) by drussell on Monday November 14 2022, @04:31PM
Oh, there was fanfare... A couple hundred people showed up in the audience just to cheer him on. :)
A virtually thankless job, though.... nonetheless
(Score: 2) by cmdrklarg on Monday November 14 2022, @07:35PM (1 child)
My posting history appears to have reverted back to October 27th. No idea how many went bye bye, but I do know I had posted a number of times in November.
Thanks for the hard work!
The world is full of kings and queens who blind your eyes and steal your dreams.
(Score: 2) by cmdrklarg on Monday November 14 2022, @07:39PM
Aaaaaand I see the post from Saturday about the site borkage. Never mind!
The world is full of kings and queens who blind your eyes and steal your dreams.