SoylentNews is people
SoylentNews
Alright, I've literally been at it for seven hours. Here's what done, here's what needs to done, and if you find something broken, let me know.
- Upgraded all machines to Ubuntu 22.04 (from 14.04 in most cases)
- Successfully brought database cluster back up to full service
- Rebuild web frontend to modern nginx/Ubuntu from very old Gentoo
Here's what's going to happen sometime tonight or tomorrow
- Outbound SMTP service will be fixed
- Start fully decommissioning the first set of servers ...
There's likely a lot of things still broken. if you find something broken, leave it in the comments. I'll get to it in the next day. At least we're not on 8 year old software anymore. Current plan after taking care of the above, going to find a good host (probably Fastmail) for editor emails, I do have the backups of everyone inbox should they want it. This has been, very much an exercise in pain. If you want to support me directly: Patreon for recurring, or Ko-fi for one time.
I'll be back in a few hours to work on this more. We raised another $560 USD for charity, and I hope folks had a great time. The next parts I need to just do, so the next stream will likely involve dealing with getting rehash to run in Docker.
~ NCommander
Update - 2022/11/12: DNS resolved, and at least site outbound email should work now
Update - 2022/11/13: Site outbound email is actually working now
(Score: 2) by drussell on Sunday November 13 2022, @11:03AM (1 child)
I think he tried to point outgoing mail at one of his own personal servers temporarily, hoping that stopgap would at least allow sigh-up e-mails and things like your automated stats to work, but there is a high probability that he was not successful in actually making it work.
Hopefully that will be unbroken relatively rapidly today...
(Score: 4, Interesting) by janrinok on Sunday November 13 2022, @01:07PM
Yes, thank you. My comment is so that we can gather all the bugs in one place as I don't appear to be able to connect to our usual site buglists. That might be a local problem and just a coincidence.
I am pleased the NCommander has returned and the site is getting some long overdue attention. Many people criticise him - particularly the way he operates - but nobody else stepped up to actually help.
The various services (IRC, email) which seem peripheral to our site are actually very important. We have lost all of the submission bots (upstart, Arthur, etc) so submissions have be done manually. Guess what? - we have a very small community and few people can be bothered to make submissions nowadays. I am sure that this comment will prompt a few more but it will be necessary to keep that up 7 days a week. To find 8 good stories usually takes at least twice that number of submissions.
Each URL takes many times longer to process as the tools that we have developed were designed to use the infrastructure that we had (RSS feeds, IRC, email, wiki etc). If I had more hours in the day I could rewrite my own software but it used information that was provided by the site - that has suddenly stopped or is inaccessible. The information that I need from wiki is not there. NC can write programs in minutes - it takes me a lot longer. IRC used to have several secure channels so that staff could discuss issues without compromising security or personal data. They are no longer available to us. The next best method is email - which is also not available to us. I am currently looking for 2 tin cans and a very long piece of string...
Basically, for the time being at least we are operating at about 30% of our former efficiency - but I accept that the systems have to be updated and we desperately need the additional security that they provide. With only 2 or 3 active editors it is a huge increase in individual workload. It would have been more sensible to have the alternatives in place before ripping what we had to pieces. Not very sexy things to work on - but essential nevertheless. I know that this is easier said than done - but that doesn't make it any less important. Perhaps we should slow down and think things through rather than try to do everything at pace. We have struggled with the existing system for several years - a few more days would not make any difference.
Having an updated and secure site with no stories to publish or being unable to discuss things with each other isn't a huge step forward. There will be real benefits in the months ahead, but we have a lot of catching up to do to even reach where we were a few days ago.
Just my €0.02 worth.