Stories
Slash Boxes
Comments

SoylentNews is people

Meta
posted by NCommander on Sunday November 13 2022, @04:44AM   Printer-friendly
from the that-was-a-lot-of-work dept.

Alright, I've literally been at it for seven hours. Here's what done, here's what needs to done, and if you find something broken, let me know.

  • Upgraded all machines to Ubuntu 22.04 (from 14.04 in most cases)
  • Successfully brought database cluster back up to full service
  • Rebuild web frontend to modern nginx/Ubuntu from very old Gentoo

Here's what's going to happen sometime tonight or tomorrow

  • Outbound SMTP service will be fixed
  • Start fully decommissioning the first set of servers ...

There's likely a lot of things still broken. if you find something broken, leave it in the comments. I'll get to it in the next day. At least we're not on 8 year old software anymore. Current plan after taking care of the above, going to find a good host (probably Fastmail) for editor emails, I do have the backups of everyone inbox should they want it. This has been, very much an exercise in pain. If you want to support me directly: Patreon for recurring, or Ko-fi for one time.

I'll be back in a few hours to work on this more. We raised another $560 USD for charity, and I hope folks had a great time. The next parts I need to just do, so the next stream will likely involve dealing with getting rehash to run in Docker.

~ NCommander

Update - 2022/11/12: DNS resolved, and at least site outbound email should work now

Update - 2022/11/13: Site outbound email is actually working now

This discussion was created by NCommander (2) for logged-in users only, but now has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
(1)
  • (Score: 2) by SomeGuy on Sunday November 13 2022, @02:15AM (2 children)

    by SomeGuy (5632) on Sunday November 13 2022, @02:15AM (#1279419)

    Damn, since the last post I made I couple of hours ago it seems the minimum required TLS/encryption level has increased. No longer works in my SeaMonkey 1.1.20pre, although Retrozilla works for the moment.

    • (Score: 3, Interesting) by AlwaysNever on Sunday November 13 2022, @02:32AM (1 child)

      by AlwaysNever (5817) on Sunday November 13 2022, @02:32AM (#1279422)

      The web front end was updated from Ubuntu 14.04 to 22.04. Probably TLS 1.0 is not supported by default by Ubuntu 22.04?

      • (Score: 2) by RS3 on Sunday November 13 2022, @05:27AM

        by RS3 (6367) on Sunday November 13 2022, @05:27AM (#1279447)

        I'm seeing:

        TLS v1.2 128 bit AES GCM ([256, 2048] bit ECDHE_RSA/SHA-256)

  • (Score: 2) by drussell on Sunday November 13 2022, @03:13AM (1 child)

    by drussell (2678) on Sunday November 13 2022, @03:13AM (#1279428) Journal

    DNS is borked right now because the primary nameserver is offline, thus cache entries are expiring but NCommander is working on it...

  • (Score: 5, Informative) by Anonymous Coward on Sunday November 13 2022, @04:14AM (3 children)

    by Anonymous Coward on Sunday November 13 2022, @04:14AM (#1279434)

    Anyone interested in the titles of the recent articles (lost in database crash??) on the SN front page can find them here:
          https://twitter.com/soylentnews/ [twitter.com]

    • (Score: 3, Informative) by captain normal on Sunday November 13 2022, @07:43PM (2 children)

      by captain normal (2205) on Sunday November 13 2022, @07:43PM (#1279536)

      You can't because once you start scrolling down, a pop-up that can't be blocked says you have to open a twitter account. That has just appeared in the last week.

      --
      "If men were angels, government would not be necessary." James Madison
  • (Score: 0) by Anonymous Coward on Sunday November 13 2022, @05:19AM (1 child)

    by Anonymous Coward on Sunday November 13 2022, @05:19AM (#1279444)

    As of midnight EST (USA east coast) the IRC link isn't working -- near the bottom of the left column, home page.

    Tries to go to chat.soylentnews.org but never gets there.

    • (Score: 5, Informative) by drussell on Sunday November 13 2022, @05:33AM

      by drussell (2678) on Sunday November 13 2022, @05:33AM (#1279448) Journal

      That box is currently offline...

      Temporarily, for IRC use the ##soylentnews channel on on libera.chat instead.

  • (Score: 4, Informative) by janrinok on Sunday November 13 2022, @08:09AM (2 children)

    by janrinok (52) Subscriber Badge on Sunday November 13 2022, @08:09AM (#1279463) Journal

    I have not received any SN email for almost 2 days now - no stats info etc.

    Cannot connect to SN email via Thunderbird - it just times out.

    Cannot contact other staff without full SN email. It isn't a trivial side issue.

    --
    I am not interested in knowing who people are or where they live. My interest starts and stops at our servers.
    • (Score: 2) by drussell on Sunday November 13 2022, @11:03AM (1 child)

      by drussell (2678) on Sunday November 13 2022, @11:03AM (#1279480) Journal

      I think he tried to point outgoing mail at one of his own personal servers temporarily, hoping that stopgap would at least allow sigh-up e-mails and things like your automated stats to work, but there is a high probability that he was not successful in actually making it work.

      Hopefully that will be unbroken relatively rapidly today...

      • (Score: 4, Interesting) by janrinok on Sunday November 13 2022, @01:07PM

        by janrinok (52) Subscriber Badge on Sunday November 13 2022, @01:07PM (#1279502) Journal

        Yes, thank you. My comment is so that we can gather all the bugs in one place as I don't appear to be able to connect to our usual site buglists. That might be a local problem and just a coincidence.

        I am pleased the NCommander has returned and the site is getting some long overdue attention. Many people criticise him - particularly the way he operates - but nobody else stepped up to actually help.

        The various services (IRC, email) which seem peripheral to our site are actually very important. We have lost all of the submission bots (upstart, Arthur, etc) so submissions have be done manually. Guess what? - we have a very small community and few people can be bothered to make submissions nowadays. I am sure that this comment will prompt a few more but it will be necessary to keep that up 7 days a week. To find 8 good stories usually takes at least twice that number of submissions.

        Each URL takes many times longer to process as the tools that we have developed were designed to use the infrastructure that we had (RSS feeds, IRC, email, wiki etc). If I had more hours in the day I could rewrite my own software but it used information that was provided by the site - that has suddenly stopped or is inaccessible. The information that I need from wiki is not there. NC can write programs in minutes - it takes me a lot longer. IRC used to have several secure channels so that staff could discuss issues without compromising security or personal data. They are no longer available to us. The next best method is email - which is also not available to us. I am currently looking for 2 tin cans and a very long piece of string...

        Basically, for the time being at least we are operating at about 30% of our former efficiency - but I accept that the systems have to be updated and we desperately need the additional security that they provide. With only 2 or 3 active editors it is a huge increase in individual workload. It would have been more sensible to have the alternatives in place before ripping what we had to pieces. Not very sexy things to work on - but essential nevertheless. I know that this is easier said than done - but that doesn't make it any less important. Perhaps we should slow down and think things through rather than try to do everything at pace. We have struggled with the existing system for several years - a few more days would not make any difference.

        Having an updated and secure site with no stories to publish or being unable to discuss things with each other isn't a huge step forward. There will be real benefits in the months ahead, but we have a lot of catching up to do to even reach where we were a few days ago.

        Just my €0.02 worth.

        --
        I am not interested in knowing who people are or where they live. My interest starts and stops at our servers.
  • (Score: 3, Informative) by janrinok on Sunday November 13 2022, @12:15PM (8 children)

    by janrinok (52) Subscriber Badge on Sunday November 13 2022, @12:15PM (#1279490) Journal

    At the moment the site seems more responsive than it did 24 hours ago - that might just be that there is less traffic blocking the tubes, the French are not on strike (unlikely), or the system is dropping most of my work in a bit bucket somewhere. But it appears currently to be slightly better than it was.

    From the editorial side everything seems to be working as expected but there might still be the odd bug lingering somewhere. Certainly no big stoppers, apart from email which you know about, found so far.

    --
    I am not interested in knowing who people are or where they live. My interest starts and stops at our servers.
    • (Score: 0) by Anonymous Coward on Sunday November 13 2022, @12:23PM

      by Anonymous Coward on Sunday November 13 2022, @12:23PM (#1279492)

      Yes, I also noticed more responsive...and for a moment the green VT100 mode looked a bit sharper too(grin).

      An apt fortune cookie just now?
      > Take everything in stride. Trample anyone who gets in your way.

    • (Score: 2) by RS3 on Sunday November 13 2022, @04:34PM (5 children)

      by RS3 (6367) on Sunday November 13 2022, @04:34PM (#1279528)

      I'm sure most know that software updates usually contain bug fixes, often add security features and layers that actually slow systems down, but sometimes (happily) bring code / algorithm refinements that improve system speed and efficiency.

      Without knowing true technical details, I'd guess the responsiveness improvement is mostly due to mysql cleanup, especially considering it's being referred to as a "cluster". Again, not knowing details, but if it's some kind of mirroring / rsync between 2 or more mysql instances, and one gets out of sync, who/what determines which is the correct one? Something, some process, anyway, and that process will greatly add to latency if it's trying to resolve a discrepancy.

      • (Score: 3, Informative) by NCommander on Sunday November 13 2022, @08:08PM (4 children)

        by NCommander (2) Subscriber Badge <michael@casadevall.pro> on Sunday November 13 2022, @08:08PM (#1279548) Homepage Journal

        No, we literally use Oracle's MySQL Cluster, although its an older community edition. That's on the overhaul list, but at least it's no longer in pieces of colorful confetti.

        --
        Still always moving
        • (Score: 2, Interesting) by fab23 on Sunday November 13 2022, @09:38PM (2 children)

          by fab23 (6605) Subscriber Badge on Sunday November 13 2022, @09:38PM (#1279565) Homepage Journal

          Other options may be MariaDB Galera Cluster or then go to PostgreSQL (this may be a little bit more headache when doing major upgrades).

          • (Score: 2) by RS3 on Monday November 14 2022, @03:25AM (1 child)

            by RS3 (6367) on Monday November 14 2022, @03:25AM (#1279593)

            Is a cluster necessary?

        • (Score: 2) by RS3 on Monday November 14 2022, @03:20AM

          by RS3 (6367) on Monday November 14 2022, @03:20AM (#1279592)

          What do you have against color? :-}

    • (Score: 2) by mrpg on Monday November 14 2022, @12:38AM

      by mrpg (5708) <mrpgNO@SPAMsoylentnews.org> on Monday November 14 2022, @12:38AM (#1279579) Homepage

      when I studied French in the book there was an episode about a strike. Years later I went to Paris and guess what, grève des trains. But the train arrived just one hour late, nothing real bad.

  • (Score: 2) by drussell on Sunday November 13 2022, @08:20PM (3 children)

    by drussell (2678) on Sunday November 13 2022, @08:20PM (#1279551) Journal

    Notes for bugsquishing:

    • (uid) -> Preferences -> Comments tab shows only a blank page with slashboxes down the left side
    • Retrieve Password function gives Backend fetch failed, Guru Meditation: XID: 2656051 from Varnish cache server, state of signup system and e-mails unknown

    More to come, I'm sure... 🙂

    • (Score: 4, Interesting) by NCommander on Sunday November 13 2022, @10:05PM (2 children)

      by NCommander (2) Subscriber Badge <michael@casadevall.pro> on Sunday November 13 2022, @10:05PM (#1279570) Homepage Journal

      Password function fixed; caused by broken mail setup. The comments page is utterly bizarre. I need to look more in-depth with that.

      --
      Still always moving
      • (Score: 2) by drussell on Monday November 14 2022, @03:00AM (1 child)

        by drussell (2678) on Monday November 14 2022, @03:00AM (#1279588) Journal

        Sweet...

        Additionally:

        • Apparently the aliases file (or mechanism thereto) didn't get migrated to the currently-running postfix install, so people who had a redirect alias (instead of accessing an SN mailbox directly via IMAP or whatever) may not be getting their mail.
        • (Score: 2) by drussell on Monday November 14 2022, @04:33PM

          by drussell (2678) on Monday November 14 2022, @04:33PM (#1279690) Journal

          Scratch that... The initial report of brokenness has been rescinded by mrpg, apparently it was working all along.

          One less thing to worry about! :)

  • (Score: 1) by fab23 on Sunday November 13 2022, @09:33PM

    by fab23 (6605) Subscriber Badge on Sunday November 13 2022, @09:33PM (#1279562) Homepage Journal

    This is just a test for email

  • (Score: 2) by acid andy on Sunday November 13 2022, @11:56PM (4 children)

    by acid andy (1683) on Sunday November 13 2022, @11:56PM (#1279576) Homepage Journal

    I can't get the Search [soylentnews.org] to work at the moment. Not sure if you're already aware.

    For example if I put in Microsoft, select Stories and sort by most recent, there are no results. None of the other searches I tried worked either. Maybe some indicies need rebuilding on the database? Or something, what do I know...

    --
    Consumerism is poison.
    • (Score: 2) by mrpg on Monday November 14 2022, @12:40AM (2 children)

      by mrpg (5708) <mrpgNO@SPAMsoylentnews.org> on Monday November 14 2022, @12:40AM (#1279580) Homepage

      It gives me No stories were found that match your query.

      • (Score: 2) by acid andy on Monday November 14 2022, @01:16AM (1 child)

        by acid andy (1683) on Monday November 14 2022, @01:16AM (#1279582) Homepage Journal

        Yes that's the actual message it gives me as well, even if I search Stories for Unix Time which is in the title of the most recent story.

        --
        Consumerism is poison.
        • (Score: 2) by nostyle on Monday November 14 2022, @04:28AM

          by nostyle (11497) on Monday November 14 2022, @04:28AM (#1279596) Journal

          I noticed this earlier myself - I was trying to search for a previous comment I'd written about unix time.

          I tried again just now, and search fails for "comments" "users" "stories" and "journals". I'd guess nothing there works.

    • (Score: 1) by khallow on Friday November 18 2022, @02:21AM

      by khallow (3766) Subscriber Badge on Friday November 18 2022, @02:21AM (#1280283) Journal
      Let me second this. The SN search function is very useful for me (Google gets worse every year and it doesn't search journal comments at all). I sorely miss it.
  • (Score: 2) by DannyB on Monday November 14 2022, @03:29PM (1 child)

    by DannyB (5839) Subscriber Badge on Monday November 14 2022, @03:29PM (#1279678) Journal

    Thank you for doing a thankless job with little fanfare.

    --
    Trump is a poor man's idea of a rich man, a weak man's idea of a strong man, and a stupid man's idea of a smart man.
    • (Score: 4, Interesting) by drussell on Monday November 14 2022, @04:31PM

      by drussell (2678) on Monday November 14 2022, @04:31PM (#1279689) Journal

      Oh, there was fanfare... A couple hundred people showed up in the audience just to cheer him on. :)

      A virtually thankless job, though.... nonetheless

  • (Score: 2) by cmdrklarg on Monday November 14 2022, @07:35PM (1 child)

    by cmdrklarg (5048) Subscriber Badge on Monday November 14 2022, @07:35PM (#1279727)

    My posting history appears to have reverted back to October 27th. No idea how many went bye bye, but I do know I had posted a number of times in November.

    Thanks for the hard work!

    --
    The world is full of kings and queens who blind your eyes and steal your dreams.
    • (Score: 2) by cmdrklarg on Monday November 14 2022, @07:39PM

      by cmdrklarg (5048) Subscriber Badge on Monday November 14 2022, @07:39PM (#1279728)

      Aaaaaand I see the post from Saturday about the site borkage. Never mind!

      --
      The world is full of kings and queens who blind your eyes and steal your dreams.
(1)