Stories
Slash Boxes
Comments

SoylentNews is people

Meta
posted by martyb on Friday May 21 2021, @12:25AM   Printer-friendly

As many of you noticed, we had a site crash today. From around 1300 until 2200 UTC (2021-05-20).

A HUGE thank you goes to mechanicjay who spent the whole time trying to get our ndb (cluster) working again. It's an uncommon configuration, which made recovery especially challenging... there's just not a lot of documentation about it on the web.

I reached out and got hold of The Mighty Buzzard on the phone. Then put him in touch with mechanicjay who got us back up and running using backups.

Unfortunately, we had to go way back until April 14 to get a working backup. (I don't know all the details, but it appears something went sideways on neon).

We're all wiped out right now. When we have rested and had a chance to discuss things, we'll post an update.

In the meantime, please join me in thanking mechanicjay and TMB for all they did to get us up and running again!

 
This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 5, Informative) by janrinok on Friday May 21 2021, @08:49AM (12 children)

    by janrinok (52) Subscriber Badge on Friday May 21 2021, @08:49AM (#1137490) Journal

    It is documented - but obviously not well enough. The configuration is extremely unusual but that it in part due to the software that we inherited and the fact that many of the original team have fallen by the wayside as real world priorities interfered with their lives. Once the team gets depleted it takes all the effort of those remaining to keep the site running without taking on the tasks that, at the time, seem less important.

    I think that we have all learned one lesson already - our team of volunteers is now much too small for the work involved. We need to expand in all areas in my opinion, thus freeing up those who are already overstretched to take things at a more reasonable pace whilst ensuring that there is sufficient expertise to handle critical events such as the downtime that we have just experienced.

    So for anybody in our community who thinks that they have something to offer and is willing to give it a try, please get in contact via email or IRC and come and join us. Even if it is only for a limited duration, you might still be able to make a valuable contribution to the site.

    Starting Score:    1  point
    Moderation   +3  
       Informative=3, Total=3
    Extra 'Informative' Modifier   0  
    Karma-Bonus Modifier   +1  

    Total Score:   5  
  • (Score: 0) by Anonymous Coward on Friday May 21 2021, @09:46AM (11 children)

    by Anonymous Coward on Friday May 21 2021, @09:46AM (#1137499)

    pipedot always looked neat and modern.

    • (Score: 2) by janrinok on Friday May 21 2021, @10:26AM (10 children)

      by janrinok (52) Subscriber Badge on Friday May 21 2021, @10:26AM (#1137503) Journal

      It is based on our software so I don't see much of a benefit there.

      • (Score: 0) by Anonymous Coward on Friday May 21 2021, @11:22AM (1 child)

        by Anonymous Coward on Friday May 21 2021, @11:22AM (#1137516)
        Are you serious. Pipedot is DEAD! Hint: the date of the top story is 2017. Unless you get lucky and get the site with the top story being from 2038. And stuff about lady gaga. Looks like they’re having “issues “, people using it as a spam pot, so it’s reset to 2017-03-14 every now and then.
        • (Score: 1, Touché) by Anonymous Coward on Friday May 21 2021, @05:44PM

          by Anonymous Coward on Friday May 21 2021, @05:44PM (#1137597)

          Pipedot didn't have people like Janrinok, TMB, Martyb, and Mechanicjay.

      • (Score: 0) by Anonymous Coward on Friday May 21 2021, @11:23AM (7 children)

        by Anonymous Coward on Friday May 21 2021, @11:23AM (#1137517)

        always got the impression it was newer base code.

        http://pipedot.org/story/2014-02-17/pipedot-status-week-1 [pipedot.org]

        but the consensus here appeared to be SN landed on better PR and reader/dev group.

        i bow to your stronger knowledge and also add my thanks for the SN teams efforts to keep the site going.

        • (Score: 0) by Anonymous Coward on Friday May 21 2021, @11:42AM

          by Anonymous Coward on Friday May 21 2021, @11:42AM (#1137521)
          It’s dead, Jim. Currently the top story is from April 2017. Earlier, the top story was from 2038. With a bunch of junk from yesterday. The site is in demo mode, with a script resetting it to the last valid post from when it was still active. Just before it reset, there was page after page of junk like Lady Gaga.

          The wayback machine has a last snapshot from January 2nd 2018, with the April 2017 post being the last one. So it’s dead.

        • (Score: 5, Informative) by janrinok on Friday May 21 2021, @01:57PM (5 children)

          by janrinok (52) Subscriber Badge on Friday May 21 2021, @01:57PM (#1137533) Journal

          As best I can recollect from over 6 years ago: Soylent and Pipedot (Bryan) cooperated together in a friendly manner to produce something to replace slashdot.beta, but following 2 different paths. The idea was that if one project should fail the other would still be around to continue. Pipedot uses a lot of new code written in PHP. Our devs upgraded Slash to the then current libs and released it as Rehash on git from where anybody is free to use it. We both went operational and for a time it seemed that we were likely to compete against each other for a community. I think we had a slight head start and Pipedot decided that it made no sense for us to be fighting for the same readership. Pipedot was put into a sleeping mode, hence the reason why the site is still live but dormant. It could be restarted if Soylent should fail.

          I am a bit hazy on the details but that is roughly how I remember it.

          See https://soylentnews.org/article.pl?sid=19/08/19/0814218 [soylentnews.org] and https://github.com/pipedot/pipecode [github.com].

          • (Score: 5, Informative) by bryan on Friday May 21 2021, @05:09PM (4 children)

            by bryan (29) <bryan@pipedot.org> on Friday May 21 2021, @05:09PM (#1137589) Homepage Journal

            Nice summary :) I was wondering what happened yesterday when I tried to click on an article and just got errors. Loosing a month of data sucks, but I see that not as a fault in Rehash but as the infrastructure being overly complex for the current need. Reverting MySQL to a simple and more common configuration will help greatly.

            As for my own site: still sleeping! Occasionally, some spam messages do get posted (common in all internet forums) - but the moderation system hides those once marked (unless you choose to see them in your preferences.)

            • (Score: 2) by janrinok on Friday May 21 2021, @05:33PM

              by janrinok (52) Subscriber Badge on Friday May 21 2021, @05:33PM (#1137594) Journal

              I was wondering if you were still around, Bryan! Good to 'see' you again.

              Yeah, it is a system config problem and mechanicjay is looking at the best way to remove part of our setup completely. It was intended to give us redundancy and resilience but has proven to be the cause of most of our system downtime.

              Rehash is working fine but it is getting increasingly difficult to find people to maintain it. We are looking at various options but the priority is the get the site stable again so we can approach any long-term changes in a calm and methodical manner.

            • (Score: 2) by Reziac on Saturday May 22 2021, @02:33AM (2 children)

              by Reziac (2489) on Saturday May 22 2021, @02:33AM (#1137667) Homepage

              Hey there! sleeping is okay; I'd be very disappointed if it were dead and gone. It seems to work well enough, and the day may come when it's needed awake again.

              --
              And there is no Alkibiades to come back and save us from ourselves.
              • (Score: 3, Funny) by kazzie on Sunday May 23 2021, @08:02AM (1 child)

                by kazzie (5309) Subscriber Badge on Sunday May 23 2021, @08:02AM (#1137914)

                Now I'm getting crossover between Pipedot and the legend of King Arthur...

                • (Score: 3, Funny) by Reziac on Sunday May 23 2021, @02:32PM

                  by Reziac (2489) on Sunday May 23 2021, @02:32PM (#1137957) Homepage

                  Zombie fora are everywhere. :D

                  --
                  And there is no Alkibiades to come back and save us from ourselves.