Stories
Slash Boxes
Comments

SoylentNews is people

Meta
posted by martyb on Sunday June 13, @04:50PM   Printer-friendly [Skip to comment(s)]

[2021-06-14 02:24:41 UTC; update 2: We made a decision to accept Linode's offer of moving up our migration of fluorine. It appears the migration has completed successfully. YAY!]


[2021-06-14 00:25:32 UTC; update 1: hydrogen appears to have successfully migrated. We had a brief 503 on the site until I bounced varnish. The site seems to be fine, now.--Bytram]


Prologue:
First off please accept my sincere wish for a happy Father's Day to all our dads in the community! (It is celebrated next Sunday in 90 countries.)

Also, I am happy to report a surge in participation on the site over the past month. I've seen increases in story submissions, subscriptions[*], and participation (comments, moderations, etc.) Community++

[*] NB: I was successful in crediting users for their subscriptions on the site after the server crash. Unfortunately, that failed to account for the dollar amount of their subscriptions in our tracking database table which is used to source our progress against our funding goal. I have a plan for getting those updates in place, but want to run it past other members of staff to make sure everything is accounted for before making any changes.

Read on for the rest of the site's news, or just wait and a new story will be out before too long.

Server Migrations:
We have received word that Linode, our web-hosting provider, will be conducting maintenance on two of our servers in the next 24 hours.

Last night Linode shut down one of our servers (boron), migrated the disk image to a new physical server, and restarted it. All seems to have gone smoothly.

Later on today, two more of our servers are due to be migrated:

  • hydrogen (alternate database server) 4 CPU Cores, 160 GB Storage, 8 GB RAM
    This Linode's physical host will be undergoing maintenance at 2021-06-14 00:00 UTC. During this time, your Linode will be shut down, cold migrated to a new host, then returned to its last state (running or powered off)
  • fluorine (primary database server) 4 CPU Cores, 96 GB Storage, 8 GB RAM
    This Linode's physical host will be undergoing maintenance at 2021-06-14 07:00 UTC. During this time, your Linode will be shut down, cold migrated to a new host, then returned to its last state (running or powered off).

Also of note, we are eligible for a free storage upgrade on fluorine from 96 GB to 160 GB. It is not clear at this moment if we will also conduct the storage upgrade at this time.

Cert Updates:
Our certs (issued by Let's Encrypt) are due to expire June 17, 2021.

We are aware and intend to have updated certs installed before then.

(NB: I may have some terminology errors in what follows, but I believe the overall process/concepts should be correct.)

I have personally installed updated certs twice before on our servers, and if need be, am prepared to do so again. It has been a couple years or so but the process should remain largely the same. The majority of the steps are automated, but historically we've preferred to handle the DNS updates manually. That way, just in case something goes sideways, we are hands-on and can take steps to mitigate problems... instead of finding we have a botched DNS and greatly restricted access the servers. (That is a bit of an overstatement, but as I understand it, it's a lot easier to make changes over SSH connections to running servers than through a console port to one server at a time.)

Also, there has been discussion about using a fully-automated Let's Encrypt cert update process, we'll keep you posted.

Site News:
Behind the scenes we've been hard at work. juggs, mechanicjay, and audioguy have put in many long and thankless hours stabilizing and documenting our service infrastructure. They've made great strides and we continue to make progress. We cannot change what was done (and not done) in the past, but we can learn from it! What services "live" on what servers? How to restart each service? Monitoring of disk usage and CPU usage? All are gradually being documented and site operations knowledge is getting shared all around.

Lastly, here's a shout-out to the editorial staff who strive to keep stories coming to you 24/7. Fnord666 just posted his 6,500th story! Also, thanks to janrinok, mrpg, chromas, and FatPhil who have all pushed out stories this past month! Teamwork++!

[N.B. Let's not forget our Editor-In-Chief martyb, who just posted his 10,100th story! This is in addition to serving as our primary QA person. - Fnord]


Original Submission

This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
(1)
  • (Score: 2, Interesting) by Anonymous Coward on Sunday June 13, @05:11PM (1 child)

    by Anonymous Coward on Sunday June 13, @05:11PM (#1144828)

    Thanks for the update and double thanks to all the SN volunteers!

    I've also noticed that the queue has been plenty full lately. My AC subs were getting picked up nearly every time...up until recently. I view this as good news, since the eds have a better pool to select from.

    • (Score: -1, Flamebait) by Anonymous Coward on Sunday June 13, @07:17PM

      by Anonymous Coward on Sunday June 13, @07:17PM (#1144852)

      Steak and a blowjob to all the staff - just ask your wives.

  • (Score: 4, Funny) by sgleysti on Sunday June 13, @05:56PM (18 children)

    by sgleysti (56) on Sunday June 13, @05:56PM (#1144832)

    Nice work.

    I'm just waiting for that one poster who insists you migrate soylentnews from slashcode to geeklog to start multiple independent threads with that suggestion under this article.

    • (Score: 1, Funny) by Anonymous Coward on Sunday June 13, @06:10PM

      by Anonymous Coward on Sunday June 13, @06:10PM (#1144839)

      I insist that you migrate to geeklog. So there.

    • (Score: 3, Funny) by martyb on Sunday June 13, @06:16PM (3 children)

      by martyb (76) Subscriber Badge on Sunday June 13, @06:16PM (#1144842) Journal

      Wait, what? UID 56!?!!

      Hadn't noticed you post much, but glad to see you're still around... 7 years and counting!!

      Hope you hang around for many more!

      --
      Wit is intellect, dancing.
      • (Score: 5, Insightful) by sgleysti on Sunday June 13, @07:36PM (2 children)

        by sgleysti (56) on Sunday June 13, @07:36PM (#1144862)

        Took a hiatus on commenting for a while, but I still remember hanging out on IRC during the "fuck beta" days. Good times.

        • (Score: 0) by Anonymous Coward on Monday June 14, @04:13AM (1 child)

          by Anonymous Coward on Monday June 14, @04:13AM (#1144956)

          With that low ID you practically invented the site and wrote all the comments. Gotta worship the low IDs.

    • (Score: 1, Insightful) by Anonymous Coward on Sunday June 13, @06:40PM (12 children)

      by Anonymous Coward on Sunday June 13, @06:40PM (#1144846)

      sorry, gave up on you guys a while ago. And it wasn't just suggesting a particular CMS - any other one would be a better choice than rehash. Looking at the current server specs, it's obvious just how much of a hash rehash is.

      I hope you succeed, and the recent purge has made that more likely , but at some point you ate going to have to start over from scratch anyway. There's just too much technical debt to even risk trying to add any significant new features. Or even optimize what's there mow. You're significantly overprovisioned for a simple web site.

      • (Score: 5, Insightful) by martyb on Sunday June 13, @07:36PM (11 children)

        by martyb (76) Subscriber Badge on Sunday June 13, @07:36PM (#1144861) Journal

        We realize we are over-provisioned and server consolidation is one of the things we are looking at. We could conceivably run on a single server with a commensurate reduction in flexibility and responsiveness.

        Also, do be aware that what you see has survived decades at the hands of creative and persistent users. I know of no other platform that can make the same claim.

        Further, finding a way to port 1.1 million comments (and their moderations) as well as ~60K stories to any new platform brings a whole slew of issues of their own.

        Lastly, what is presented to the community is only about 10-15% of the code that drives this site... there's a lot more under the covers than what you see here. Any projection as to what it takes to run this site, without having actually done so, borders on Dunning-Kruger. I know when I started here I was absolutely astonished at all the customization and admin tools that exists "under the covers".

        --
        Wit is intellect, dancing.
        • (Score: 5, Informative) by bryan on Sunday June 13, @08:14PM (2 children)

          by bryan (29) <bryan@pipedot.org> on Sunday June 13, @08:14PM (#1144867) Homepage Journal

          Im running on the "2 GB RAM, 1 Core, 50 GB SSD, $10 / mo" plan, which seems to be the best value currently. Linode is pretty linear with it pricing, but I guess they tweaked a few knobs down near the bottom to match Digital Ocean. I also recently made use of their separate block storage ($1 per 10 GB) for article thumbnails, since RSS feeds and external article links have those.

          I wish LetsEncrypt used their easy HTTP-based check for wildcard domains. The DNS-based check is kinda annoying. I just renewed it manually yesterday, and it only takes a minute or two, but the dozens of single domain certs are so much easier since they are automatic.

          • (Score: 3, Interesting) by bryan on Sunday June 13, @08:26PM

            by bryan (29) <bryan@pipedot.org> on Sunday June 13, @08:26PM (#1144871) Homepage Journal

            BTW, Linode's recent upgrade changed my host CPU from a "Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz" to a "AMD EPYC 7601 32-Core Processor"

            Those pesky 2016 Xeon's [intel.com] with a "mere" 18 cores! Of course, those where 14nm just like today's Xeons. I suppose that's why they are moving to the more core-rich AMD [wikichip.org] systems. My other Linodes had already been converted to AMD a few years ago.

          • (Score: 1, Informative) by Anonymous Coward on Monday June 14, @04:30AM

            by Anonymous Coward on Monday June 14, @04:30AM (#1144964)

            The problem with using HTTP or ALPN for the wildcard is that there isn't really a good way to show you own the entire subdomain space without using the DNS check.

        • (Score: 0) by Anonymous Coward on Monday June 14, @02:10AM (7 children)

          by Anonymous Coward on Monday June 14, @02:10AM (#1144930)

          Have you looked into unikernels? https://github.com/nanovms/ops [github.com] should work for perl... might help to save some money?

          • (Score: 0) by Anonymous Coward on Monday June 14, @04:22AM (6 children)

            by Anonymous Coward on Monday June 14, @04:22AM (#1144958)

            Their stack is too complex to run on something like that and I doubt they have the acumen or time to port to it (FWIW not a snipe, I can't think of a single person who could that would not consider it a complete waste of their time on it) even if it would make a difference compared to all the other things they could do to improve the situation.

            • (Score: 0) by Anonymous Coward on Monday June 14, @05:03AM (5 children)

              by Anonymous Coward on Monday June 14, @05:03AM (#1144974)

              You're probably right, which just makes the argument for a new CMS stronger. Ugly but useful legacy codebases like slash are rapidly losing their staying power as the cloud computing model matures and more modern languages come of age.

              • (Score: 1, Insightful) by Anonymous Coward on Monday June 14, @07:30AM (4 children)

                by Anonymous Coward on Monday June 14, @07:30AM (#1144999)

                And what CMS do you think can run on a unikernel like that? I'm genuinely curious how much of this is speaking from experience and how much is just puffery.

                • (Score: 0) by Anonymous Coward on Monday June 14, @12:35PM (3 children)

                  by Anonymous Coward on Monday June 14, @12:35PM (#1145043)

                  Anything modern will run, easily. I just tried a random forum (satellity) written in golang, got it running quickly.

                  • (Score: 3, Insightful) by maxwell demon on Monday June 14, @01:25PM (1 child)

                    by maxwell demon (1608) on Monday June 14, @01:25PM (#1145051) Journal

                    So how long did it take you to migrate a large amount of users and content from a Rehash instance to it? Well, let me guess: You didn't even try.

                    And how well did it withstand heavy load? Well, let me guess: You didn't test that one either.

                    --
                    The Tao of math: The numbers you can count are not the real numbers.
                    • (Score: 0) by Anonymous Coward on Monday June 14, @02:01PM

                      by Anonymous Coward on Monday June 14, @02:01PM (#1145058)

                      Of course not, I don't know the project, and there is basically no documentation, so why would I try that? I'm not a masochist.

                  • (Score: 0) by Anonymous Coward on Tuesday June 15, @06:13AM

                    by Anonymous Coward on Tuesday June 15, @06:13AM (#1145402)

                    All the choices and you went to Satellity? All of the choices and you picked one that almost doesn't exist and had almost zero features but happens to be written in Go. Interesting. I am curious how long your static build of PostgreSQL took. You did do one, right?

  • (Score: 4, Funny) by Anonymous Coward on Sunday June 13, @06:00PM (6 children)

    by Anonymous Coward on Sunday June 13, @06:00PM (#1144833)

    "Father's Day" is a sexist term that has no place in a progressive site like SoylentNews. The correct term is "Inseminator's Day" to recognize the tiny part played in providing a bit of DNA.

    • (Score: 1, Funny) by Anonymous Coward on Sunday June 13, @06:08PM (4 children)

      by Anonymous Coward on Sunday June 13, @06:08PM (#1144836)

      Progressive? All I hear is how we are a bunch of right-wing sock-puppets and bigots. Make up your mind.

      • (Score: 1, Insightful) by Anonymous Coward on Sunday June 13, @06:16PM (2 children)

        by Anonymous Coward on Sunday June 13, @06:16PM (#1144841)

        Iron Knee?

        • (Score: 2) by kazzie on Monday June 14, @05:28AM

          by kazzie (5309) Subscriber Badge on Monday June 14, @05:28AM (#1144982)

          No, but I need to iron a shirt this morning.

        • (Score: 2) by DannyB on Monday June 14, @06:33PM

          by DannyB (5839) Subscriber Badge on Monday June 14, @06:33PM (#1145164) Journal

          Is iron knee a weapon used to attack people using the knee?

          Like brass knuckles?

          --
          Employers should not mandate wearing clothing. It should be a personal choice. It only affects me. Junk can't breathe!
      • (Score: 0) by Anonymous Coward on Sunday June 13, @07:20PM

        by Anonymous Coward on Sunday June 13, @07:20PM (#1144854)

        I am an independent libertarian, you are a SJW, they are frothing at the mouth Trumptards.

    • (Score: 0) by Anonymous Coward on Monday June 14, @02:40AM

      by Anonymous Coward on Monday June 14, @02:40AM (#1144939)

      matry-b, eh.

  • (Score: 3, Interesting) by The Vocal Minority on Monday June 14, @05:51AM

    by The Vocal Minority (2765) on Monday June 14, @05:51AM (#1144986) Journal

    Thanks everyone!

  • (Score: 1) by khallow on Monday June 14, @02:20PM

    by khallow (3766) Subscriber Badge on Monday June 14, @02:20PM (#1145064) Journal

    We made a decision to accept Linode's offer of moving up our migration of fluorine.

    Why in the world are you exporting the pollution to innocent third world countries, especially of a dangerous chemical like fluorine? And getting an evil US corporation to assist in this crime?! This is horrible.

    Just do it in New Jersey. Nobody will notice a little extra something in the water.

  • (Score: 0) by Anonymous Coward on Monday June 14, @03:24PM

    by Anonymous Coward on Monday June 14, @03:24PM (#1145089)

    Including the "we want you to migrate all this, and history to x"

    Gtg.. pstd work flashbacksr rising...

(1)