Stories
Slash Boxes
Comments

SoylentNews is people

Meta
posted by martyb on Sunday June 13 2021, @04:50PM   Printer-friendly

[2021-06-14 02:24:41 UTC; update 2: We made a decision to accept Linode's offer of moving up our migration of fluorine. It appears the migration has completed successfully. YAY!]


[2021-06-14 00:25:32 UTC; update 1: hydrogen appears to have successfully migrated. We had a brief 503 on the site until I bounced varnish. The site seems to be fine, now.--Bytram]


Prologue:
First off please accept my sincere wish for a happy Father's Day to all our dads in the community! (It is celebrated next Sunday in 90 countries.)

Also, I am happy to report a surge in participation on the site over the past month. I've seen increases in story submissions, subscriptions[*], and participation (comments, moderations, etc.) Community++

[*] NB: I was successful in crediting users for their subscriptions on the site after the server crash. Unfortunately, that failed to account for the dollar amount of their subscriptions in our tracking database table which is used to source our progress against our funding goal. I have a plan for getting those updates in place, but want to run it past other members of staff to make sure everything is accounted for before making any changes.

Read on for the rest of the site's news, or just wait and a new story will be out before too long.

Server Migrations:
We have received word that Linode, our web-hosting provider, will be conducting maintenance on two of our servers in the next 24 hours.

Last night Linode shut down one of our servers (boron), migrated the disk image to a new physical server, and restarted it. All seems to have gone smoothly.

Later on today, two more of our servers are due to be migrated:

  • hydrogen (alternate database server) 4 CPU Cores, 160 GB Storage, 8 GB RAM
    This Linode's physical host will be undergoing maintenance at 2021-06-14 00:00 UTC. During this time, your Linode will be shut down, cold migrated to a new host, then returned to its last state (running or powered off)
  • fluorine (primary database server) 4 CPU Cores, 96 GB Storage, 8 GB RAM
    This Linode's physical host will be undergoing maintenance at 2021-06-14 07:00 UTC. During this time, your Linode will be shut down, cold migrated to a new host, then returned to its last state (running or powered off).

Also of note, we are eligible for a free storage upgrade on fluorine from 96 GB to 160 GB. It is not clear at this moment if we will also conduct the storage upgrade at this time.

Cert Updates:
Our certs (issued by Let's Encrypt) are due to expire June 17, 2021.

We are aware and intend to have updated certs installed before then.

(NB: I may have some terminology errors in what follows, but I believe the overall process/concepts should be correct.)

I have personally installed updated certs twice before on our servers, and if need be, am prepared to do so again. It has been a couple years or so but the process should remain largely the same. The majority of the steps are automated, but historically we've preferred to handle the DNS updates manually. That way, just in case something goes sideways, we are hands-on and can take steps to mitigate problems... instead of finding we have a botched DNS and greatly restricted access the servers. (That is a bit of an overstatement, but as I understand it, it's a lot easier to make changes over SSH connections to running servers than through a console port to one server at a time.)

Also, there has been discussion about using a fully-automated Let's Encrypt cert update process, we'll keep you posted.

Site News:
Behind the scenes we've been hard at work. juggs, mechanicjay, and audioguy have put in many long and thankless hours stabilizing and documenting our service infrastructure. They've made great strides and we continue to make progress. We cannot change what was done (and not done) in the past, but we can learn from it! What services "live" on what servers? How to restart each service? Monitoring of disk usage and CPU usage? All are gradually being documented and site operations knowledge is getting shared all around.

Lastly, here's a shout-out to the editorial staff who strive to keep stories coming to you 24/7. Fnord666 just posted his 6,500th story! Also, thanks to janrinok, mrpg, chromas, and FatPhil who have all pushed out stories this past month! Teamwork++!

[N.B. Let's not forget our Editor-In-Chief martyb, who just posted his 10,100th story! This is in addition to serving as our primary QA person. - Fnord]


Original Submission

 
This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 1, Insightful) by Anonymous Coward on Sunday June 13 2021, @06:40PM (12 children)

    by Anonymous Coward on Sunday June 13 2021, @06:40PM (#1144846)

    sorry, gave up on you guys a while ago. And it wasn't just suggesting a particular CMS - any other one would be a better choice than rehash. Looking at the current server specs, it's obvious just how much of a hash rehash is.

    I hope you succeed, and the recent purge has made that more likely , but at some point you ate going to have to start over from scratch anyway. There's just too much technical debt to even risk trying to add any significant new features. Or even optimize what's there mow. You're significantly overprovisioned for a simple web site.

    Starting Score:    0  points
    Moderation   +1  
       Insightful=2, Overrated=1, Total=3
    Extra 'Insightful' Modifier   0  

    Total Score:   1  
  • (Score: 5, Insightful) by martyb on Sunday June 13 2021, @07:36PM (11 children)

    by martyb (76) Subscriber Badge on Sunday June 13 2021, @07:36PM (#1144861) Journal

    We realize we are over-provisioned and server consolidation is one of the things we are looking at. We could conceivably run on a single server with a commensurate reduction in flexibility and responsiveness.

    Also, do be aware that what you see has survived decades at the hands of creative and persistent users. I know of no other platform that can make the same claim.

    Further, finding a way to port 1.1 million comments (and their moderations) as well as ~60K stories to any new platform brings a whole slew of issues of their own.

    Lastly, what is presented to the community is only about 10-15% of the code that drives this site... there's a lot more under the covers than what you see here. Any projection as to what it takes to run this site, without having actually done so, borders on Dunning-Kruger. I know when I started here I was absolutely astonished at all the customization and admin tools that exists "under the covers".

    --
    Wit is intellect, dancing.
    • (Score: 5, Informative) by bryan on Sunday June 13 2021, @08:14PM (2 children)

      by bryan (29) <bryan@pipedot.org> on Sunday June 13 2021, @08:14PM (#1144867) Homepage Journal

      Im running on the "2 GB RAM, 1 Core, 50 GB SSD, $10 / mo" plan, which seems to be the best value currently. Linode is pretty linear with it pricing, but I guess they tweaked a few knobs down near the bottom to match Digital Ocean. I also recently made use of their separate block storage ($1 per 10 GB) for article thumbnails, since RSS feeds and external article links have those.

      I wish LetsEncrypt used their easy HTTP-based check for wildcard domains. The DNS-based check is kinda annoying. I just renewed it manually yesterday, and it only takes a minute or two, but the dozens of single domain certs are so much easier since they are automatic.

      • (Score: 3, Interesting) by bryan on Sunday June 13 2021, @08:26PM

        by bryan (29) <bryan@pipedot.org> on Sunday June 13 2021, @08:26PM (#1144871) Homepage Journal

        BTW, Linode's recent upgrade changed my host CPU from a "Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz" to a "AMD EPYC 7601 32-Core Processor"

        Those pesky 2016 Xeon's [intel.com] with a "mere" 18 cores! Of course, those where 14nm just like today's Xeons. I suppose that's why they are moving to the more core-rich AMD [wikichip.org] systems. My other Linodes had already been converted to AMD a few years ago.

      • (Score: 1, Informative) by Anonymous Coward on Monday June 14 2021, @04:30AM

        by Anonymous Coward on Monday June 14 2021, @04:30AM (#1144964)

        The problem with using HTTP or ALPN for the wildcard is that there isn't really a good way to show you own the entire subdomain space without using the DNS check.

    • (Score: 0) by Anonymous Coward on Monday June 14 2021, @02:10AM (7 children)

      by Anonymous Coward on Monday June 14 2021, @02:10AM (#1144930)

      Have you looked into unikernels? https://github.com/nanovms/ops [github.com] should work for perl... might help to save some money?

      • (Score: 0) by Anonymous Coward on Monday June 14 2021, @04:22AM (6 children)

        by Anonymous Coward on Monday June 14 2021, @04:22AM (#1144958)

        Their stack is too complex to run on something like that and I doubt they have the acumen or time to port to it (FWIW not a snipe, I can't think of a single person who could that would not consider it a complete waste of their time on it) even if it would make a difference compared to all the other things they could do to improve the situation.

        • (Score: 0) by Anonymous Coward on Monday June 14 2021, @05:03AM (5 children)

          by Anonymous Coward on Monday June 14 2021, @05:03AM (#1144974)

          You're probably right, which just makes the argument for a new CMS stronger. Ugly but useful legacy codebases like slash are rapidly losing their staying power as the cloud computing model matures and more modern languages come of age.

          • (Score: 1, Insightful) by Anonymous Coward on Monday June 14 2021, @07:30AM (4 children)

            by Anonymous Coward on Monday June 14 2021, @07:30AM (#1144999)

            And what CMS do you think can run on a unikernel like that? I'm genuinely curious how much of this is speaking from experience and how much is just puffery.

            • (Score: 0) by Anonymous Coward on Monday June 14 2021, @12:35PM (3 children)

              by Anonymous Coward on Monday June 14 2021, @12:35PM (#1145043)

              Anything modern will run, easily. I just tried a random forum (satellity) written in golang, got it running quickly.

              • (Score: 3, Insightful) by maxwell demon on Monday June 14 2021, @01:25PM (1 child)

                by maxwell demon (1608) on Monday June 14 2021, @01:25PM (#1145051) Journal

                So how long did it take you to migrate a large amount of users and content from a Rehash instance to it? Well, let me guess: You didn't even try.

                And how well did it withstand heavy load? Well, let me guess: You didn't test that one either.

                --
                The Tao of math: The numbers you can count are not the real numbers.
                • (Score: 0) by Anonymous Coward on Monday June 14 2021, @02:01PM

                  by Anonymous Coward on Monday June 14 2021, @02:01PM (#1145058)

                  Of course not, I don't know the project, and there is basically no documentation, so why would I try that? I'm not a masochist.

              • (Score: 0) by Anonymous Coward on Tuesday June 15 2021, @06:13AM

                by Anonymous Coward on Tuesday June 15 2021, @06:13AM (#1145402)

                All the choices and you went to Satellity? All of the choices and you picked one that almost doesn't exist and had almost zero features but happens to be written in Go. Interesting. I am curious how long your static build of PostgreSQL took. You did do one, right?