Stories
Slash Boxes
Comments

SoylentNews is people

Meta
posted by martyb on Sunday June 13, @04:50PM   Printer-friendly

[2021-06-14 02:24:41 UTC; update 2: We made a decision to accept Linode's offer of moving up our migration of fluorine. It appears the migration has completed successfully. YAY!]


[2021-06-14 00:25:32 UTC; update 1: hydrogen appears to have successfully migrated. We had a brief 503 on the site until I bounced varnish. The site seems to be fine, now.--Bytram]


Prologue:
First off please accept my sincere wish for a happy Father's Day to all our dads in the community! (It is celebrated next Sunday in 90 countries.)

Also, I am happy to report a surge in participation on the site over the past month. I've seen increases in story submissions, subscriptions[*], and participation (comments, moderations, etc.) Community++

[*] NB: I was successful in crediting users for their subscriptions on the site after the server crash. Unfortunately, that failed to account for the dollar amount of their subscriptions in our tracking database table which is used to source our progress against our funding goal. I have a plan for getting those updates in place, but want to run it past other members of staff to make sure everything is accounted for before making any changes.

Read on for the rest of the site's news, or just wait and a new story will be out before too long.

Server Migrations:
We have received word that Linode, our web-hosting provider, will be conducting maintenance on two of our servers in the next 24 hours.

Last night Linode shut down one of our servers (boron), migrated the disk image to a new physical server, and restarted it. All seems to have gone smoothly.

Later on today, two more of our servers are due to be migrated:

  • hydrogen (alternate database server) 4 CPU Cores, 160 GB Storage, 8 GB RAM
    This Linode's physical host will be undergoing maintenance at 2021-06-14 00:00 UTC. During this time, your Linode will be shut down, cold migrated to a new host, then returned to its last state (running or powered off)
  • fluorine (primary database server) 4 CPU Cores, 96 GB Storage, 8 GB RAM
    This Linode's physical host will be undergoing maintenance at 2021-06-14 07:00 UTC. During this time, your Linode will be shut down, cold migrated to a new host, then returned to its last state (running or powered off).

Also of note, we are eligible for a free storage upgrade on fluorine from 96 GB to 160 GB. It is not clear at this moment if we will also conduct the storage upgrade at this time.

Cert Updates:
Our certs (issued by Let's Encrypt) are due to expire June 17, 2021.

We are aware and intend to have updated certs installed before then.

(NB: I may have some terminology errors in what follows, but I believe the overall process/concepts should be correct.)

I have personally installed updated certs twice before on our servers, and if need be, am prepared to do so again. It has been a couple years or so but the process should remain largely the same. The majority of the steps are automated, but historically we've preferred to handle the DNS updates manually. That way, just in case something goes sideways, we are hands-on and can take steps to mitigate problems... instead of finding we have a botched DNS and greatly restricted access the servers. (That is a bit of an overstatement, but as I understand it, it's a lot easier to make changes over SSH connections to running servers than through a console port to one server at a time.)

Also, there has been discussion about using a fully-automated Let's Encrypt cert update process, we'll keep you posted.

Site News:
Behind the scenes we've been hard at work. juggs, mechanicjay, and audioguy have put in many long and thankless hours stabilizing and documenting our service infrastructure. They've made great strides and we continue to make progress. We cannot change what was done (and not done) in the past, but we can learn from it! What services "live" on what servers? How to restart each service? Monitoring of disk usage and CPU usage? All are gradually being documented and site operations knowledge is getting shared all around.

Lastly, here's a shout-out to the editorial staff who strive to keep stories coming to you 24/7. Fnord666 just posted his 6,500th story! Also, thanks to janrinok, mrpg, chromas, and FatPhil who have all pushed out stories this past month! Teamwork++!

[N.B. Let's not forget our Editor-In-Chief martyb, who just posted his 10,100th story! This is in addition to serving as our primary QA person. - Fnord]


Original Submission

 
This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 0) by Anonymous Coward on Monday June 14, @12:35PM (3 children)

    by Anonymous Coward on Monday June 14, @12:35PM (#1145043)

    Anything modern will run, easily. I just tried a random forum (satellity) written in golang, got it running quickly.

  • (Score: 3, Insightful) by maxwell demon on Monday June 14, @01:25PM (1 child)

    by maxwell demon (1608) Subscriber Badge on Monday June 14, @01:25PM (#1145051) Journal

    So how long did it take you to migrate a large amount of users and content from a Rehash instance to it? Well, let me guess: You didn't even try.

    And how well did it withstand heavy load? Well, let me guess: You didn't test that one either.

    --
    The Tao of math: The numbers you can count are not the real numbers.
    • (Score: 0) by Anonymous Coward on Monday June 14, @02:01PM

      by Anonymous Coward on Monday June 14, @02:01PM (#1145058)

      Of course not, I don't know the project, and there is basically no documentation, so why would I try that? I'm not a masochist.

  • (Score: 0) by Anonymous Coward on Tuesday June 15, @06:13AM

    by Anonymous Coward on Tuesday June 15, @06:13AM (#1145402)

    All the choices and you went to Satellity? All of the choices and you picked one that almost doesn't exist and had almost zero features but happens to be written in Go. Interesting. I am curious how long your static build of PostgreSQL took. You did do one, right?