Stories
Slash Boxes
Comments

SoylentNews is people

Submission Preview

Link to Story

Site Maintenance -- Server Migrations, Possible Storage Upgrade, Cert Updates, and More!

Accepted submission by martyb at 2021-06-13 15:37:17
News
Prologue:
First off please accept my sincere wish for a happy Father's Day [wikipedia.org] to all our dads in the community! (It is celebrated next Sunday in 90 countries .)

Also, I am happy to report a surge in participation on the site over the past month. I've seen increases in story submissions [soylentnews.org], subscriptions[*] [soylentnews.org], and participation (comments, moderations, etc.) on the site!

[*] NB: I was successful in crediting users for their subscriptions on the site after the server crash. Unfortunately, that failed to account for the dollar amount of their subscriptions in our tracking table which it used to source our progress against our funding goal. I have a plan for getting those updates in place, but want to run it past other members oo staff to make sure everything is accounted for before making any changes.

Read on for the rest of the site's news, or just wait and a new story will be out before too long.

=====EXTENDED COPY=====

Server Migrations:
We have received word that Linode, our web-hosting provider, will be conducting maintenance on two of our servers in the next 24 hours.

Last night Linode shut down one of our servers (boron), migrated the disk image to a new physical server, and restarted it. All seems to have gone smoothly.

Later on today, two more of our servers are due to be migrated:

  • hydrogen (alternate database server) 4 CPU Cores, 160 GB Storage, 8 GB RAM
    This Linode's physical host will be undergoing maintenance at 2021-06-14 00:00 UTC. During this time, your Linode will be shut down, cold migrated to a new host, then returned to its last state (running or powered off)
  • fluorine (primary database server) 4 CPU Cores, 96 GB Storage, 8 GB RAM
    This Linode's physical host will be undergoing maintenance at 2021-06-14 07:00 UTC. During this time, your Linode will be shut down, cold migrated to a new host, then returned to its last state (running or powered off).

Also of note, we are eligible for a free storage upgrade on fluorine from 96 GB to 160 GB. It is not clear at this moment if we will also conduct the storage upgrade at this time.

Cert Updates:
Our certs (issued by Let's Encrypt) are due to expire June 17, 2021.

We are aware and intend to have updated certs installed before then

(NB: I may have some terminology errors in what follows, but I believe the overriding process/concepts should be correct.)

I have personally installed updated certs twice before, and if need be, am prepared to do so again. It has been a couple years or so and, for the most part, the process should remain the same. The majority of the steps are automated, but historically we've preferred to handle the DNS updates manually. That way, just in case something goes sideways, we are hands-on and can take steps to mitigate problems... instead of finding we have a botched DNS and greatly restricted access the servers. (That is a bit of an overstatement, but as I understand it, it's a lot easier to make changes over a secure SSH connection to running servers than through a console port to one server at a time.)

Also, there has been discussion about using Let's Encrypt fully-automated cert update process, we'll keep you posted.

Site News:
Behind the scenes we've been hard at work. juggs, mechanicjay, and audioguy have put in many long and thankless hours stabilizing and documenting our service infrastructure. They've made great strides and we continue to make progress. We cannot change what was done (and not done) in the past, but we can learn from it! What services "live" on what servers? How to restart each service? Monitoring of disk usage and CPU usage? All are gradually being documented and site operations knowledge is getting shared all around.


Original Submission