Stories
Slash Boxes
Comments

SoylentNews is people

Meta

Log In

Log In

Create Account  |  Retrieve Password


Site News

Join our Folding@Home team:
Main F@H site
Our team page


Funding Goal
For 6-month period:
2022-07-01 to 2022-12-31
(All amounts are estimated)
Base Goal:
$3500.00

Currently:
$438.92

12.5%

Covers transactions:
2022-07-02 10:17:28 ..
2022-10-05 12:33:58 UTC
(SPIDs: [1838..1866])
Last Update:
2022-10-05 14:04:11 UTC --fnord666

Support us: Subscribe Here
and buy SoylentNews Swag


We always have a place for talented people, visit the Get Involved section on the wiki to see how you can make SoylentNews better.

posted by martyb on Thursday June 17 2021, @11:19AM   Printer-friendly

Last night (actually, very early this morning) mechanicjay generated and installed new Let's Encrypt certs for our servers.

I made a quick check and everything seems to be in place. The old certs were due to expire right about now, so if you do have any issues, please pop onto IRC (preferred) or reply here and let us know!

Thanks mechanicjay!


Original Submission

posted by martyb on Sunday June 13 2021, @04:50PM   Printer-friendly

[2021-06-14 02:24:41 UTC; update 2: We made a decision to accept Linode's offer of moving up our migration of fluorine. It appears the migration has completed successfully. YAY!]


[2021-06-14 00:25:32 UTC; update 1: hydrogen appears to have successfully migrated. We had a brief 503 on the site until I bounced varnish. The site seems to be fine, now.--Bytram]


Prologue:
First off please accept my sincere wish for a happy Father's Day to all our dads in the community! (It is celebrated next Sunday in 90 countries.)

Also, I am happy to report a surge in participation on the site over the past month. I've seen increases in story submissions, subscriptions[*], and participation (comments, moderations, etc.) Community++

[*] NB: I was successful in crediting users for their subscriptions on the site after the server crash. Unfortunately, that failed to account for the dollar amount of their subscriptions in our tracking database table which is used to source our progress against our funding goal. I have a plan for getting those updates in place, but want to run it past other members of staff to make sure everything is accounted for before making any changes.

Read on for the rest of the site's news, or just wait and a new story will be out before too long.

Server Migrations:
We have received word that Linode, our web-hosting provider, will be conducting maintenance on two of our servers in the next 24 hours.

Last night Linode shut down one of our servers (boron), migrated the disk image to a new physical server, and restarted it. All seems to have gone smoothly.

Later on today, two more of our servers are due to be migrated:

  • hydrogen (alternate database server) 4 CPU Cores, 160 GB Storage, 8 GB RAM
    This Linode's physical host will be undergoing maintenance at 2021-06-14 00:00 UTC. During this time, your Linode will be shut down, cold migrated to a new host, then returned to its last state (running or powered off)
  • fluorine (primary database server) 4 CPU Cores, 96 GB Storage, 8 GB RAM
    This Linode's physical host will be undergoing maintenance at 2021-06-14 07:00 UTC. During this time, your Linode will be shut down, cold migrated to a new host, then returned to its last state (running or powered off).

Also of note, we are eligible for a free storage upgrade on fluorine from 96 GB to 160 GB. It is not clear at this moment if we will also conduct the storage upgrade at this time.

Cert Updates:
Our certs (issued by Let's Encrypt) are due to expire June 17, 2021.

We are aware and intend to have updated certs installed before then.

(NB: I may have some terminology errors in what follows, but I believe the overall process/concepts should be correct.)

I have personally installed updated certs twice before on our servers, and if need be, am prepared to do so again. It has been a couple years or so but the process should remain largely the same. The majority of the steps are automated, but historically we've preferred to handle the DNS updates manually. That way, just in case something goes sideways, we are hands-on and can take steps to mitigate problems... instead of finding we have a botched DNS and greatly restricted access the servers. (That is a bit of an overstatement, but as I understand it, it's a lot easier to make changes over SSH connections to running servers than through a console port to one server at a time.)

Also, there has been discussion about using a fully-automated Let's Encrypt cert update process, we'll keep you posted.

Site News:
Behind the scenes we've been hard at work. juggs, mechanicjay, and audioguy have put in many long and thankless hours stabilizing and documenting our service infrastructure. They've made great strides and we continue to make progress. We cannot change what was done (and not done) in the past, but we can learn from it! What services "live" on what servers? How to restart each service? Monitoring of disk usage and CPU usage? All are gradually being documented and site operations knowledge is getting shared all around.

Lastly, here's a shout-out to the editorial staff who strive to keep stories coming to you 24/7. Fnord666 just posted his 6,500th story! Also, thanks to janrinok, mrpg, chromas, and FatPhil who have all pushed out stories this past month! Teamwork++!

[N.B. Let's not forget our Editor-In-Chief martyb, who just posted his 10,100th story! This is in addition to serving as our primary QA person. - Fnord]


Original Submission

posted by martyb on Friday May 21 2021, @12:25AM   Printer-friendly

As many of you noticed, we had a site crash today. From around 1300 until 2200 UTC (2021-05-20).

A HUGE thank you goes to mechanicjay who spent the whole time trying to get our ndb (cluster) working again. It's an uncommon configuration, which made recovery especially challenging... there's just not a lot of documentation about it on the web.

I reached out and got hold of The Mighty Buzzard on the phone. Then put him in touch with mechanicjay who got us back up and running using backups.

Unfortunately, we had to go way back until April 14 to get a working backup. (I don't know all the details, but it appears something went sideways on neon).

We're all wiped out right now. When we have rested and had a chance to discuss things, we'll post an update.

In the meantime, please join me in thanking mechanicjay and TMB for all they did to get us up and running again!

posted by on Wednesday March 31 2021, @12:52AM   Printer-friendly
from the automotive dept.

We had a few hours this evening where there were issues we didn't notice with new submissions. We had been mothballing (setting the number of matches necessary beyond what would ever match anything) some spam filters exclusive to submissions and managed to somehow confuse one of the Apache web frontends. The offending filters are now just deleted entirely and incapable of confusing even MS Paint.

Please excuse the embuggerance. We now return you to your regularly scheduled arguments.

TMB