Slash Boxes

SoylentNews is people


Log In

Log In

Create Account  |  Retrieve Password

Site News

Join our Folding@Home team:
Main F@H site
Our team page

Funding Goal
For 6-month period:
2018-07-01 to 2018-12-31
Estimated Base Goal: $3000.00
Progress So Far:
Approximately: $951.29

Covers transactions:
2018-07-01 00:00:00 ..
2018-11-10 11:32:55 UTC
(SPIDs: [942..986])
Last Update:
2018-11-10 12:56:46 UTC

Support us: Subscribe Here
and buy SoylentNews Swag

We always have a place for talented people, visit the Get Involved section on the wiki to see how you can make SoylentNews better.

posted by janrinok on Thursday October 18, @01:00PM   Printer-friendly
from the always-look-on-the-bright-side-of-life dept.

Some of you will recall that I recently underwent several bouts of surgery and, despite your welcome comments and good wishes, your best guesses that I was having a sex change, having my breasts enhanced in either size or number, or receiving some fairly radical treatment for hemorrhoids, were all fairly wide of the mark. The surgery is now long past and I have made a reasonable recovery for someone of my age. But I do wish to thank you all for your comments because, almost without exception, they raised a smile when things were not going too well for me.

My wife suffers from a severe medical condition and I have been her full-time carer for over a decade now. Although I wish she had never developed the condition, I expend great effort, and also receive considerable personal satisfaction, in providing many hours of care each day so that she can remain in our home and we can continue our lives to the fullest extent possible. However, she has recently suffered from a deterioration in her condition. This was not unexpected but no-one could say when the next problems would affect her. But the result is that I now have to provide more support to her and my free time is reduced.

I joined this site at its inception and have enjoyed every minute of my time here. But I cannot dedicate the time that role of Editor-in-Chief (E-in-C) deserves and, several weeks ago, I made the difficult decision to stand down from the post. (I can hear the cheering from some in our community even here in France!) The fact that most of you will not have noticed any of this means that the transition has been successful. The entire editorial team (which is nowhere near as large as that phrase makes it sound!) has stepped up to the plate and has maintained the output as it was before, ably led by Martyb who has assumed the role as E-in-C in addition to his numerous other roles on this site. I am grateful to them for their efforts and support both during my time as E-in-C and more recently in their work in editing the stories that we read each day. Thanks guys, you do a tremendous job with relatively little recognition. I've asked the powers-that-be to increase your salaries by an appropriate percentage.[*] I am also grateful to the other folk who do so much in the background keeping this site on-line. You have all become good friends although we could be standing next to each other and wouldn't know it.

Equally important to the site's success are you - the community. You provide the submissions, the comments, the funding, and you are the reason that we have a site at all. I thank each and every one of you for your contribution; from the regular submitters, the ACs, the 'characters', and those of you who just visit to read the stories that we publish. If I have offended anyone then I apologise but being E-in-C has been likened to herding cats in the dark: an almost impossible task and one in which you are certain to make a few mistakes.

Hopefully, I will remain on the site as an editor making whatever contribution I can. But, for the next few months at least, that contribution will be minimal as I have to solve several practical problems on how to make our home function satisfactorily for us both. I trust that you will give Martyb and the team the same support in the future that I have been fortunate enough to enjoy over the past couple of years. Thank you.


[*] Clarification: This is an inside joke among the staff. Nobody here receives any kind of payment for their efforts on the site; we are strictly volunteers. So, a 20% raise on zero is... still zero. --martyb

posted by martyb on Wednesday October 10, @03:00PM   Printer-friendly

As you probably have noticed, our site has been a bit sluggish lately.

We are aware of the issue and are developing plans for dealing with it. The primary issue lies in the database structure and contents. On-the-fly joins across multiple tables cause a performance hit which is exacerbated by the number of stories we have posted over the years (yes, it HAS been that long... YAY!). Further, stories which have been "archived" — allowing no further comments or moderation — are still sitting in the in-RAM DB and could be offloaded to disk for long-term access. Once offloaded, there would be much less data in the in-RAM database (queries against empty DBs tend to be pretty quick!) so this should result in improved responsiveness.

A complicating factor is that changing the structure on a live, replicated database would cause most every page load to 500 out. So the database has to be offlined and the code updated. That would likely entail on the order of the better part of a day. Obviously, shorter is better. On the other hand "The longest distance between two points is a short cut." We're aiming to do it right, the first time, and be done with it, rather than doing it quick-and-dirty, which usually ends up being not quick and quite dirty.

So, we ARE aware of the performance issues, are working towards a solution, and don't want to cause any more disruption than absolutely necessary.

We will give notice well in advance of taking any actions.

Original Submission

posted by The Mighty Buzzard on Friday September 14, @01:30PM   Printer-friendly
from the necessary-roughness dept.

Pardon the short but unscheduled/unannounced site interruption. The cluster backend for our database needed to be resized as it was over 90% full and was causing performance issues. Hopefully it should all be taken care of now and shouldn't need poked at for another couple years at least.

posted by The Mighty Buzzard on Wednesday August 29, @01:32PM   Printer-friendly
from the head-meet-desk dept.

Gift subscriptions from ACs (Anonymous Cowards) are working again. If you're curious what was broken, have a look.

If you attempted to make a gift subscription as an AC since early to mid May, and received an error, please try again at: (Or click the link in the "Navigation" Slashbox).

As is standing SN policy, martyb is to blame for anything warranting blame. =) You can go about your business. Move along.

posted by martyb on Monday August 20, @11:45AM   Printer-friendly
from the teamwork++ dept.

We strive for openness about site operations here at SoylentNews. This story continues in that tradition.

tl;dr: We believe all services are now functioning properly and all issues have been attended to.

Problem Symptoms: I learned at 1212 UTC on Sunday 2018-08-19, that some pages on the site were returning 50x error codes. Sometimes, choosing 'back' in the browser and trying to resubmit the page would work. Oftentimes, it did not. We also started receiving reports of problems with our RSS and Atom feeds.

Read on past the break if you are interested in the steps taken to isolate and correct the problems.

Problem Isolation: As many of you may be aware, TheMightyBuzzard is away on vacation. I logged onto our IRC (Internet Relay Chat) Sunday morning (at 1212 UTC) when I saw chromas had posted (at 0224 UTC) there had been reports of problems with the RSS and Atom feeds we publish. I also noticed that one of our bots, Bender, was double-posting notifications of stories appearing on the site.

While I was investigating Bender's loquaciousness, chromas popped in to IRC (at 1252 UTC) and informed me that he was getting 502 and 503 error codes when he tried to load index.rss using a variety of browsers. I tried and found no issues when using Pale Moon. We tried a variety of wget requests from different servers. To our surprise we received incomplete replies which then caused multiple retries even when trying to access it from one of our SoylentNews servers. So, we surmised, it was probably not a communications issue.

At 1340 UTC, SemperOss (Our newest sysadmin staff member... Hi!) joined IRC and reported that he, too, was getting retry errors. Unfortunately, his account setup has not been completed leaving him with access to only one server (boron). Fortunately for us, he has a solid background in sysops. We combined his knowledge and experience with my access privileges and commenced to isolate the problem.

(Aside: If you have ever tried to isolate and debug a problem remotely, you know how frustrating it can be. SemperOss had to relay commands to me through IRC. I would pose questions until I was certain of the correct command syntax and intention. Next, I would issue the command and report back the results; again in IRC. On several occasions, chromas piped up with critical observations and suggestions — plus some much-needed humorous commentary! It could have been an exercise in frustration with worn patience and frazzled nerves. In reality, there was only professionalism as we pursued various possibilities and examined outcomes.)

From the fact we were receiving 50x errors, SemperOss surmised we were probably having a problem with nginx. We looked at the logs on sodium (which runs Ubuntu), one of our two load balancers, but nothing seemed out of the ordinary. Well, let's try the other load balancer, on magnesium (running Gentoo). Different directory structure, it seems, but we tracked down the log files and discovered that access.log had grown to over 8GB... and thus depleted all free space on /dev/root, the main file system of the machine.

That's not a good thing, but at least we finally knew what the problem was!

Problem Resolution: So, we renamed the original access.log file and created a new one for nginx to write to. Next up came a search for a box with sufficient space that we could copy the file to. SemperOss reported more than enough space free on boron. We had a few hiccups with ACLs and rsync, so moved the file to /tmp and tried rsync again, which resulted in the same ACL error messages. Grrrr. SemperOss suggested I try to pull the file over to /tmp on boron using scp. THAT worked! A few minutes later and the copy was completed. Yay!

But, we still had the original, over-sized log file to deal with. No problemo. I ssh'd back over to magnesium and did an rm of the copy of the access.log and... we were still at 100% usage. Doh! Needed to bounce nginx so it would release its hold on the file's inode so it could actually be cleaned up. Easy peasy; /etc/init.d/nginx restart and... voila! We were now back down to 67% in use.

Finally! Success! We're done, right?
Umm, no.

Did you see what we missed? The backup copy of access.log was now sitting on boron on /tmp which means the next system restart would wipe it. So, a simple mv from /tmp to my ~/tmp and now the file was in a safe place.

By 1630 UTC, we had performed some checks with loads of various RSS and atom feeds and all seemed well. Were unable to reproduce 50x errors, either.

And we're still not done.

Why/how did the log file get so large in the first place? There was no log rotation in place for it on magnesium. That log file had entries going back to 2017-06-20. At the moment, we have more than sufficient space to allow us to wait until TMB returns from vacation. (We checked free disk space on all of our servers.) The plan is we will look over all log files and ensure rotation is in place so as to avoid a recurrence of this issue.

Problem Summary: We had a problem with an oversized logfile taking up all free space on one of our servers but believe we have fixed it and that all services are now functioning properly and all issues have been attended to.

Conclusion: Please join me in thanking chromas and SemperOss for all the time they gave up on a Sunday to isolate the problem and come up with a solution. Special mention to Fnord666 who we later learned silently lurked, but was willing to jump in had he sensed we needed any help. Thank-you for having our backs! Further, please join me in publicly welcoming SemperOss to the team and wishing him well on his efforts here!

Lastly, this is an all-volunteer, non-commercial site — nobody is paid anything for their efforts in support of the site. We are, therefore, entirely dependent on the community for financial support. Please take a moment and consider subscribing to SoylentNews, either with a new subscription, by extending an existing subscription, or making a gift subscription to someone else on the site. Any amount entered in the payment amount field, above and beyond the minimum amount is especially appreciated!

Original Submission

posted by martyb on Wednesday July 18, @11:00AM   Printer-friendly
from the steady-as-she-goes dept.

First the good news. I just received word that janrinok, our Editor-in-Chief, is finally out of the hospital and back in his own home! He is very tired and has severe restrictions on his activities but is otherwise in excellent spirits. He very much appreciated the kind thoughts and wishes expressed by the community in our prior stories. It will still be many weeks or months before he can resume his prior level of activities on SoylentNews, but hopes to pop in once in a while to "second" stories that are in the story queue. Please join me in welcoming him back home!

Next, the good news. In janrinok's absence, the other editors have stepped up to the challenge. I'd like to call out chromas, fnord666, mrpg, and takyon who have all freely given from their spare time to make sure we have a steady stream of stories appearing here. I even saw CoolHand pop in on occasion to second some stories! teamwork++

Then, I have to bring up the good news that our development and systems staff have kept this whole thing running so smoothly. Besides the site, there is e-mail, the wiki, our IRC server, and a goodly number of other processes and procedures that make this all happen. That they are largely invisible attests to how well they have things set up and running!

Lastly, the good news. This is what's known in the press as the "silly season". Summer in the Northern Hemisphere means most educational institutions are on break, so less research is done and reported. other ventures are closed or running on reduced staffing levels. In short, the amount of news to draw from is greatly diminished. Yet, even in that environment, the vast majority of the time finds us with a selection of stories in the submissions queue to draw from.

We recently hit a low spot where I combed the web for a couple quick stories I could submit, but that has been the exception rather than the rule. Generally, we look for stories that have some kind of tech-related angle to them. The community has spoken loud and clear that there are plenty of other sites to read about celebrities, politics, and religion. We make a slight nod to politics in so much as it affects technical areas or has large scale ramifications (e.g. a story about President Trump having a meeting with Russian president Vladimir Putin would fit that description). Even then we generally try to keep it down to one story per day.

That said, if you see a story on the 'net that catches your fancy, please send it in! Feel free to draw upon titles listed on our Storybot page, then pop onto IRC (Internet Relay Chat) and simply issue the command ~arthur $code where $code is taken from the second column on the Storybot page.


Whether you contribute by submitting a story, buying a subscription, writing in one's journal, moderating or making a comment, we continue to provide a place where people can discuss, share knowledge and perspectives, and maybe learn a thing or two, too!


Original Submission

posted by martyb on Tuesday July 10, @10:33PM   Printer-friendly
from the watching-out-for-one-of-our-own dept.

As you might recall, in an earlier story we noted that SoylentNews' Editor-in-Chief, janrinok, was scheduled for a medical procedure.

I have just received word that there were some (not totally unanticipated, but thought to be a very unlikely) complications and the expected 3 day hospital stay has now lasted over a week. In his own words:

No idea what will happen next is anybody's guess. My first objective is to be well enough to get home again but that looks like being the end of the week at the earliest.

He seemed to be in good spirits. In his inimitable style of humor, he noted the internet connection available to him in Hospital would lie in the bottom-most tier of our current poll!

I am torn in revealing personal details that were shared with me, and wish to not sound overly alarmist. I'll just leave here that I am reminded of a saying by Ralph Waldo Emerson "You cannot do a kindness too soon, for you never know how soon it will be too late."

JR has tirelessly (and tiredly, too) gone over and above in support of this site -- please keep him in your thoughts and, if you are of a mind to, your prayers. --martyb

Original Submission

posted by martyb on Monday July 02, @02:30PM   Printer-friendly
from the teamwork++ dept.

This is a followup to: SoylentNews Site Certificates Expiring... We ARE Working on It, But... [Updated]
and: Site Services Restored

Certs (Not Just a Breath Mint):

Thanks to the efforts of The Mighty Buzzard and NCommander we now have valid certs, issued by LetsEncrypt installed on all of our servers. Except that the IRC server needs to be bounced to make its cert active on the backup daemon, all should now be in effect. As in our original story, you can check our certificate status with these links:


The past few days have brought into focus a situation that has been building for several months: We really only have a single person who is working on developing features for the site, The Mighty Buzzard. As with any large and on-going undertaking, this burden is taking its toll. I try to help out as I can, but as I am the primary QA/Test guy who is much better at the user-facing things than what all happens "under the covers", my abilities and assistance are limited. If you have any spare time and would like to lend a hand (and every bit helps), please reply in the comments or contact The Might Buzzard directly on IRC.


I recall in the early days of this site when things would fall over several times a day. That has largely become a thing of the past... to the point where it is unusual for any issues to appear on the site and the support services we maintain (email, wiki, IRC, etc.) The baseline code on which this site was founded (open-sourced, out-of-date, back-level, and non-functional) was not promising, but the staff managed to bludgeon it into shape and we now have a solid foundation. That it continues to run as smoothly as it has is a testament to our SysOps folk who toil largely in the background and just keep things working... as well as the continued care-and-feeding that TMB so generously provides. To all of you, please accept my heartfelt thanks and appreciation!

Some numbers: we are approaching the 23,000th story posted; have recently passed 700,000 comments submitted; have had over 3,300 journal articles posted; and are on the cusp of having our 120th Poll!

Though all numbers are approximate and unofficial, it appears we surpassed our funding goal for the first half of the year ($3,000) with a net subscription tally of just over $3,250! I'll leave it to our treasurer to collate and post the official numbers. I'll leave the "Funding Goal" side bar as is for a week or so to commemorate this accomplishment. Do note that subscriptions are still being accepted and will count towards the second half of the year's funding needs.

Folding@Home: Not all of you may be aware, but our soylentnews team for Folding@Home is currently at 240th place... in the world! It started with a single story posted to this site. Just over four years ago, we were at 230,319th place! If you have any spare computes you would like to contribute, especially GPU-based, we'd love to have you sign up! Just reply in the comments and I'm sure someone will get back to you.

Whenever I write one of these stories, I always fear I'll have omitted someone or something important. Please accept my humble apologies if I have done so as there is no intent to slight any contributor.

To the community, I offer my thanks for your contributions to the site as well as your patience and understanding during the challenges of the past few days. Contributions are not just financial (though we wouldn't be here without them -- Thank You!), but also submitting stories and comments, and moderating comments, too! The community continues to impress me with your wide-ranging knowledge and expertise; I have learned much from the exchanges in the story comments!

Lastly, please keep janrinok (our Editor-in-Chief) in your thoughts and wishes while he undergoes a medical procedure and attendant recovery period. Best of luck JR!


Original Submission

posted by NCommander on Sunday July 01, @01:33AM   Printer-friendly

Just a quick note, as previously noted, our SSL certificates were due to expire. Due to various headaches involving issues changing our DNS records, and my personal unavailability, we were unable to renew our certificates in time. Right now, is running on a LetsEncrypt certificate that was issued after quite a bit of pain. We're still trying to fix the fundamental issues that prevented us from being issued a two year Gandi certificate. Currently, I'm unable to resolve this issue more in-depth, but I've granted access to TheMightyBuzzard to be able help handle the necessary issues that caused the downtime. I will try to get a full writeup of the situation, but for the time being, the main site is up. Secondary services remain down due to the same renewal issues.

Apologies for any inconvenience,


posted by martyb on Saturday June 30, @03:20PM   Printer-friendly
from the certs-are-not-just-a-breath-mint dept.

[Updated 2018/06/30 15:19:00 UTC. I've received word that "certificate renewed, but hasn't been issued" and that it will be installed as soon as we receive the new cert. Original story follows. --martyb]

We've been up front with the community right from the start... and intend to keep doing so in the future.

Where we are at:

We have encountered an issue with Lets Encrypt (LE), the certificate issuer for the majority of our [sub]domains. Even though we can 'see' these domains from any number of different servers... for some unknown reason, LE fails to see them. So, at the moment, we are unable to get them to generate certs for us.

Separately, the cert for is handled by Gandhi. As far as I understand it (and I'm no sysadmin so take this with a healthy dose of salt) there are only two members of our staff who have the ability to update that cert. (We obviously don't want to let world+dog have access to that, right? My guess is that at that time, having a couple people seemed sufficiently redundant and secure).

What it means to you:

You may encounter a warning from your browser when trying to access the site that a certificate has expired. I cannot speak for all browsers, but I've generally seen that along with the warning is an option to trust the cert anyway. (Note: along with allowing that exception, I've sen at least one browser default a checkbox to make the exception permanent. It's entirely up to you, but I see no reason to make it a permanent exception at this point.)

We are working on it, and obviously hope to have things straightened out sooner than later! On the other hand, should things go sideways, I want to keep the community informed about what's up, what's happening, and what you can expect.

Checking status:

If you would like another means to check on the status of the certs, Comodo makes it easy with queries such as these:


If the site becomes unavailable because of an expired cert, yeah, we know and we're working on it. Accept a temporary exception in your browser and we'll let you now when things are back to normal.


P.S. Our Editor-in-Chief, janrinok, is currently undergoing preparations for a medical procedure... it's hard to say at this point, but it's likely he may be unavailable to help with the site for a couple weeks. Please join me in wishing him well for the procedure and for a speedy recovery!

Original Submission

posted by mrpg on Friday June 08, @09:00AM   Printer-friendly
from the community++ dept.

Just three days ago, I asked for the community's help to raise funds for continuing operations. Incoming funds were trending upwards — but way too slowly — and it was clear we were going to come up short. I was genuinely worried about our ability to continue as a going concern.

My fears were, apparently, unwarranted... over those three days we received 49 subscriptions totaling approximately $1534.74 which led us to exceed our base goal of $3000.00 for the first half of the year!


Of these 49 subscriptions, the vast majority were for $20.00 (36 of these), but we also had 3@$40.00, 1@$50, and (drum roll please) 3@$200.00! And for those who subscribed at other amounts, I also thank you. I've had times in my life when my finances did not allow me to help others as I'd wished. Every bit helps!

Let me interrupt the funding-focus for a moment to call attention to a notable accomplishment among our staff: Our Editor in Chief, janrinok (aka JR), just passed 4,000 stories posted to the site! There was a long stretch about a year or so into SoylentNews' existence when JR and LaminatorX (aka LamX) handled pretty much *all* of the story duties on the site. During that period, story submissions were rare and often little more than a bare link to a story of interest. So they'd tag-team finding stories on the web, making submissions, posting stories, and reviewing same. Were it not for their extended and heroic efforts, there would have been a period of months with almost no stories for us to read. LamX left us a couple years ago (would love to have ya back!) But JR has stuck with us, and we are forever indebted to him for his generous contributions of time, expertise, and guidance. Please join me in congratulating JR for reaching this amazing milestone!

Many thanks, too, for the feedback in the comments to the earlier story — I read every one of them — but am in severe crunch mode at work and lacked the spare time to respond individually as I would have liked. I'll respond to a couple items here.

There was a suggestion to set up a "war chest" from which we could draw off interest to support continuing operations among other things. I don't anticipate we will have $100K in the bank any time soon (one of the suggestions), but having a rainy-day fund "just in case" does seem prudent to me. In light of that, you may have noticed in the "Site News" slashbox there are now two goals listed: a base goal and a stretch goal.

So how much should the stretch goal be for?

As it now stands, we have received approximately $125 in excess of our base goal. So, how about $2000.00? With only about 3 weeks remaining in the period, that seemed like too much of a stretch. I pulled a number out of my butt and came up with $1000.00 which is still a stretch (doh!), but I believe is attainable. I am mindful that we have the second half of the year ahead of us and will be needing funds for that period, too. So, I am NOT asking for folks to go crazy to reach this stretch goal at the expense (pun intended!) of the needs for the rest of the year... if you can help out, that's great! If not, well that's okay, too.

Another suggestion was a means to generate funds besides subscriptions. I failed to mention this in my earlier exhortation, but we DO have SoylentNews-related things for sale over at*. (NOTE: The asterisk at the end of the link is intentional and gives SoylentNews a slightly larger return on each purchase; please leave it in.) This link is also listed in the Site News slashbox ("buy SoylentNews Swag"), should you wish to refer back to it.

We are still in the process of taking advantage of the free disk space upgrade available from Linode. It is unclear at the moment whether these free disk upgrades will make further server consolidation possible, but we are looking into it. We will keep you informed as we proceed with the upgrades, as well.

One more thing. One of the comments mentioned how had become their "goto site" to see if the internet was up. I nearly fell out of my chair! Those who were with us the first few weeks of the site will well recall how we would have multiple crashes each day. That things have changed so dramatically as this... please join me in thanking the sysops who may stay out of the limelight, but keep things running like a top!

Again, please accept my thanks for the huge outpouring of support for the site. I am humbled by the generosity of the community and count it a distinct honor to be able to play some role in it. --martyb

Original Submission

posted by chromas on Wednesday June 06, @02:30PM   Printer-friendly
from the community++ dept.

[Update #2: Wed 20180606 @ 14:30 UTC: WOW! Since this story first posted, we have received 36 subscriptions totaling approximately $1126.27 which brings us to a total of approximately $2716.36 which works out to just over 90% of our $3000.00 goal!! I am extremely impressed with the showing of support from the community and reminds me why I volunteer so much of my time here. It really is the community! THANK YOU! --martyb]

[Update #1: Mon 20180604 @ 14:14 UTC: So far today we have received 16 subscriptions bringing us to just shy of $2000.00 raised for the period and just over $1000 to go. THANK-YOU! --martyb]

tl;dr: SoylentNews needs your financial support... $1400 by June 30th. Please Subscribe.

We have raised approximately $1590.09 towards our funding goal of $3000.00 which covers the period from 2018-01-01 though 2018-06-30.

We run the site with a team of volunteers — absolutely nobody gets paid for their work on this site... all the funds raised go directly into keeping the site up and running.

From the time we went live on February 17, 2014 through today, we have posted over 22,000 stories and nearly 120 polls. In kind, the community has posted over 3270 journal articles and over 688,000 comments.

We understand that not everyone can afford to support the site financially. Those who make contributions "in kind" through submitting stories or comments, making moderations, and the like... please accept my sincere "Thank-You!" Without your contributions, there would either be no site or no reason to come here.

For those of you who have previously subscribed, thank YOU for "keeping the lights on". Good wishes did not pay for servers, federal and state tax returns, or domain registrations... You did! Thank you!

A complicating factor came from our having a problem with the subscription page about a month ago... but TMB put up a fix and we've seen no regressions, since. Perhaps this bit you and you meant to "try again later"? (As of my writing this, we have 123 users who have accessed the site within the past month, who had previously subscribed, but whose subscription is now lapsed.)

Whether your subscription has lapsed or you never previously subscribed, would you please consider signing up? Go to the subscription page, pick a duration, and type in an amount that is at least the amount suggested for that duration (excess amounts are, of course, most welcome!). If you would like to help the site without getting a subscription for yourself, you can make a gift subscription by specifying a UID and an amount, instead. (If you have nobody in particular in mind, please feel free to use UID 6 — "mcasadevall")

Original Submission

posted by martyb on Sunday June 03, @01:45PM   Printer-friendly
from the free-is-good! dept.

[Update: as of 20180603_135454 UTC, all 4 server upgrades have completed and all servers should be running normally. --martyb]

Our server provider, Linode, has made available a free upgrade for our servers. Generally, it's a storage upgrade and sometimes a bandwidth upgrade, too. We use only the tiniest fraction of our bandwidth allocation each month (something like < 1%), but the extra storage IS useful.

We (read: The Mighty Buzzard) have already taken 4 of our servers out of rotation this morning. We anticipate no community-visible interruptions of service as a result of these upgrades.

So far, migrations of sodium, hydrogen, and lithium have completed. Migration of neon has started and should complete within the next 10 minutes or so.

Migrations of the remaining servers may cause some community-visible effects. We will provide advance notice before we perform those upgrades. Very tentatively planned for Thursday.

Original Submission

posted by janrinok on Thursday May 17 2018, @06:10PM   Printer-friendly
from the draining-the-queue dept.

[Update: I do not have access to the mail queue, but the server dashboard shows that, as of 23:00 UTC, beryllium has returned to normal disk and cpu levels. That said, I see a gap in daily story headlines and daily story e-mails that were sent to me. We are continuing to monitor the situation. Please let us know if you have any outstanding issues. --martyb]

We have been open with the community since the outset, and in keeping with that practice: we just fixed an issue with the site.

On or about May 9th, our mail server, beryllium, stopped sending out e-mails. The cause was the antivirus handler failing to be loaded, so all outgoing mail that would be processed by that handler ended up waiting indefinitely.

Many thanks to mechanicjay for debugging and fixing the issue!

Impact: If you signed up for emails from this site (such as notification of comment replies or moderation, subscription being low or expired, etc.) these have been delayed. It may take some time for the queue to be processed and for all pending e-mails to be sent out.

I well remember when SoylentNews launched and each day brought a seemingly endless supply of crashes and failures. It is a tribute to our volunteer staff that site issues now happen so rarely!

Original Submission

posted by martyb on Thursday May 10 2018, @01:58AM   Printer-friendly

[Update 5: All done. Nuttin but net. --TMB]

[Update 4: As of 20180509 @ 11:55 UTC, beryllium has been successfully rebooted. This leaves hydrogen to be rebooted in just over 13 hours. --martyb]

[Update 3: As of 20180509 @ 0414 UTC, both lithium and sodium appear to have successfully completed their reboots. That leaves beryllium (1hr45m from now) and hydrogen (20h45m from now) to complete their reboots. --martyb]

[Update 2: The second round of reboots went peachy keen as well. Next round starts at 3AM UTC (7 hours from this story's time) with our dev server (lithium). An hour later the load balancer (sodium) that I switched us off of this morning will reboot. Two hours after that the box (beryllium) that hosts the wiki, mail, IRC, and some other lesser-used stuff will get bounced. If you can't stand being disconnected from IRC for a few minutes, add (6667/6697) to the list of servers for this network. --TMB]

[Update 1: The first scheduled reboot (of fluorine) was successful. The two-hour reboot window for helium starts 1.5 hours from the date/time stamp for this story. Two hours after that marks the commencement of the two-hour reboot window for boron, magnesium, and neon. We do not anticipate any site interruption as a result of these reboots. --martyb]

We have been informed by Linode (on which all of the SoylentNews servers are hosted) that maintenance is required to mitigate against the Spectre (v1 and v2) attacks. As a result, all of our servers will require a reboot. Historically, any given server is down for anywhere from 15-30 minutes. We have redundancies in place for many of our operations, but there may be some unavoidable downtime. We ask your patience and understanding during this process.

The scheduled reboots are:

Sat 2018-05-05 1:00:00 AM UTC fluorine [1] Production Cluster Completed
Tue 2018-05-08 1:00:00 AM UTC helium Production Cluster Completed
Tue 2018-05-08 3:00:00 AM UTC boron Services Cluster Completed
Tue 2018-05-08 3:00:00 AM UTC magnesium Frontend Proxy Completed
Tue 2018-05-08 3:00:00 AM UTC neon Production Cluster Completed
Wed 2018-05-09 3:00:00 AM UTC lithium Development Cluster Completed
Wed 2018-05-09 4:00:00 AM UTC sodium Frontend Proxy Completed
Wed 2018-05-09 6:00:00 AM UTC beryllium [2] Services Cluster Completed
Thu 2018-05-10 1:00:00 AM UTC hydrogen Production Cluster Completed

[1] Unable to process subscriptions or update comment counts or deliver messages until it reboots.

[2] IRC (Internet Relay Chat) server will be unavailable.

Original Submission