2018-07-01 00:00:00 ..
2018-07-06 10:04:11 UTC
2018-07-14 12:58:04 UTC
We always have a place for talented people, visit the Get Involved section on the wiki to see how you can make SoylentNews better.
As you might recall, in an earlier story we noted that SoylentNews' Editor-in-Chief, janrinok, was scheduled for a medical procedure.
I have just received word that there were some (not totally unanticipated, but thought to be a very unlikely) complications and the expected 3 day hospital stay has now lasted over a week. In his own words:
No idea what will happen next is anybody's guess. My first objective is to be well enough to get home again but that looks like being the end of the week at the earliest.
He seemed to be in good spirits. In his inimitable style of humor, he noted the internet connection available to him in Hospital would lie in the bottom-most tier of our current poll!
I am torn in revealing personal details that were shared with me, and wish to not sound overly alarmist. I'll just leave here that I am reminded of a saying by Ralph Waldo Emerson "You cannot do a kindness too soon, for you never know how soon it will be too late."
JR has tirelessly (and tiredly, too) gone over and above in support of this site -- please keep him in your thoughts and, if you are of a mind to, your prayers. --martyb
This is a followup to: SoylentNews Site Certificates Expiring... We ARE Working on It, But... [Updated]
and: Site Services Restored
Certs (Not Just a Breath Mint):
Thanks to the efforts of The Mighty Buzzard and NCommander we now have valid certs, issued by LetsEncrypt installed on all of our servers. Except that the IRC server needs to be bounced to make its cert active on the backup daemon, all should now be in effect. As in our original story, you can check our certificate status with these links:
The past few days have brought into focus a situation that has been building for several months: We really only have a single person who is working on developing features for the site, The Mighty Buzzard. As with any large and on-going undertaking, this burden is taking its toll. I try to help out as I can, but as I am the primary QA/Test guy who is much better at the user-facing things than what all happens "under the covers", my abilities and assistance are limited. If you have any spare time and would like to lend a hand (and every bit helps), please reply in the comments or contact The Might Buzzard directly on IRC.
I recall in the early days of this site when things would fall over several times a day. That has largely become a thing of the past... to the point where it is unusual for any issues to appear on the site and the support services we maintain (email, wiki, IRC, etc.) The baseline code on which this site was founded (open-sourced, out-of-date, back-level, and non-functional) was not promising, but the staff managed to bludgeon it into shape and we now have a solid foundation. That it continues to run as smoothly as it has is a testament to our SysOps folk who toil largely in the background and just keep things working... as well as the continued care-and-feeding that TMB so generously provides. To all of you, please accept my heartfelt thanks and appreciation!
Some numbers: we are approaching the 23,000th story posted; have recently passed 700,000 comments submitted; have had over 3,300 journal articles posted; and are on the cusp of having our 120th Poll!
Though all numbers are approximate and unofficial, it appears we surpassed our funding goal for the first half of the year ($3,000) with a net subscription tally of just over $3,250! I'll leave it to our treasurer to collate and post the official numbers. I'll leave the "Funding Goal" side bar as is for a week or so to commemorate this accomplishment. Do note that subscriptions are still being accepted and will count towards the second half of the year's funding needs.
Folding@Home: Not all of you may be aware, but our soylentnews team for Folding@Home is currently at 240th place... in the world! It started with a single story posted to this site. Just over four years ago, we were at 230,319th place! If you have any spare computes you would like to contribute, especially GPU-based, we'd love to have you sign up! Just reply in the comments and I'm sure someone will get back to you.
Whenever I write one of these stories, I always fear I'll have omitted someone or something important. Please accept my humble apologies if I have done so as there is no intent to slight any contributor.
To the community, I offer my thanks for your contributions to the site as well as your patience and understanding during the challenges of the past few days. Contributions are not just financial (though we wouldn't be here without them -- Thank You!), but also submitting stories and comments, and moderating comments, too! The community continues to impress me with your wide-ranging knowledge and expertise; I have learned much from the exchanges in the story comments!
Lastly, please keep janrinok (our Editor-in-Chief) in your thoughts and wishes while he undergoes a medical procedure and attendant recovery period. Best of luck JR!
Just a quick note, as previously noted, our SSL certificates were due to expire. Due to various headaches involving issues changing our DNS records, and my personal unavailability, we were unable to renew our certificates in time. Right now, soylentnews.org is running on a LetsEncrypt certificate that was issued after quite a bit of pain. We're still trying to fix the fundamental issues that prevented us from being issued a two year Gandi certificate. Currently, I'm unable to resolve this issue more in-depth, but I've granted access to TheMightyBuzzard to be able help handle the necessary issues that caused the downtime. I will try to get a full writeup of the situation, but for the time being, the main site is up. Secondary services remain down due to the same renewal issues.
Apologies for any inconvenience,
[Updated 2018/06/30 15:19:00 UTC. I've received word that "certificate renewed, but hasn't been issued" and that it will be installed as soon as we receive the new cert. Original story follows. --martyb]
We've been up front with the community right from the start... and intend to keep doing so in the future.
Where we are at:
We have encountered an issue with Lets Encrypt (LE), the certificate issuer for the majority of our [sub]domains. Even though we can 'see' these domains from any number of different servers... for some unknown reason, LE fails to see them. So, at the moment, we are unable to get them to generate certs for us.
Separately, the cert for soylentnews.org is handled by Gandhi. As far as I understand it (and I'm no sysadmin so take this with a healthy dose of salt) there are only two members of our staff who have the ability to update that cert. (We obviously don't want to let world+dog have access to that, right? My guess is that at that time, having a couple people seemed sufficiently redundant and secure).
What it means to you:
You may encounter a warning from your browser when trying to access the site that a certificate has expired. I cannot speak for all browsers, but I've generally seen that along with the warning is an option to trust the cert anyway. (Note: along with allowing that exception, I've sen at least one browser default a checkbox to make the exception permanent. It's entirely up to you, but I see no reason to make it a permanent exception at this point.)
We are working on it, and obviously hope to have things straightened out sooner than later! On the other hand, should things go sideways, I want to keep the community informed about what's up, what's happening, and what you can expect.
If you would like another means to check on the status of the certs, Comodo makes it easy with queries such as these:
If the site becomes unavailable because of an expired cert, yeah, we know and we're working on it. Accept a temporary exception in your browser and we'll let you now when things are back to normal.
P.S. Our Editor-in-Chief, janrinok, is currently undergoing preparations for a medical procedure... it's hard to say at this point, but it's likely he may be unavailable to help with the site for a couple weeks. Please join me in wishing him well for the procedure and for a speedy recovery!
Just three days ago, I asked for the community's help to raise funds for continuing operations. Incoming funds were trending upwards — but way too slowly — and it was clear we were going to come up short. I was genuinely worried about our ability to continue as a going concern.
My fears were, apparently, unwarranted... over those three days we received 49 subscriptions totaling approximately $1534.74 which led us to exceed our base goal of $3000.00 for the first half of the year!
Of these 49 subscriptions, the vast majority were for $20.00 (36 of these), but we also had 3@$40.00, 1@$50, and (drum roll please) 3@$200.00! And for those who subscribed at other amounts, I also thank you. I've had times in my life when my finances did not allow me to help others as I'd wished. Every bit helps!
Let me interrupt the funding-focus for a moment to call attention to a notable accomplishment among our staff: Our Editor in Chief, janrinok (aka JR), just passed 4,000 stories posted to the site! There was a long stretch about a year or so into SoylentNews' existence when JR and LaminatorX (aka LamX) handled pretty much *all* of the story duties on the site. During that period, story submissions were rare and often little more than a bare link to a story of interest. So they'd tag-team finding stories on the web, making submissions, posting stories, and reviewing same. Were it not for their extended and heroic efforts, there would have been a period of months with almost no stories for us to read. LamX left us a couple years ago (would love to have ya back!) But JR has stuck with us, and we are forever indebted to him for his generous contributions of time, expertise, and guidance. Please join me in congratulating JR for reaching this amazing milestone!
Many thanks, too, for the feedback in the comments to the earlier story — I read every one of them — but am in severe crunch mode at work and lacked the spare time to respond individually as I would have liked. I'll respond to a couple items here.
There was a suggestion to set up a "war chest" from which we could draw off interest to support continuing operations among other things. I don't anticipate we will have $100K in the bank any time soon (one of the suggestions), but having a rainy-day fund "just in case" does seem prudent to me. In light of that, you may have noticed in the "Site News" slashbox there are now two goals listed: a base goal and a stretch goal.
So how much should the stretch goal be for?
As it now stands, we have received approximately $125 in excess of our base goal. So, how about $2000.00? With only about 3 weeks remaining in the period, that seemed like too much of a stretch. I pulled a number out of my butt and came up with $1000.00 which is still a stretch (doh!), but I believe is attainable. I am mindful that we have the second half of the year ahead of us and will be needing funds for that period, too. So, I am NOT asking for folks to go crazy to reach this stretch goal at the expense (pun intended!) of the needs for the rest of the year... if you can help out, that's great! If not, well that's okay, too.
Another suggestion was a means to generate funds besides subscriptions. I failed to mention this in my earlier exhortation, but we DO have SoylentNews-related things for sale over at https://www.zazzle.com/soylentnews*. (NOTE: The asterisk at the end of the link is intentional and gives SoylentNews a slightly larger return on each purchase; please leave it in.) This link is also listed in the Site News slashbox ("buy SoylentNews Swag"), should you wish to refer back to it.
We are still in the process of taking advantage of the free disk space upgrade available from Linode. It is unclear at the moment whether these free disk upgrades will make further server consolidation possible, but we are looking into it. We will keep you informed as we proceed with the upgrades, as well.
One more thing. One of the comments mentioned how soylentnews.org had become their "goto site" to see if the internet was up. I nearly fell out of my chair! Those who were with us the first few weeks of the site will well recall how we would have multiple crashes each day. That things have changed so dramatically as this... please join me in thanking the sysops who may stay out of the limelight, but keep things running like a top!
Again, please accept my thanks for the huge outpouring of support for the site. I am humbled by the generosity of the community and count it a distinct honor to be able to play some role in it. --martyb
[Update #2: Wed 20180606 @ 14:30 UTC: WOW! Since this story first posted, we have received 36 subscriptions totaling approximately $1126.27 which brings us to a total of approximately $2716.36 which works out to just over 90% of our $3000.00 goal!! I am extremely impressed with the showing of support from the community and reminds me why I volunteer so much of my time here. It really is the community! THANK YOU! --martyb]
[Update #1: Mon 20180604 @ 14:14 UTC: So far today we have received 16 subscriptions bringing us to just shy of $2000.00 raised for the period and just over $1000 to go. THANK-YOU! --martyb]
tl;dr: SoylentNews needs your financial support... $1400 by June 30th. Please Subscribe.
We have raised approximately $1590.09 towards our funding goal of $3000.00 which covers the period from 2018-01-01 though 2018-06-30.
We run the site with a team of volunteers — absolutely nobody gets paid for their work on this site... all the funds raised go directly into keeping the site up and running.
From the time we went live on February 17, 2014 through today, we have posted over 22,000 stories and nearly 120 polls. In kind, the community has posted over 3270 journal articles and over 688,000 comments.
We understand that not everyone can afford to support the site financially. Those who make contributions "in kind" through submitting stories or comments, making moderations, and the like... please accept my sincere "Thank-You!" Without your contributions, there would either be no site or no reason to come here.
For those of you who have previously subscribed, thank YOU for "keeping the lights on". Good wishes did not pay for servers, federal and state tax returns, or domain registrations... You did! Thank you!
A complicating factor came from our having a problem with the subscription page about a month ago... but TMB put up a fix and we've seen no regressions, since. Perhaps this bit you and you meant to "try again later"? (As of my writing this, we have 123 users who have accessed the site within the past month, who had previously subscribed, but whose subscription is now lapsed.)
Whether your subscription has lapsed or you never previously subscribed, would you please consider signing up? Go to the subscription page, pick a duration, and type in an amount that is at least the amount suggested for that duration (excess amounts are, of course, most welcome!). If you would like to help the site without getting a subscription for yourself, you can make a gift subscription by specifying a UID and an amount, instead. (If you have nobody in particular in mind, please feel free to use UID 6 — "mcasadevall")
[Update: as of 20180603_135454 UTC, all 4 server upgrades have completed and all servers should be running normally. --martyb]
Our server provider, Linode, has made available a free upgrade for our servers. Generally, it's a storage upgrade and sometimes a bandwidth upgrade, too. We use only the tiniest fraction of our bandwidth allocation each month (something like < 1%), but the extra storage IS useful.
We (read: The Mighty Buzzard) have already taken 4 of our servers out of rotation this morning. We anticipate no community-visible interruptions of service as a result of these upgrades.
So far, migrations of sodium, hydrogen, and lithium have completed. Migration of neon has started and should complete within the next 10 minutes or so.
Migrations of the remaining servers may cause some community-visible effects. We will provide advance notice before we perform those upgrades. Very tentatively planned for Thursday.
[Update: I do not have access to the mail queue, but the server dashboard shows that, as of 23:00 UTC, beryllium has returned to normal disk and cpu levels. That said, I see a gap in daily story headlines and daily story e-mails that were sent to me. We are continuing to monitor the situation. Please let us know if you have any outstanding issues. --martyb]
We have been open with the community since the outset, and in keeping with that practice: we just fixed an issue with the site.
On or about May 9th, our mail server, beryllium, stopped sending out e-mails. The cause was the antivirus handler failing to be loaded, so all outgoing mail that would be processed by that handler ended up waiting indefinitely.
Many thanks to mechanicjay for debugging and fixing the issue!
Impact: If you signed up for emails from this site (such as notification of comment replies or moderation, subscription being low or expired, etc.) these have been delayed. It may take some time for the queue to be processed and for all pending e-mails to be sent out.
I well remember when SoylentNews launched and each day brought a seemingly endless supply of crashes and failures. It is a tribute to our volunteer staff that site issues now happen so rarely!
[Update 5: All done. Nuttin but net. --TMB]
[Update 4: As of 20180509 @ 11:55 UTC, beryllium has been successfully rebooted. This leaves hydrogen to be rebooted in just over 13 hours. --martyb]
[Update 3: As of 20180509 @ 0414 UTC, both lithium and sodium appear to have successfully completed their reboots. That leaves beryllium (1hr45m from now) and hydrogen (20h45m from now) to complete their reboots. --martyb]
[Update 2: The second round of reboots went peachy keen as well. Next round starts at 3AM UTC (7 hours from this story's time) with our dev server (lithium). An hour later the load balancer (sodium) that I switched us off of this morning will reboot. Two hours after that the box (beryllium) that hosts the wiki, mail, IRC, and some other lesser-used stuff will get bounced. If you can't stand being disconnected from IRC for a few minutes, add irc2.sylnt.us (6667/6697) to the list of servers for this network. --TMB]
[Update 1: The first scheduled reboot (of fluorine) was successful. The two-hour reboot window for helium starts 1.5 hours from the date/time stamp for this story. Two hours after that marks the commencement of the two-hour reboot window for boron, magnesium, and neon. We do not anticipate any site interruption as a result of these reboots. --martyb]
We have been informed by Linode (on which all of the SoylentNews servers are hosted) that maintenance is required to mitigate against the Spectre (v1 and v2) attacks. As a result, all of our servers will require a reboot. Historically, any given server is down for anywhere from 15-30 minutes. We have redundancies in place for many of our operations, but there may be some unavoidable downtime. We ask your patience and understanding during this process.
The scheduled reboots are:
|Sat||2018-05-05 1:00:00 AM UTC||fluorine ||Production Cluster||Completed|
|Tue||2018-05-08 1:00:00 AM UTC||helium||Production Cluster||Completed|
|Tue||2018-05-08 3:00:00 AM UTC||boron||Services Cluster||Completed|
|Tue||2018-05-08 3:00:00 AM UTC||magnesium||Frontend Proxy||Completed|
|Tue||2018-05-08 3:00:00 AM UTC||neon||Production Cluster||Completed|
|Wed||2018-05-09 3:00:00 AM UTC||lithium||Development Cluster||Completed|
|Wed||2018-05-09 4:00:00 AM UTC||sodium||Frontend Proxy||Completed|
|Wed||2018-05-09 6:00:00 AM UTC||beryllium ||Services Cluster||Completed|
|Thu||2018-05-10 1:00:00 AM UTC||hydrogen||Production Cluster||Completed|
 Unable to process subscriptions or update comment counts or deliver messages until it reboots.
 IRC (Internet Relay Chat) server will be unavailable.
Although lots of different tasks are required to keep a site such as this operating, interesting, and relevant, the most visible are the stories that we publish on the front page. Some stories can be edited in a matter of a few minutes, others take rather longer. Among the editors we celebrate when we reach certain milestones. The first published story is a milestone of sorts but, more importantly, the first 100, 250, 500 and 1000th etc are all events that are more important milestones to an editor and each marks a significant contribution in effort.
Today we have an editor who is the first to achieve having his 5000th published story on SoylentNews: martyb (aka Bytram on IRC) has been with the site from the very beginning but has not limited his contribution to being an editor. He is also the site's one-man QA team and has spent many hours testing software and investigating many of the bugs that pop up from time to time.
Like all of us, he has to manage his own life and work too, and it is impossible to calculate the hours he has expended to keep SoylentNews worthy of being a place that many of us call 'home'.
So let me invite you to join me in thanking martyb for his contribution to this site. Congratulations martyb, and here's to the next 5000!
martyb here. It feels awkward receiving such praise so publicly. Yes, at many times it has been a labor of love. As most of you are aware, all staff work on a purely volunteer basis -- nobody has been paid for their work on the site. So it is with great pleasure that I can attest how privileged I feel to work with such an outstanding team... there is no way I could have achieved this milestone alone!
Along with editing and posting stories, we editors strive to second each story before it goes live. Many a mistake of mine has been caught and fixed by my fellow editors. I cannot thank them enough for saving my bacon on far too many occasions!
It has, indeed, been a team effort to keep the stories coming. A quick look at the Authors page shows that several editors are approaching milestones as well! janrinok (our Editor-in-Chief) just attained 3700 stories; cmn32480 is not far from 3000 stories; Fnord666 though arriving later on the scene is on the cusp of reaching 2000 stories. Coolhand, at 1121 stories, pops in from time to time and continues to help keep the story queue filled. Not to be left out, mrpg joined us comparatively recently, and yet has nearly reached 500 stories.
I would also like to take this opportunity to call out FatPhil who answered our call for editors not too long ago, as well as our newest editors chromas and fyngyrz -- please join me in welcoming them to our editorial staff!
A special thanks goes to takyon who has not only posted over 900 stories, but has also single-handily provided well over 3700 well-written story submissions which helps make our lives as editors so much easier.
A special shoutout to janrinok and LaminatorX; there was a long stretch where they were the editorial team and who, when the submissions queue went dry, would rustle up stories from across the web and submit them for the other to push out onto the site. I'd hate to imagine what would have happened without their steadfast effort and perseverance.
Lastly, a sincere thanks goes to the SoylentNews community. Your story submissions are tremendously important -- not just in giving us something to post on the site, but also in providing insight into what topics are suitable for the site, as well. Please keep those story submissions and comments coming!
Okay, that wasn't the last thing. We seem to be running behind prior periods in subscriptions to the site. We need the money to pay for the servers, domain name renewal, taxes, etc. Again, none of the staff receive any payment of any kind for our efforts -- we are all volunteers. Please take a moment to go to our Subscription Page. Even Anonymous Cowards can, via a gift subscription, make a contribution. If you are logged in and looking to start, renew, or extend your subscription, be certain that you select the correct radio button for renewal (i.e. NOT a gift subscription). Click through the FAQ link there to see the benefits to you for subscribing. Also, the amounts shown are minimums for the duration shown -- please consider changing to a larger amount to further help the site continue as a going concern.
Over the past week we've had at least three occurrences of this particular bug crop up. It's currently already fixed but I thought I'd fill you lot in just in case it got you too and you haven't noticed yet.
On the subscription page there are two radio buttons if you're logged in. One is to subscribe for yourself and one is to give a gift subscription. For some reason they were both set unchecked. If you didn't check one your subscription would to go NCommander's non-admin account, mcasadevall. It beats the complete hell out of me why this would be the default but it is.
If you've purchased a subscription recently please check that you got credit for it. If you didn't please let us know either here or via email.
It is my great pleasure to announce that SoylentNews has just celebrated four years of service to the community! The very first story on the site appeared on 2014-02-12 and actually went live to everyone 2014-02-17.
It all started when a story on Slashdot made reference to its "audience" which ticked off quite a number of people. Soon after came a boycott of Slashdot — aka the "Slashcott". While this was in effect, an intrepid few people somehow managed to take a years-old, out-of-date, unsupported, open-sourced version of slashcode and somehow managed to get it up to speed to run on much more recent versions of Apache, MySql, etc. Recurring crashes and outages were the norm. (See last year's anniversary story for many more details!) Further, on July 4th, 2014 our application to become a Public Benefit Corporation was approved — this set the stage for us to be able to accept funding from the community.
By the time you read this, we will have posted 20,980 stories to the site to which over 639,907 comments have been made!
We could not have done this without all of you. You (the community) submit the stories for the site. You write the comments... and moderate them, too. You made recommendations for improvements to the site. You are SoylentNews.
It has been my privilege and honor to work with a great group of folks who have done the behind-the-scenes skunk-work which has kept this site running. It does bear mentioning that this site is entirely staffed by volunteers. Nobody here has received even a penny's worth of income from the site. Like you, we have home and work responsibilities, but in our spare time we still strive to provide an environment that is conducive to discussions of predominantly tech-related matters.
Having said all that, I must add that income to the site has dropped recently. Having let my subscription lapse in the past, I know how easy that can be. Take a moment to check your subscription status. We have on the order of 100 people who have subscribed in the past, have visited the site in the past month, and whose subscription has expired. If your subscription is up-to-date, please consider either extending it or making a gift subscription (default is to UID 6 - "mcasadevall" aka NCommander). NB the dollar amounts presented are the minimum payments required for that duration -- we'll happily accept larger amounts. =)
If financial contributions are infeasible for you, we always appreciate story submissions. Submit a link, a few paragraphs from the story, and ideally a sentence or two about what you found interesting and send it to us. Any questions, please take a look at the Submission Guidelines.
Of course, the comments are where it's at. Thoughtful, well-reasoned, well-supported comments seem to do best here. Inflammatory histrionics garner attention, and usually down-mods, too. Speaking of which, if you have good Karma, and have been registered with the site for at least a month, you are invited to participate in moderation. Unlike other sites that up-mod or down-mod to infinity, we have something more like olympic-scoring here. A comment score can vary from -1 to a +5. This is how I look at scores: -1 (total waste of your time), 0 (meh), +1 (okay), +2 (good), +3 (quite good), +4 (very good), +5 (don't miss this one!).
As you're probably aware we experienced some unplanned downtime today. It has been claimed it was entirely the fault of Russian Hackers. They invaded fluorine and caused the database updating code in rehash to not update the database this last site update. Which is just as well, I suppose, since two of the SQL statements refuse to complete even when run manually. That I'm going to have to chalk up to a misconfigured ndbd on helium and neon.
tl;dr The long and short of it is, we'll be fine until we can get those updates into the database, but it is going to mean more downtime this weekend.
Yeah, so life has managed to delay our December Update until February. Things happen. The only change to what's in it is that I wrote up a simple plugin to syndicate content to Twitter, which is very much preferable to our current situation since we're pushing stories from a hacky little script on my desktop at the moment and I'd like to be able to boot Windows 7 for some vidya once in a while.
Downtime should be five minutes or less starting around 2:00AM UTC (an hour and forty minutes from now).
Continuing Linode's efforts to mitigate the Meltdown/Spectre issues, we have learned that the last of our servers has been scheduled for its first reboot. This time it is hydrogen which
, among other things, hosts a copy of our database is one of our web frontends. The reboot is scheduled for tonight, 2018-01-19 @ 05:00 AM UTC (Midnight EST) or approximately 9 hours from the time this story is posted. Our plans are to move hydrogen's workload over to fluorine to cover the load during the hiatus and expect there to be no interruption of service on the site.
Please be aware that Linode considers these reboots to be "Phase 1" of their remediation efforts. We will keep you posted when we learn of other phase(s) being scheduled.
We appreciate your understanding and patience as we deal with this situation.
I recently came upon an article on Ars Technica which revealed that Linode (amongst many, many others) learned of these vulnerabilities at the same time as the rest of us -- when the news hit the press. They had no advance notice with which they could have performed mitigation planning of any kind. That has to count as one of their worst days, ever.