2019-01-01 00:00:00 ..
2019-03-24 13:20:44 UTC
2019-03-24 19:30:16 UTC
We always have a place for talented people, visit the Get Involved section on the wiki to see how you can make SoylentNews better.
[Updated to correct time of neon CPU's spiking. --martyb]
We experienced an unexpected outage of the site this morning (20190110 00:15-07:45 UTC). At
shortly after midnight approximately 0415 (UTC), CPU usage on neon suddenly pegged at 400% and things went downhill from there. Am not sure at this point what happened between 0015 and 0415.
Root cause is being investigated, but for now it seems the site is back up and working. Please let us know if you have any issues.
Note: you may need to have your browser ignore its cache (e.g. refresh with Ctrl+F5) and bring down everything fresh.
FWIW, system came back up after we rebooted neon (using the Linode manager page), and then bounced varnishd on fluorine and hydrogen (/home/bob/bin/bounce on each.)
Many thanks go to SemperOSS and cosurgi for problem determination and steps to rectify and FatPhil for his cheerleading!
[Update: TMB] So, the deal was that some unknown time in the past the ndb database node on helium had gone down. This wasn't a problem since we run a clustered database but nobody noticing it was. Then last night something caused neon to lose its cheese. Since it hosts the other node of the db, we had no db for a while. Bytram(martyb) has sysadmin powers for when unpleasant substances of various types hit the fan and thankfully he knew enough to get the neon db node back up and bounce apache/varnish on the web frontends, so kudos to him and all the folks who were backseat driving at the time due to lack of admin perms on their parts.
My brain's currently fried from going from asleep to OMGWTFBBQ without so much as a cup of coffee and a cigarette first, so I'm not going to dig into the root causes until it unfries itself but as a stopgap we have four more staff with shiny, new admin access that I'll be emergency bootcamping in the very near future. There's also going to be some monitoring reimplemented very soon so we notice this kind nonsense before it blows up in our faces again. I'll either update and bump this story or post a new one if we manage to figure out what the root causes were but at the moment the logs aren't being particularly helpful.
While shutting down terminal windows to the SN servers this morning, one of them had the create table syntax for the comments table still on screen. I gave it a read through just because it was there and noticed that there wasn't an index for the opid (top level comment id for speeding up entire thread pulling) column. So I got some before numbers, added one, and ran some After tests. Heavily commented stories show a 50-150% pages-per-second speed increase in threaded views. Low hanging fruit FTW.
After a morning of slow query logging and cussing, a misplaced "GROUP BY" that was turning a 0.04 second query into one that took over fifteen seconds has been fixed. I'll leave the slow query log running overnight though just to make sure I didn't miss any less common ones. If you're still seeing any serious site slowdowns, let us know.
First off, it's been a week since we published Meta: The End of the Year is Nigh; End-of-the-Year Summary and Subscription Request and I am pleased to report we have received nearly $1000 in site subscriptions. Please accept my sincere and heartfelt thanks to all who have contributed! We are looking for another $1000 to make our funding goal. If you can help out either by starting a subscription, or extending a current one, that would be greatly appreciated! We have had a few people subscribe for $100; any amount you can provide will mean a lot.
As for IRC: it was brought to our attention a couple days ago that there was an issue in gaining access to SoylentNews' IRC securely.
tl;dr: Secure access to IRC should be functional; please reply in comments (with steps to reproduce) if you are still having issues. Read on for the scoop.
TheMightyBuzzard recently updated our Let's Encrypt certs across all of SoylentNews as they were set to expire on December 27. (Certs are issued with a 3-month duration.) Everything seemed to be working fine, so he took off on a much-deserved, week-long vacation.
I saw messages on IRC that there were issues in attempting to gain secure access to IRC. I thought that was strange as I was having no difficulty accessing it using HexChat on my PC. I tried the link on the SoylentNews home page: IRC and was also able to connect without issue. As the reports I'd seen were of the form "IRC is broken", and it was obviously working for me... now what? Pushed back asking for more details on steps to reproduce.
Some back-and-forth discussion (on IRC) isolated the problem to being involved with the recently-installed certs. They were in the correct location. They had the correct file permissions (read/write/execute). The irc daemon was running.
What could it be?
Kudos to audioguy who dove in and via guidance from Deucalion (our IRC master who was also on vacation and who was answering questions without benefit of being able to see what audioguy was looking at) was able to isolate the problem to incorrect Access Control List (ACL) settings for irc. AFAIK, nowhere else on the site do we use those — let's just say that IRC is a strange beast and leave it at that.
So, audioguy got the ACLs straightened out... yay! But it was unclear how to get ircd to actually use them. (There is a script for that but we didn't know it at the time) I had a chat (on IRC!) with chromas which worked out the command needed to restart the irc daemon (sudo kill -hup $irc_PID) and... voila! Back in service!
Oh! Anybody wondering how some of us were having no trouble securely using IRC through all this? Figured it out? Up until the move to Let's Encrypt, we used self-signed certs. I, as well as several others, still had those loaded on their system and they had not yet expired. Also, many of us were accessing irc using a 'bouncer' which would hold the connection to the daemon open and buffer transmissions between the server and our client. Connections to the bouncer were, thankfully, unaffected.
NOTE: We still have our alternate irc server, creatively named irc2, to bring forward but that can wait until staff is back from vacations.
P.S. Here's wishing all of you a Happy New Year!!!
P.P.S. When I wrote this, the fortune appearing at the bottom of the page read: "All is well that ends well. -- John Heywood".
So it's not the end of the world as we know it, but it's nearly the end of the calendar year. How'd we get here, where are we at, and what have we been up to? Oh, and please send us some monies.
It all started with slashdot attempting to push a redesigned UI (User Interface) onto the community (aka the "Beta"). There was tremendous discontent with the decision which culminated in the The Slashcott. Some like-minded folk getting together and bashing an unmaintained open-sourced version of slashcode into working order. For some of the early pre-history of the site there is a very readable entry at https://operationsysadmin.smbfc.net/. This was a Herculean task, but they somehow managed in a couple week's time to get something running and opened it up to the world on February 16, 2014. The site had some "stability issues" at first, but these tenacious folk persevered and we are now fortunate to have a stable and functioning discussion system that is fast approaching five years of operation!
Since that inauspicious beginning we have seen:
Check out the Hall of Fame for some additional statistics.
It bears mentioning that this site and its operations are supported entirely by volunteers. Nobody has ever received any payment for their efforts. In my own case, I work retail and have no two weeks' schedule the same. During the holiday season work hours are extended; as an example, I am scheduled to work past midnight tonight and the same again tomorrow. There have been many times I have popped into the site during a break and (painfully) used my mobile phone to push out a couple stories. Others here also have outside activities and commitments yet make the time to keep this site up and running.
Deucalion keeps us cats herded and keeps an eye on our IRC (Internet Relay Chat) daemons. The Mighty Buzzard handles the vast majority of actual site maintenance and security certs as well. Audioguy and Mechanicjay quietly work behind the scenes and take care of e-mail as well as other sysadmin-y tasks. Ncommander and paulej72 hover around in the background to help out when needed. Then we have fyngyrz, xyem, and FatPhil who pop in and help out from time to time. Lastly, there are the editors who are the main face of the site. Theirs is the seemingly thankless task of choosing story submissions, making corrections as needed, and queuing them up to appear on the main page. To all of you, I offer a profound and deep "THANK YOU!"
[TMB Note: He's being modest and left out that he's both Editor in Chief and HMFIC of breaking all of the things.]
I would be remiss in not mentioning the many others who were critical in the site's creation and launch who have since moved on to other projects -- though they no longer contribute to the site, their efforts were key at the time and we would not be where we are today without their efforts.
We have a tight-knit group behind the scenes, but generally go about things with a sense of humor. If you'd like to be a part of what makes this site what it is, and are willing to work with our sometimes curmudgeon-ly group, just send an e-mail to "admin [at] soylentnews.org" or mention it in the comments.
Though the staff is entirely volunteer, there are still actual costs to keeping this site operational: monthly server bills, domain fees, accountant charges for filing taxes, and the like.
We have never carried advertisements on this site. Thanks to the continuing generosity of the community, we hope to keep it that way.
As of this writing, of our $3,000 funding goal for the second half of the year, we have received approximately $1142.48 -- that's just 38.1%.
If you have already contributed, THANK YOU! (If you are in a position to do so, please consider extending your subscription.)
We have approximately 110 users who had previously subscribed to the site, and have logged into it in the past 30 days, whose subscription has lapsed. I'll be the first to admit that our reminders are not the best! Maybe that happened to you?
If you wish to remain anonymous, designate your subscription as a gift; by default it goes to "mcasadevall" who has UID 6.
Whatever amount you choose IS appreciated!! Please subscribe now.
It's not just the numbers!
The thing that keeps me volunteering here, though, is not the minutia of site operations (though I have learned so much by watching pros at work!). It is the community. I've come to know and appreciate the points of view of many regular commenters here. I'm regularly reminded people just don't fit into simple, compartmentalized boxes. Certain grand strokes may be consistent, but I've lost track of the number of times when a post has caused me to step back and go "Whoah!" Some deeply-held thought or belief gets re-examined and I'm a better person for that introspection.
The Subject Matter Experts (SMEs) also keep me coming back for more. These folk know stuff about areas that I didn't even know were areas! I am humbled by their contributions; more detailed than the press kit blurbs, but not so erudite that I need to grok the entirety of their specialty.
Whatever your interest or contribution, SoylentNews is People! Thank-you seems so little, but mine is heart-felt. Thank-you to all of you who contribute to this site, whatever way that may be.
Some of you will recall that I recently underwent several bouts of surgery and, despite your welcome comments and good wishes, your best guesses that I was having a sex change, having my breasts enhanced in either size or number, or receiving some fairly radical treatment for hemorrhoids, were all fairly wide of the mark. The surgery is now long past and I have made a reasonable recovery for someone of my age. But I do wish to thank you all for your comments because, almost without exception, they raised a smile when things were not going too well for me.
My wife suffers from a severe medical condition and I have been her full-time carer for over a decade now. Although I wish she had never developed the condition, I expend great effort, and also receive considerable personal satisfaction, in providing many hours of care each day so that she can remain in our home and we can continue our lives to the fullest extent possible. However, she has recently suffered from a deterioration in her condition. This was not unexpected but no-one could say when the next problems would affect her. But the result is that I now have to provide more support to her and my free time is reduced.
I joined this site at its inception and have enjoyed every minute of my time here. But I cannot dedicate the time that role of Editor-in-Chief (E-in-C) deserves and, several weeks ago, I made the difficult decision to stand down from the post. (I can hear the cheering from some in our community even here in France!) The fact that most of you will not have noticed any of this means that the transition has been successful. The entire editorial team (which is nowhere near as large as that phrase makes it sound!) has stepped up to the plate and has maintained the output as it was before, ably led by Martyb who has assumed the role as E-in-C in addition to his numerous other roles on this site. I am grateful to them for their efforts and support both during my time as E-in-C and more recently in their work in editing the stories that we read each day. Thanks guys, you do a tremendous job with relatively little recognition. I've asked the powers-that-be to increase your salaries by an appropriate percentage.[*] I am also grateful to the other folk who do so much in the background keeping this site on-line. You have all become good friends although we could be standing next to each other and wouldn't know it.
Equally important to the site's success are you - the community. You provide the submissions, the comments, the funding, and you are the reason that we have a site at all. I thank each and every one of you for your contribution; from the regular submitters, the ACs, the 'characters', and those of you who just visit to read the stories that we publish. If I have offended anyone then I apologise but being E-in-C has been likened to herding cats in the dark: an almost impossible task and one in which you are certain to make a few mistakes.
Hopefully, I will remain on the site as an editor making whatever contribution I can. But, for the next few months at least, that contribution will be minimal as I have to solve several practical problems on how to make our home function satisfactorily for us both. I trust that you will give Martyb and the team the same support in the future that I have been fortunate enough to enjoy over the past couple of years. Thank you.
[*] Clarification: This is an inside joke among the staff. Nobody here receives any kind of payment for their efforts on the site; we are strictly volunteers. So, a 20% raise on zero is... still zero. --martyb