We had a minor site hiccup today. All seems to be working, now.
We have always been open and upfront about the site, so in the interests of full disclosure here is a summary of the problem and steps taken to fix it.
tl;dr Comment counts shown for each story on the main page seem to have stopped getting updated since about midnight this morning; appears to be working now. Please accept our apologies for any who were inconvenienced.
Read on past the fold for details.
Problem: Comment counts on the main page showed "0" comments on recent stories, but opening a story showed the correct number of comments for it.
Actions Taken:
1.) Try bouncing the front-end servers to restart apache (This is a low-risk step that seems to fix a surprising number of issues).
No joy.
2.) Ask for help on the #dev channel on IRC.
Ncommander replied asking if slashd (an over-seeing daemon for the site) was running.
Looked through my log files and on the site wiki; determined that slashd should be running on server: fluorine
ps -AF | grep slashd | wc showed 32 processes
Ncommander suggested: killall -9 slashd
Try: killall -9 slashd
"No process found."
Inspection of output of PS -AF suggested this one-liner should do it:
$(ps -AF | grep slashd | awk '{print "kill -9 " $2}' )
Got most of the processes, but there still seemed to be some stragglers.
/etc/init.d/./slash stop
/etc/init.d/./slash restart
Conclusion:
Looked like it might have worked... reloaded main page... see updated comment counts!
Looks like all is working again.
It's a credit to the staff here that the site has been running so smoothly and without crashing or hiccups for... I can't remember when we last had an outage. Given that in the early days of the site we had maybe a few hours of uptime between crashes, we have come a long ways!
I'm going to assume this is one of those "have you tried turning it off and back on again" kind of problems, and unless the problem re-occurs, assume it is solved.
Need to hurry to get to work, so I apologize for the brevity of this posting.
--martyb
(Score: 3, Funny) by Anonymous Coward on Tuesday April 23 2019, @03:28PM (4 children)
Can I get a refund for my 1 post that wasn't counted?
(Score: 4, Funny) by c0lo on Tuesday April 23 2019, @03:37PM (3 children)
Nope; but here, have a voucher for another post that will be counted.
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 2) by DannyB on Tuesday April 23 2019, @05:34PM
Give me Liberty or give me something of lesser or equal value! Or a coupon for it!
Protesters: we demand change! we demand change!
Child: hey dad, why aren't they demanding paper currency instead of change?
To transfer files: right-click on file, pick Copy. Unplug mouse, plug mouse into other computer. Right-click, paste.
(Score: 4, Touché) by MostCynical on Tuesday April 23 2019, @10:10PM (1 child)
Voucher issued as part of a promotional activity
Voucher expires 24 hours after issue.
Not redeemable for cash
Not transferable
Limit one voucher per user (all ACs have to share)
Voucher must be presented at time of posting
Voucher must be presented in person to a site admin
"I guess once you start doubting, there's no end to it." -Batou, Ghost in the Shell: Stand Alone Complex
(Score: 3, Insightful) by coolgopher on Tuesday April 23 2019, @11:50PM
Terms and conditions may change without notice.
(Score: 0) by Anonymous Coward on Tuesday April 23 2019, @03:45PM (8 children)
In general, these kinds of problems always bug me. Unless a random problem is the result of someone just dorking around on the server, it usually means there is something wrong somewhere even if it is an edge case. There will always be small unfixed bugs in whatever system, but with me it always seems to be the most obscure, random, impossible to intentionally recreate, bugs that cause the most headache.
Have you being rebooted your compooter? :P
Nice thing about an occasional full reboot it it usually rules out any environment problem in RAM caused by random messing about.
(Score: 2) by DannyB on Tuesday April 23 2019, @05:40PM (7 children)
True Story: yesterday I put out an update on an internal server for a QA person to test.
It was seriously broken. When exercising the fix, it hung. Hard.
After getting it back this morning, I reproduced it. Initially I couldn't tell if it hung in the server or the browser. You couldn't press F12 to bring up the browser debuggifier. You couldn't click X to stop. You had to close the browser tab.
Had a difficult time tracking down the problem. It was elusive. In fact, a cascade of things that led to it getting in a tight loop in JavaScript in the browser.
Ultimately it was caused by a minor last minute change, so minor that it couldn't possibly cause any problem and therefore wasn't worth me doing a quick gross error check first.
To transfer files: right-click on file, pick Copy. Unplug mouse, plug mouse into other computer. Right-click, paste.
(Score: 2) by RS3 on Tuesday April 23 2019, @06:40PM (5 children)
I don't / haven't done much javascript (and I'm a detractor, but that's another discussion). Are there development tools, code analyzers, single-steppers, breakpoints, debuggers, etc. for javascript?
Is jshint useful?
Is this useful at all: https://raygun.com/javascript-debugging-tips [raygun.com]??
(Score: 4, Interesting) by DannyB on Tuesday April 23 2019, @07:13PM (4 children)
I actually use some of those techniques already. The actual flaw was in the server code. The symptom was it would cause an infinite loop in JS code.
Nothing wrong with that. I wouldn't generally want to use JS if it isn't needed. I now try to do animations and other things using CSS and other techniques where possible -- and fully supported among mainstream browsers.
In a business application, JS is simply a fact of life. This is an application. Not a 'web site'. It is highly interactive with the user and with the server. None of the important business logic happens in the browser -- other than to the extent it supports the UI interaction. All information sent to the server is verified, even redundantly because we don't trust it just because we think it came from our code in the browser.
Simply put, the UI experience without JS would be vastly inferior for this type of application.
To transfer files: right-click on file, pick Copy. Unplug mouse, plug mouse into other computer. Right-click, paste.
(Score: 2) by RS3 on Tuesday April 23 2019, @07:37PM (1 child)
What an awesome answer, thanks! Wow, cool stuff! You're obviously a great programmer. Kudos! Sounds like your application wants to calculate pi next? :-} If you do, please let me know- I'll start a CPU cooling fan business. But I digress (yet again...)
Speaking of a truly treat UI, I just had to use the McMaster-Carr website a few days ago (looking up screws, etc.) What I truly LOVED was that the column headings STAYED VISIBLE as I scrolled through the pages of screw sizes. It's really clean. If you don't already do that, you might look at their code, get ideas, adapt (never copy...).
I don't hate js, in fact as an occasional programmer, I think it's awesome and I'm big on the client-server model. I got to hating js because especially back in the bad old POTS modem days, (late 90s into 2000s) I quickly discovered that if I turned js OFF, all websites would load MUCH faster. I started using Opera which let me do that, and I still use Old Opera (12.18 or something) because of that. Looking at the code I saw js "pulling" in all kinds of trackers, analytical, ads, etc. Don't need that!! JS that does F to C, or furlongs to meters, that's way cool.
What I also and mostly hate about js is really the fault of the browser and OS- far too much of my computer, files, etc., is available to js. That most malware uses js really set me off. Disposable containerized browsing is the only true safe way. Of course that depends on a 100% robust OS / virtualization...
You mentioned "flaw in server code" and I'm a bit confused. Is it "backend" js? Or browser js that was pulling in 3rd party js libs?
(Score: 2) by DannyB on Tuesday April 23 2019, @08:39PM
Accounting application, not that cool, but it pays the bills. As I have described on SN previously, customers given a choice, prefer to subscribe to it in the cloud even when offered an option to install it on their own equipment.
It's not like the math is complicated. Although the application is complex in its sheer size, database design complexity, etc.
That is one feature on Wikipedia Tables, and also other web sites that I despise.
My employer does not do any monkey business. It's not our business model. That includes not trying to collect and sell information.
The server backend is Java on a beefy server. Front end browser is JS code. There are 3rd party libs (example jquery) but served from our servers. Nothing is pulled from a 3rd party site. No frames. Strictly SSL/TLS.
To transfer files: right-click on file, pick Copy. Unplug mouse, plug mouse into other computer. Right-click, paste.
(Score: 2) by RS3 on Tuesday April 23 2019, @07:39PM (1 child)
PS: you're probably not old enough, but way back in the day when a new computer would come out, like the Cray-1, people would joke that "it does an infinite loop in 5 seconds".
Ar ar ar.
(Score: 2) by DannyB on Tuesday April 23 2019, @08:43PM
I am old enough to remember that. When BYTE magazine started, I was in high school. :-)
To transfer files: right-click on file, pick Copy. Unplug mouse, plug mouse into other computer. Right-click, paste.
(Score: 0) by Anonymous Coward on Thursday April 25 2019, @11:15AM
Ahhhh yes
Minor last minute changes that "can't possibly affect anything"
Yeah. Seen a few of those before. Nasty when it happens in Production.
(Score: -1, Troll) by Anonymous Coward on Tuesday April 23 2019, @03:53PM (3 children)
Did you see any Moslems lurking around the data center? Before you jump all over this remember that people were scoffing at the idea that the Sri Lanka attacks were done by Moslems.. which turned out to be 100% correct.
(Score: 0, Flamebait) by Anonymous Coward on Tuesday April 23 2019, @04:18PM (1 child)
The servers were not working properly, not blown up.
(Score: 3, Touché) by Anonymous Coward on Tuesday April 23 2019, @05:30PM
Yes, but the fact that they were *headless* servers suggests the jihadis had been and gone.
(Score: 2) by DannyB on Tuesday April 23 2019, @05:43PM
They weren't scoffing. They were looking for the wrong thing. Watching carefully for Siri Lanka attacks.
To transfer files: right-click on file, pick Copy. Unplug mouse, plug mouse into other computer. Right-click, paste.
(Score: 1, Informative) by Anonymous Coward on Tuesday April 23 2019, @05:23PM (5 children)
Learn about pgrep and pkill, especially the -f flag
(Score: 0) by Anonymous Coward on Tuesday April 23 2019, @05:29PM
They replace most combination of ps and grep and kill that you can think of
(Score: 2) by RS3 on Tuesday April 23 2019, @06:43PM (2 children)
Thank you, didn't know them. I rarely need to kill anything, but those are great to add to the toolbox.
(Score: 0) by Anonymous Coward on Tuesday April 23 2019, @07:40PM (1 child)
I felt really stupid for spending a lot of time writing very robust shell functions to pgrep and pkill when I found out about them.
(Score: 2) by RS3 on Wednesday April 24 2019, @02:03AM
The knowledge and neural connections you gained has only enhanced you, your work, and who you are. (I feel like I should have written that in Yoda-ese).
(Score: 0) by Anonymous Coward on Tuesday April 23 2019, @11:11PM
Be carefull with pkill unless using the -x flag too.
(Score: 1, Informative) by Anonymous Coward on Tuesday April 23 2019, @05:42PM (1 child)
Nobody comments here anyways. No biggie.
(Score: 4, Funny) by DannyB on Tuesday April 23 2019, @05:44PM
Your counter must be broken.
To transfer files: right-click on file, pick Copy. Unplug mouse, plug mouse into other computer. Right-click, paste.
(Score: 2) by RS3 on Tuesday April 23 2019, @06:45PM (6 children)
Is "bounce" nerdspeak for "reboot"?
I forget which OS these run (and too lazy to look it up) but "apachectl start | stop | restart" works very well and much faster than "service httpd restart" for ex.
(Score: 2) by sjames on Tuesday April 23 2019, @11:20PM (2 children)
apachectl graceful is a better choice there unless something is clearly hung up..
(Score: 2) by RS3 on Wednesday April 24 2019, @02:06AM (1 child)
There's no such thing as hung up- I have the power cord in my hand nyuk nyuk.
But you reminded me of "zombie" processes. I haven't seen them in years. Life really is getting better.
(Score: 2) by sjames on Wednesday April 24 2019, @04:08AM
I certainly don't miss those :-)
(Score: 3, Informative) by The Mighty Buzzard on Wednesday April 24 2019, @12:37PM (2 children)
Just stopping restarting the apache/varnish processes. I even wrote a script named "bounce" so folks not comfortable with the system can do everything properly and in the correct order.
The actual cause was slashd though, which is basically a silly-assed reinvention of a cron daemon by the the folks who wrote slashcode in the first place. It only takes about a minute to fix, even counting sshing in and such, if you've done it a time or three. If it gave us problems more than once a year or so, I'd look into fixing it. I was out fishing/camping this time or it wouldn't have caused poor martyb any headaches this time around.
My rights don't end where your fear begins.
(Score: 2) by RS3 on Wednesday April 24 2019, @02:57PM (1 child)
> I was out fishing/camping this time...
Good! I need to make time for something outdoors more than an hour here and there.
Someday the really big fish will reel YOU in. Then martyb, et al, will learn what "we're fu....." means!
slashd is the systemd of slashcode? Someday I'm gonna download that slashcode and marvel...
What would happen if your ran your "bounce" script in cron.weekly or monthly just for the heck of it? Or maybe the problem occurs randomly, well, due to an unknown problem and bounce needs run on demand. So maybe a cron script that scans maybe every 5 minutes for whatever the problem's symptoms are and calls (or does) bounce?
(Score: 2) by The Mighty Buzzard on Wednesday April 24 2019, @11:43PM
Well, bounce didn't need run this time. slashd is an independent daemon that almost never gives us any trouble but requires someone who knows what/how to bitchslap, or at least someone able to figure it out.
My rights don't end where your fear begins.