Stories
Slash Boxes
Comments

SoylentNews is people

posted by martyb on Tuesday April 23 2019, @03:19PM   Printer-friendly
from the and-then-there-were-some dept.

We had a minor site hiccup today. All seems to be working, now.

We have always been open and upfront about the site, so in the interests of full disclosure here is a summary of the problem and steps taken to fix it.

tl;dr Comment counts shown for each story on the main page seem to have stopped getting updated since about midnight this morning; appears to be working now. Please accept our apologies for any who were inconvenienced.

Read on past the fold for details.

Problem: Comment counts on the main page showed "0" comments on recent stories, but opening a story showed the correct number of comments for it.

Actions Taken:

1.) Try bouncing the front-end servers to restart apache (This is a low-risk step that seems to fix a surprising number of issues).

No joy.

2.) Ask for help on the #dev channel on IRC.

Ncommander replied asking if slashd (an over-seeing daemon for the site) was running.

Looked through my log files and on the site wiki; determined that slashd should be running on server: fluorine

ps -AF | grep slashd | wc showed 32 processes

Ncommander suggested: killall -9 slashd

Try: killall -9 slashd

"No process found."

Inspection of output of PS -AF suggested this one-liner should do it:
$(ps -AF | grep slashd | awk '{print "kill -9 " $2}' )

Got most of the processes, but there still seemed to be some stragglers.

/etc/init.d/./slash stop
/etc/init.d/./slash restart

Conclusion:

Looked like it might have worked... reloaded main page... see updated comment counts!

Looks like all is working again.

It's a credit to the staff here that the site has been running so smoothly and without crashing or hiccups for... I can't remember when we last had an outage. Given that in the early days of the site we had maybe a few hours of uptime between crashes, we have come a long ways!

I'm going to assume this is one of those "have you tried turning it off and back on again" kind of problems, and unless the problem re-occurs, assume it is solved.

Need to hurry to get to work, so I apologize for the brevity of this posting.

--martyb


Original Submission

 
This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 2) by DannyB on Tuesday April 23 2019, @05:40PM (7 children)

    by DannyB (5839) Subscriber Badge on Tuesday April 23 2019, @05:40PM (#833943) Journal

    True Story: yesterday I put out an update on an internal server for a QA person to test.

    It was seriously broken. When exercising the fix, it hung. Hard.

    After getting it back this morning, I reproduced it. Initially I couldn't tell if it hung in the server or the browser. You couldn't press F12 to bring up the browser debuggifier. You couldn't click X to stop. You had to close the browser tab.

    Had a difficult time tracking down the problem. It was elusive. In fact, a cascade of things that led to it getting in a tight loop in JavaScript in the browser.

    Ultimately it was caused by a minor last minute change, so minor that it couldn't possibly cause any problem and therefore wasn't worth me doing a quick gross error check first.

    --
    People today are educated enough to repeat what they are taught but not to question what they are taught.
    Starting Score:    1  point
    Karma-Bonus Modifier   +1  

    Total Score:   2  
  • (Score: 2) by RS3 on Tuesday April 23 2019, @06:40PM (5 children)

    by RS3 (6367) on Tuesday April 23 2019, @06:40PM (#833987)

    I don't / haven't done much javascript (and I'm a detractor, but that's another discussion). Are there development tools, code analyzers, single-steppers, breakpoints, debuggers, etc. for javascript?

    Is jshint useful?

    Is this useful at all: https://raygun.com/javascript-debugging-tips [raygun.com]??

    • (Score: 4, Interesting) by DannyB on Tuesday April 23 2019, @07:13PM (4 children)

      by DannyB (5839) Subscriber Badge on Tuesday April 23 2019, @07:13PM (#833999) Journal

      I actually use some of those techniques already. The actual flaw was in the server code. The symptom was it would cause an infinite loop in JS code.


      It's an accounting application. A money amount is being pro-rated across several different accounts something is being charged to. On server, loop over the account lines, and take that line's fraction of the total, and create a charge for it against that account. Once you do this, all of the account lines added together may not pay the entire amount. (eg, prorate a $10 charge evenly split among 3 accounts. Each account gets $3.33, totaling to $9.99) Some account(s) must pay any remaining unpaid cents. So loop over those account lines again, distributing extra pennies among accounts until the total $10 amount has been paid. In this case, one iteration, adding one cent to the first account. (Over simplified because three account lines doesn't mean that each of those accounts is going to pay a 1/3 portion of the total charge.)

      Now imagine what happens if the server doesn't generate some of the account line information into the page. (Not seen by the user.) The user sees the UI normally. Does an operation -- which happens in JS code in the browser, so they can see it before actually sending the change to the server. But since there are no account lines, then that 2nd loop that loops until the entire amount is paid, never terminates.

      This is a bit of an oversimplification.

      I don't / haven't done much javascript (and I'm a detractor, but that's another discussion).

      Nothing wrong with that. I wouldn't generally want to use JS if it isn't needed. I now try to do animations and other things using CSS and other techniques where possible -- and fully supported among mainstream browsers.

      In a business application, JS is simply a fact of life. This is an application. Not a 'web site'. It is highly interactive with the user and with the server. None of the important business logic happens in the browser -- other than to the extent it supports the UI interaction. All information sent to the server is verified, even redundantly because we don't trust it just because we think it came from our code in the browser.

      Simply put, the UI experience without JS would be vastly inferior for this type of application.

      --
      People today are educated enough to repeat what they are taught but not to question what they are taught.
      • (Score: 2) by RS3 on Tuesday April 23 2019, @07:37PM (1 child)

        by RS3 (6367) on Tuesday April 23 2019, @07:37PM (#834007)

        What an awesome answer, thanks! Wow, cool stuff! You're obviously a great programmer. Kudos! Sounds like your application wants to calculate pi next? :-} If you do, please let me know- I'll start a CPU cooling fan business. But I digress (yet again...)

        Speaking of a truly treat UI, I just had to use the McMaster-Carr website a few days ago (looking up screws, etc.) What I truly LOVED was that the column headings STAYED VISIBLE as I scrolled through the pages of screw sizes. It's really clean. If you don't already do that, you might look at their code, get ideas, adapt (never copy...).

        I don't hate js, in fact as an occasional programmer, I think it's awesome and I'm big on the client-server model. I got to hating js because especially back in the bad old POTS modem days, (late 90s into 2000s) I quickly discovered that if I turned js OFF, all websites would load MUCH faster. I started using Opera which let me do that, and I still use Old Opera (12.18 or something) because of that. Looking at the code I saw js "pulling" in all kinds of trackers, analytical, ads, etc. Don't need that!! JS that does F to C, or furlongs to meters, that's way cool.

        What I also and mostly hate about js is really the fault of the browser and OS- far too much of my computer, files, etc., is available to js. That most malware uses js really set me off. Disposable containerized browsing is the only true safe way. Of course that depends on a 100% robust OS / virtualization...

        You mentioned "flaw in server code" and I'm a bit confused. Is it "backend" js? Or browser js that was pulling in 3rd party js libs?

        • (Score: 2) by DannyB on Tuesday April 23 2019, @08:39PM

          by DannyB (5839) Subscriber Badge on Tuesday April 23 2019, @08:39PM (#834029) Journal

          Wow, cool stuff!

          Accounting application, not that cool, but it pays the bills. As I have described on SN previously, customers given a choice, prefer to subscribe to it in the cloud even when offered an option to install it on their own equipment.

          It's not like the math is complicated. Although the application is complex in its sheer size, database design complexity, etc.

          What I truly LOVED was that the column headings STAYED VISIBLE as I scrolled through the pages of screw sizes. It's really clean.

          That is one feature on Wikipedia Tables, and also other web sites that I despise.

          far too much of my computer, files, etc., is available to js.

          My employer does not do any monkey business. It's not our business model. That includes not trying to collect and sell information.

          Is it "backend" js? Or browser js that was pulling in 3rd party js libs?

          The server backend is Java on a beefy server. Front end browser is JS code. There are 3rd party libs (example jquery) but served from our servers. Nothing is pulled from a 3rd party site. No frames. Strictly SSL/TLS.

          --
          People today are educated enough to repeat what they are taught but not to question what they are taught.
      • (Score: 2) by RS3 on Tuesday April 23 2019, @07:39PM (1 child)

        by RS3 (6367) on Tuesday April 23 2019, @07:39PM (#834008)

        PS: you're probably not old enough, but way back in the day when a new computer would come out, like the Cray-1, people would joke that "it does an infinite loop in 5 seconds".
        Ar ar ar.

        • (Score: 2) by DannyB on Tuesday April 23 2019, @08:43PM

          by DannyB (5839) Subscriber Badge on Tuesday April 23 2019, @08:43PM (#834032) Journal

          I am old enough to remember that. When BYTE magazine started, I was in high school. :-)

          --
          People today are educated enough to repeat what they are taught but not to question what they are taught.
  • (Score: 0) by Anonymous Coward on Thursday April 25 2019, @11:15AM

    by Anonymous Coward on Thursday April 25 2019, @11:15AM (#834697)

    Ahhhh yes
    Minor last minute changes that "can't possibly affect anything"

    Yeah. Seen a few of those before. Nasty when it happens in Production.