[2021-02-14 15:53:00 UTC: UPDATE added need to check apache log before doing a slash -restart]
We seem to have experienced some difficulties with the SoylentNews site.
I've noticed that both the number of hits and comments for each story do not seem to be updating.
Corrective measures taken:
This is my personal "bounce" script:
cat ~/bin/bounce#!/bin/bash
servers='hydrogen fluorine'
for server in ${servers} ; do echo Accessing: ${server} && rsh ${server} /home/bob/bin/bounce ; doneWhich, in turn, runs the following script on each of the above servers:
cat /home/bob/bin/bounce
#!/bin/bash
sudo /etc/init.d/varnish restart
sudo -u slash /srv/soylentnews.org/apache/bin/apachectl -k restart
# Go to the appropriate server:
ssh fluorine
# Ensure the apache log is not showing issues: tail -f /srv/soylentnews.org/apache/logs/error_log
# Restart slash:
sudo /etc/init.d/slash restart
>> slashd slash has no PID file
>> Sleeping 10 seconds in a probably futile attempt to be clean: ok.
>> Starting slashd slash: ok PID = 3274
NB: this failed to run to a successful conclusion when I originally tried it a few hour ago. I gave it one more try while writing this story... it seemed to run okay this time?!
Things appears to be running okay, now. Please reply in the comments if anything else is amiss. Alternatively, mention it in the #dev channel on IRC (Internet Relay Chat, or send an email to admin (at) soylentnews (dot) org.
We now return you to the ongoing discussion of: teco or ed?
(Score: 5, Informative) by The Mighty Buzzard on Sunday February 14 2021, @05:11AM (8 children)
The db is fine. Slashd has always been slightly persnickety though, so it needs restarted a couple times a year. When the comment counts aren't updating, it's always slashd. This was just martyb's first time trying to restart it and he wasn't aware that the init script had to be run as root and that the user slash (the user that all the SN-specific stuff on the web frontends runs as) doesn't have sudo perms of any sort.
My rights don't end where your fear begins.
(Score: 0) by Anonymous Coward on Sunday February 14 2021, @09:29AM (4 children)
If slash seems to require restarting a few times a year, does there seem to be any pattern of how many days before that seems to happen? Or is it completely random? Because if it is the former, maybe some preventative scheduled reboot would be in order. But you've probably already thought of that.
(Score: 0) by Anonymous Coward on Sunday February 14 2021, @11:04AM (3 children)
I've suggested various watchdogs in the past but the hiccups seem to be too random and far between to be worth the trouble.
(Score: 3, Interesting) by The Mighty Buzzard on Sunday February 14 2021, @11:32AM (2 children)
Bingo. The "a couple times a year" is on average. I don't remember having to restart it in 2019 at all aside from it being restarted on server reboots.
My rights don't end where your fear begins.
(Score: 2) by RS3 on Sunday February 14 2021, @12:06PM (1 child)
cron.monthly job? Maybe with a file that contains a countdown variable so the restart only happens every so many months?
Hate those kinds of workarounds though. If I had time I'd look into the code...
(Score: 3, Interesting) by The Mighty Buzzard on Sunday February 14 2021, @02:16PM
Don't know that the issue is time-based but it wouldn't hurt anything to restart it once in a while.
My rights don't end where your fear begins.
(Score: 3, Insightful) by martyb on Sunday February 14 2021, @02:26PM (2 children)
/me makes a mental note of this for future reference.
I think I may have done this once before, but it's certainly not something I am entirely comfortable with.
Was not aware that user slash "doesn't have sudo perms of any sort". And... now I know; thanks!
Wit is intellect, dancing.
(Score: 3, Insightful) by The Mighty Buzzard on Sunday February 14 2021, @03:02PM (1 child)
Nod nod, we don't give slash sudo perms so we don't have to worry as much about it being an attack vector that could compromise the entire server. Not that it'd be terribly easy anyway being as the web frontends are behind an nginx reverse proxy that we're using as a load balancer. But any bit of extra security that doesn't slow stuff down or take too much effort is worth doing.
My rights don't end where your fear begins.
(Score: 0) by Anonymous Coward on Sunday February 14 2021, @09:49PM
Just one reverse proxy? Why not a double reverse?