SoylentNews

[Update 1; 2020-12-27 16:05:32 UTC: Added my conversation with TMB. Scroll down to view. Minor rewording of original story. --martyb]
[Update 2; 2020-12-27 19:38:48 UTC: TMB started investigation and added a short update below. --Bytram]

aristarchus writes:

Not an actual sub. If this had been an actual sub, there would be something here. But just noticing that the list of recent journal entries is gone from the front page?

Thanks for that, aristarchus!

We've got a problem:
Where did the "Most Recent Journal Entries" slashbox go?

I don't know what happened. And, of course, TheMightyBuzzard is on vacation for the week.

No code changes have been made recently. Hmmm.

Way back when, I had read Niklaus Wirth's Algorithms+Data Structures = Programs. Well, if it's not code, then maybe it's data? It's a longshot, but worth a look.

I looked in the DB. Found the most recent journal entry. Looked like a Cut-and-Paste of some computer-generated HTML wrapping some marketing blurb written in Portuguese. Not only were there inappropriate HTML tags, they were saddled down with a plethora of attributes. For more fun, ladle in a heaping helping of UNICODE characters.

Still doubtful, but seemed the most likely culprit given the timing. I updated that journal entry manually in the DB to replace journal.article text with something benign.

No Joy. :(

FWIW, I also "bounced" the servers (varnishd on hydrogen and fluorine). Still no luck.

Current Workaround:
Use the site's search function to list the most recent journal entries:

https://soylentnews.org/search.pl?tid=&query=&author=&sort=2&op=journals

Besides, I had nothing else to do first thing on a Sunday morning, right?

[Update #2]:

I don't have the patience to fix db transaction timeouts on someone else's chicklet-keyboard-having Windows laptop. It can wait until I get home on the 2nd. --TMB

[Update #1]:

(See below the fold.)

I just got off the phone with TMB (The Mighty Buzzard)... (thanks Buzz for taking time from your vacation to help!)

After originally posting this story, started seeing the site behave even more erratically. None of the Slashboxes on the LHS (Left-Hand Side) of the page were displayed. Story title bars were not displayed. Admin bar (staff only) was not displayed. Without the Admin bar, staff can't do much at all. Ruh-roh!

I checked in on IRC and got confirmation that others were also experiencing issues.

Nothing to do but ask for help; txted TMB alerting him to the issues. After a couple txts back and forth, I called him up and he walked me through several things. There's more, but here's the high-level overview of what happened. (Details in the <spoiler>)

We started by checking free disk space and memory on fluorine (our main MySQL server). df and free Looked okay.

Used less to look at the apache error_log. Terribly slow (it's huge)!

Let's try another approach: tail -F error_log and watched a few seconds and started seeing Perl/mysql error messages appear.

Restarted MySQL on fluorine and then bounced apache on fluorine.

Site seems to have stopped its hiccups. But, we still do not have the MRJE (Most Recent Journal Entries).

TMB is still waking up; will take a look at it later when he can get to a computer. Since he is on vacation, and things seem stable atm, I suggested he didn't need to rush.

*SPOILER* (click to show) *SPOILER* (click to hide)

martyb@fluorine:/srv/soylentnews.org/apache/logs$ tail -F error_log
Sun Dec 27 15:13:04 2020] [error] /meta/comments.pl:Slash::DB::MySQL:/srv/soylentnews.org/perl/lib/perl5/site_perl/5.20.1/Slash/DB/MySQL.pm:9343:virtuser='slash' -- hostinfo='Localhost via UNIX socket' -- Lock wait timeout exceeded; try restarting transaction -- SELECT  id FROM journals WHERE discussion='36753'  ;; Which was called by:Slash::DB::MySQL:/srv/soylentnews.org/perl/lib/perl5/site_perl/5.20.1/Slash/DB/MySQL.pm:9279
[Sun Dec 27 15:13:08 2020] [error] /meta/comments.pl:Slash::DB::MySQL:/srv/soylentnews.org/perl/lib/perl5/site_perl/5.20.1/Slash/DB/MySQL.pm:9343:virtuser='slash' -- hostinfo='Localhost via UNIX socket' -- Lock wait timeout exceeded; try restarting transaction -- SELECT  id FROM journals WHERE discussion='36753'  ;; Which was called by:Slash::DB::MySQL:/srv/soylentnews.org/perl/lib/perl5/site_perl/5.20.1/Slash/DB/MySQL.pm:9279
[Sun Dec 27 15:13:14 2020] [error] /meta/comments.pl:Slash::DB::MySQL:/srv/soylentnews.org/perl/lib/perl5/site_perl/5.20.1/Slash/DB/MySQL.pm:9343:virtuser='slash' -- hostinfo='Localhost via UNIX socket' -- Lock wait timeout exceeded; try restarting transaction -- SELECT  id FROM journals WHERE discussion='36753'  ;; Which was called by:Slash::DB::MySQL:/srv/soylentnews.org/perl/lib/perl5/site_perl/5.20.1/Slash/DB/MySQL.pm:9279
[Sun Dec 27 15:13:18 2020] [error] /meta/comments.pl:Slash::DB::MySQL:/srv/soylentnews.org/perl/lib/perl5/site_perl/5.20.1/Slash/DB/MySQL.pm:9343:virtuser='slash' -- hostinfo='Localhost via UNIX socket' -- Lock wait timeout exceeded; try restarting transaction -- SELECT  id FROM journals WHERE discussion='6511'  ;; Which was called by:Slash::DB::MySQL:/srv/soylentnews.org/perl/lib/perl5/site_perl/5.20.1/Slash/DB/MySQL.pm:9279
^C
martyb@fluorine:/srv/soylentnews.org/apache/logs$ sudo /etc/init.d/mysql_sn restart
* Stopping MySQL database server mysqld

   ...done.
* Starting MySQL database server mysqld
   ...done.
martyb@fluorine:/srv/soylentnews.org/apache/logs$ /home/bob/bin/bounce
* Stopping HTTP accelerator varnishd
   ...done.
* Starting HTTP accelerator varnishd
   ...done.
martyb@fluorine:/srv/soylentnews.org/apache/logs$ tail -F error_log

[Update_End]

Original Submission