This is to inform you we *are* aware of a problem with the site.
Symptom: The "Most Recent Journal Entries" box that normally appears on the right-hand-side of the main page is currently not currently displaying.
We have tried a few things, but have had no luck so far. We will update this story when we know more.
Problem first became apparent at about 17:45 UTC today.
Workaround: Use Search to look at the most recent journal entries. Specifically:
We will update this story when we know more. I have done all I (martyb) know how; hopefully TheMightyBuzzard will show up before too long and get things straightened out.
[Update-TMB]: The local mysqld instances on both web frontends both decided to be On The Crack for some reason or other. The database itself running over on the db servers was just fine and a restart of mysqld (and bouncing of Apache/Varnish which has to be done whenever mysqld is restarted) on the web frontends put things back to normal.
The cause of mysqld's drug use is unknown and shall remain so unless it happens more than once.
(Score: 0) by Anonymous Coward on Saturday June 27 2020, @09:13AM
Have you thought about implementing a watchdog? Most problems you seem to have are fixed by restarting the process. That could be a quick setup with the right canary calls and PID checks. The watchdog daemon is very flexible and allows custom test/repair executables and can take stronger actions if necessary. Despite what many of the tutorials online say, you don't have to resort to STONITH if you use it, despite that being the original use case.