Slash Boxes

SoylentNews is people


Log In

Log In

Create Account  |  Retrieve Password

Site News

Join our Folding@Home team:
Main F@H site
Our team page

Funding Goal
For 6-month period:
2019-07-01 to 2019-12-31
(All amounts are estimated)
Base Goal:


Covers transactions:
2019-07-01 00:00:00 ..
2019-11-12 18:13:17 UTC
(SPIDs: [1128..1178])
Last Update:
2019-11-14 09:42:28 UTC

Support us: Subscribe Here
and buy SoylentNews Swag

We always have a place for talented people, visit the Get Involved section on the wiki to see how you can make SoylentNews better.

posted by The Mighty Buzzard on Wednesday November 13, @11:10PM   Printer-friendly
from the lingere dept.

Just a quick note to let those of you who care know that our load balancer finally got bumped up to openssl 1.1.x and is now TLSv1.3 happy. For those of you who are especially paranoid, "ssl_early_data" is explicitly set to "off" in the nginx conf file, actively disabling 0-RTT, even though it's disabled by default.

That's all, carry on.

posted by Fnord666 on Saturday November 09, @05:24PM   Printer-friendly
from the hoping-for-a-speedy-recovery dept.

The observant among you may have noticed I'd been posting fewer stories recently.

On October 27th I went to the hospital. After a physical examination, CT scan, and MRI, it was determined that I had had a minor stroke. I had mild loss of use of the pinky and ring finger of my left hand as well as some loss of fine motor control in my left arm and leg. A couple days later saw me in the operating room with a stent being inserted in my right carotid artery. I was released on Halloween day for a few days' recuperation at a relative's home and am now cleared to go back to work. I'm starting easy with slightly shortened work days, as I lack the stamina I had before.

Still, as these things go, I can't help but think of how fortunate I am. Nothing internal seems to have been affected (heart, lungs, etc. all working fine.) I am right-handed, so no problems there. No problem with talking or swallowing, so that's a huge plus. Thanks to neural plasticity and prescribed exercises, I have already mostly recovered. The lack of stamina manifests as my just being too tired after a day at work to be able to do much in the way of posting stories out to the site. As for my mind, I can attest that I am still as tarp as a shack! =)

As much as I would like to think I'll just bounce back to normal in no time, I acknowledge that my activities here will be at a somewhat diminished capacity for a while; time will tell.

With my absence, the rest of the editorial team rose to the occasion and kept the main page fed with stories. This meant extra time and effort on their part. Please join me in thanking Fnord666, Janrinok, cmn32480, takyon, chromas, NotSanguine, and CoolHand for their efforts to help push out stories to the main page. There were probably others whom I failed to notice; please accept my apologies for their omission. Call them out in the comments, and join me in thanking them for their efforts.

On a related note, it is my pleasure to announce that Fnord666 has accepted my invitation to step up to fill the position of Alternate-Editor-in-Chief. When janrinok was Editor-in-Chief (EiC), I accepted becoming Alternate-EiC, and when he stepped down as EiC, it was a privilege to take on becoming the EiC. I foresee no imminent demise on my part, but recent events made it abundantly clear to me the value of having this position filled. Please join me in congratulating Fnord666 on his promotion!

(For completeness' sake, I wish to point out that all of the staff at SoylentNews are volunteers. Nobody here has ever been paid anything for their work on the site. Any monies received when you subscribe go towards paying hosting fees, domain name registration, tax preparation expenses, and other costs required to keep this site running. Speaking of which, we are at nearly 70% of our goal for this half of our fiscal year—many thanks to those who have already subscribed!)

Lastly, I now place a request to the community. With the holiday season coming, my free time will become even more limited by work demands. It would be such a help to us to see story submissions from members of the community. It's really not that hard to do. Take a look at what the general layout of each article looks like on the site. Nothing very fancy or elaborate is needed. Click the Submit Story link in the SlashBox on the left-hand side of the main page of this site. Provide a title for the story, select a topic, provide a link to what you are writing in about, and a few paragraphs from the linked story would prove extremely helpful to us! We aim for mostly tech-oriented stories, but that is not an absolute requirement. Please submit your story in English (either North American, British or other variant is fine). Do be aware that we aim for balanced reporting here; attempts to push an agenda with one-sided, biased, or slanted submissions will likely result in the story being declined. The story submission page provides a link to the Submission Guidelines that, if followed, greatly improves the chance that your story will be accepted. (NB: We tend to relax the standards a bit on the weekend so as to include slightly offbeat or humorous stories; we appreciate a good laugh, too!

Original Submission

posted by martyb on Saturday October 19, @09:00PM   Printer-friendly
from the so-much-for-uptimes dept.

[Update (2019/10/19 21:02:00 UTC): Both sodium and fluorine have rebooted. Next up will be beryllium rebooting 1d12h from now. That leaves rebooting of helium 18h later and 3h after that will have boron being rebooted. Again, any impact visible to the community should be minimal. See TMB's note, below. --martyb]

We have just learned that Linode, the provider of SoylentNews' server infrastructure, is planning a number of reboots.

[TMB Note]: This shouldn't mean any downtime for anything user-facing except IRC. There will be a few minutes where the comment counts won't update on the front page but those aren't realtime anyway and a few minutes where subscription updates will be delayed until the server that processes them comes back up.

Recently, we identified a commit to the upstream Linux kernel[1] as the cause of an increase in emergency maintenance on our platform. After implementing, testing, deploying, and gaining confidence in a fix, we are now ready to roll this update out to the remainder of our fleet. We're confident this will resolve the bug and ultimately lessen the amount of unplanned maintenance for your Linodes as a result of this specific issue.

To complete this, we will be performing maintenance on a subset of Linode's host machines. This maintenance will update the underlying infrastructure that Linodes reside on and will not affect the data stored within them.

If you are on an affected host, your maintenance window will be communicated to you via a Support ticket within the next few days. You can prepare your Linode for this maintenance by following our Reboot Survival Guide[2].

During the actual maintenance window, your Linode will be cleanly shut down and will be unavailable while we perform the updates. A two-hour window is allocated, however the actual downtime should be much less. After the maintenance has concluded, each Linode will be returned to its last state (running or powered off).

This status page will be updated once maintenance is complete.


The first server reboot is currently scheduled for Friday, 2019-10-18 at 05:00:00 UTC.

Read on after the fold for more details on the scheduled maintenance dates and times.

Note: All dates and times are in UTC:

Affected systems:

lithiumNo Maintenance RequiredLinode 4GB 
magnesiumNo Maintenance RequiredLinode 2GB(pending upgrade)
sodium2019-10-18 05:00 AMLinode 2GB 
fluorine2019-10-19 02:00 AMLinode 8GB(pending upgrade)
helium2019-10-22 03:00 AMLinode 8GB 
hydrogenNo Maintenance RequiredLinode 8GB 
neonNo Maintenance RequiredLinode 8GB 
beryllium2019-10-21 09:00 AMLinode 4GB(pending upgrade)
boron2019-10-22 05:00 AMLinode 4GB(pending upgrade)

Original Submission

posted by martyb on Monday October 14, @03:00AM   Printer-friendly

I am inviting the editorial team to take a much-deserved "break" this weekend by using "weekend story spacing"[*] on Monday. This is a long holiday weekend in the United States in celebration of Columbus Day (or Indigenous Peoples' Day). As a result, sites tend to post fewer stories. And, of the stories that are posted, a larger fraction are "filler" stories or fluff pieces, if you will. This, in turn, makes it harder for the editorial staff to find stories of interest to post to SoylentNews.

One of our editors is still on leave and the remaining staff has been stretched thin with his absence. Further, several of the editorial staff are facing real-life challenges that conspire to reduce the amount of time and energy that can be given to posting stories on SoylentNews. Do recall that all staff here are volunteers and what you see here is freely given of their own spare time.

We generally try to post 14-15 stories per day on weekdays, and about 10 stories per day on weekends.

Also, a reminder that Linode has informed us of some server maintenance they will need to perform. Except for a short while on IRC (Internet Relay Chat), any downtime should not be visible to the community. Linode reserves up to a two-hour window for their maintenance, but past experience has show that most prior maintenance is completed in less than 30 minutes and often as little as 10-15 minutes. See our earlier story Linode to Perform Maintenance; Several SoylentNews Servers Selected for Servicing for details. The first of our servers to be affected is sodium whose maintenance window starts: 2019-10-18 05:00 AM.

We will keep you informed as things progress.

Original Submission

posted by martyb on Monday September 30, @05:30PM   Printer-friendly
from the WallOfText++ dept.

What started it all:

On 2019-08-24 13:02:01 UTC an accusation ( was made that a Journal Entry "It would have been posted before 6 hours ago" (i.e. posted at approximately 2019-08-24 07:00:00 UTC) was deleted by a member of the staff at SoylentNews. The circumstances surrounding the making of the Journal Entry are elaborated upon in this comment. (

I have been with this site since before it went live. Its founding principal has been the making available of a forum whereby the community can submit stories — and post comments — to predominantly tech-related items. Further, each logged-in user has been made available the ability to post entries to their Journal.

As Editor-in-Chief I took this allegation seriously and performed an independent and in-depth investigation. My findings are presented below.

Note: It is not lost on me the futility of trying to prove a negative. It is for good reason that the criminal justice system in the US is founded on the principle of "innocent until proven guilty." It is not up the the accused to vindicate themselves, but for the accuser to bring sufficient evidence to bring about conviction.

NB: In the course of writing this, I discovered a bug in how the site displays wide elements contained in an ECODE element. It incorrectly wraps the text onto the next line (leading to a jumbled mess) when it should, instead, provide horizontal scroll bars. Please accept my apologies for its current appearance.

Executive Summary:

An in-depth investigation making use of: external resources, the UI presented by SoylentNews, and ad-hoc queries of the site database (DB) failed to locate a "smoking gun", i.e. found no clear proof that a Journal Entry was posted to the site and subsequently deleted by anyone other than an author.

It is my estimation that the user submitted an entry, but the site failed to receive and save it correctly. In other words, the user tripped over some kind of bug be it in the site's code, communications between the user and the site, or something else.

Recommendation: When a user completes making a Journal Entry and submits it to the site, the code should respond by using the newly-created journal parameters in conjunction with the normal journal-loading code to present the Journal Entry to the user as confirmation that the entry was properly received and saved. That is to say, affirmative feedback of receipt, storage, and accessibility of the Journal Entry.


For those who may not be aware, all staff on SoylentNews are volunteers; nobody has ever been paid anything for their work on this site. Everything has been done in people's spare time. Early morning before work. During work breaks. Nights, weekends, holidays, and vacations too.

I hereby express my gratitude to all who have supported my work on this investigation. I have invested at least two person-weeks of effort digging into this. This necessitated my letting up on processing story submissions and making them available on the site. Please join me in thanking janrinok, fnord666, takyon, and chromas who stepped up and carried much of the editing load while I have been busy on this investigation. Thanks are in order, too, for @SemperOSS; lending a most-welcome helping hand in finding locations of — and incantations for extracting pertinent data from — important information about how the site is configured and operates.


Early Site History:

For those who may not have been here at the beginning, SoylentNews arose as a result of frustration with how Slashdot failed to listen to what they called "their audience" in response to their rollout of a Beta version of their site. (And also lead to the infamous "Buck Feta!" war cry.) This, in turn, lead to the infamous "Slashcott" — a week-long period where users voluntarily abstained from visiting Slashdot. (Slashdot boycott). This Slashcott was then extended for an additional week.

While this was happening, some like-minded folk got together and found an open-sourced release of slashcode. This code was a few years out of date and depended on old versions of other applications. Among other things, this included the database server (MySQL), the web server (Apache), web cache (memcached), and Perl in which the site code had been written. In a frantic two-week-long period, these intrepid coders took this ancient software and hacked it into "running condition" and SoylentNews went public on February 16th, 2014.

The first few weeks of SoylentNews were filled with site crashes and reports of display artifacts and problems with the general operation of the site. In short, many bugs resulting from the feverish pace of development revealed themselves and, over time, were found and quashed. It is a testament to their hard work and dedication that problems with the site are, today, viewed as an aberration instead of a regular occurrence. I do not know whether or not the original Slashcode ever underwent a rigorous review. But, given the amount of hacking involved in just getting SoylentNews up-and-running, any stability of the current site is mostly a result of extremely dedicated Whack-A-Mole debugging.


  • Site Crash on 2019-08-15
  • Malicious URL?
  • Other Sources
  • Use the Source, Luke!
  • The Journals-Related Tables
  • The Discussion-Related Tables
  • Or is it?
  • The Comment-Related Tables
  • Trying Another Perspective
  • The Message-Related Tables
  • Conclusion
  • Recommendation

Site Crash on 2019-08-15:

We had a site crash on August 15, 2019. We do not know why. Given the site had already been down for several hours, it was deemed necessary to restore a backup to the drive on which the site database resides. No root cause analysis is practicable at this time. Over the next couple days, we found some artifacts from that unplanned crash and site restart. For example, we discovered that slashd — the daemon which launches periodic processes in support of the site (such as emailing notifications about comment moderations and replies) had not been restarted. Also, it is possible that some race condition arose that interfered with the site properly saving a Journal Entry.

Malicious URL?:

It was suggested that someone could construct a malicious URL to delete another user's journal. The author of a Journal Entry has a button they can click so as to delete one of their own Journal Entries. It was thought that, perhaps, a modification of the resultant URL might suffice to delete a Journal Entry for someone else. I tried this on our development server where I have "god mode" privileges; absolutely nobody on the production SoylentNews site has this privilege level. Even with these elevated privileges, my attempt failed: I saw no error message, but neither was the Journal Entry deleted.

Finding: Even the most-priviledged user cannot use a malicious URL to delete a Journal Entry.

Other Sources:

As part of this investigation, I looked for data that may have been captured externally. I posed queries to Google for the Journal Entry author's nickname and the date of the post, and found nothing. Another place I looked was the Wayback Machine on the Internet Archive. I found these archives for that date:

Notice how the link contains the date and time of when the site was scanned: "YYYY MM DD hh mm ss". The first two links bracket the date/time in question. I could find no indication of a Journal Entry there, either.

Finding: Failed to find an external source which showed that the Journal Entry had been posted to the site.

Use the Source, Luke!

Well, the source of data for the entire site: the database, that is. Having found no external corroboration, there was nothing left but to dive into the database.

It is worth mentioning at this point that the source code for SoylentNews has been made available on GitHub under Note that the original slashcode on which soylentnews was built was saved under and had a last commit in September of 2009. Observant readers will note that leaves a nearly 4.5 year gap from last commit and the use of it as a base for SoylentNews which launched on 2014-02-16 . Many dependent packages had undergone major updates and bug fixes since that time. This required significant modification in the received code just to be able to run at all.

The code which runs the current SoylentNews site can be found under

The Journals-Related Tables:

This was the obvious starting point. There are 4 potential tables of interest:

  • journal_themes
  • journal_transfer
  • journals
  • journals_text

Nothing much of interest in journal_themes:

mysql> DESCRIBE journal_themes ;
| Field | Type                | Null | Key | Default | Extra          |
| id    | tinyint(3) unsigned | NO   | PRI | NULL    | auto_increment |
| name  | varchar(30)         | NO   | UNI | NULL    |                |
2 rows in set (0.00 sec)

mysql> SELECT * FROM journal_themes ORDER BY id ;
| id | name     |
|  1 | generic  |
|  2 | greypage |
|  3 | bluebox  |
3 rows in set (0.00 sec)

There is nothing at all in the journal_transfer table:

mysql> SELECT count(*) FROM journal_transfer ;
| count(*) |
|        0 |
1 row in set (0.00 sec)


The journals table provides an "abstraction" of a Journal Entry. The actual text of a Journal Entry is stored in the journals_text table:

mysql> DESCRIBE journals_text;
| Field     | Type                  | Null | Key | Default | Extra |
| id        | mediumint(8) unsigned | NO   | PRI | NULL    |       |
| article   | mediumtext            | NO   |     | NULL    |       |
| introtext | mediumtext            | NO   |     | NULL    |       |
3 rows in set (0.00 sec)


We will come back to this table shortly. For now, suffice it to say that the id field provides an index into both the journals and journals_text tables, the introtext field contains the HTML for the Journal Entry, and the article field contains a plain-text rendition of the introtext field.

Now, let's look closer at the journals table:

mysql> DESCRIBE journals ;
| Field       | Type                               | Null | Key | Default           | Extra                       |
| id          | mediumint(8) unsigned              | NO   | PRI | NULL              | auto_increment              |
| uid         | mediumint(8) unsigned              | NO   | MUL | NULL              |                             |
| date        | datetime                           | NO   |     | NULL              |                             |
| description | varchar(80)                        | NO   |     | NULL              |                             |
| posttype    | tinyint(4)                         | NO   |     | 2                 |                             |
| discussion  | mediumint(8) unsigned              | YES  |     | NULL              |                             |
| tid         | smallint(5) unsigned               | NO   | MUL | NULL              |                             |
| promotetype | enum('publicize','publish','post') | NO   |     | publish           |                             |
| last_update | timestamp                          | NO   |     | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
| srcid_32    | bigint(20) unsigned                | NO   | MUL | 0                 |                             |
| srcid_24    | bigint(20) unsigned                | NO   | MUL | 0                 |                             |
11 rows in set (0.01 sec)


Let's first look for entries made within a day on either side of 2019-08-24:

mysql> SELECT id, date, uid, description, discussion FROM journals WHERE "2019-08-23" < date AND date < "2019-08-26" ORDER BY id ;
| id   | date                | uid  | description                                                 | discussion |
| 4512 | 2019-08-23 15:02:42 | 4388 | You Are Banned.                                             |      33255 |
| 4513 | 2019-08-23 16:25:06 | 6150 | "Our great American companies are hereby ordered..."        |      33256 |
| 4514 | 2019-08-24 04:38:24 | 4543 | Perdurabo.                                                  |      33265 |
| 4515 | 2019-08-24 08:17:01 | 4543 | By request, The HU.                                         |      33266 |
| 4517 | 2019-08-24 20:09:11 | 5086 | No, Virginia, "conservatism" is not the new counter-culture |      33274 |
| 4518 | 2019-08-25 13:35:30 | 2926 | The 'Soft' Racism of Gun Control                            |      33285 |
| 4520 | 2019-08-25 16:38:48 | 5291 | Valediction: On corruption and moderation                   |      33287 |
| 4521 | 2019-08-25 22:04:51 | 3682 | Plasma Tsunamis Responsible for Sunspots                    |      33289 |
8 rows in set (0.02 sec)


Notice that (id==4520) was posted by this user (uid==5291), but... it is not the droid Journal Entry we were looking for. Looking closer, notice how we are missing (id==4516) and (id=4519). Apparently, deleting a Journal Entry actually deletes it from the database, instead of leaving it in the DB but flagging it as being deleted (i.e. unavailable).

Historical Perspective: Slashdot was registered on 1996-11-17. Disk space usage was a much greater concern back then. For historical perspective consider that Windows 95 had just been released and that the Pentium MMX and Pentium Pro were introduced in about the same time period. As for disk storage, this timeline of hard disk drives notes that in 1996 Seagate shipped its first 10,000 RPM "Cheetah" disk drive and 1997 saw the release of the "IBM Deskstar 16GP 'Titan' - 16,800 megabytes, five 3.5-inch disks". Yes, 16,800 MB or 16.8 GB. Flash drives of that capacity can be purchased for under $10 today.

So, getting back to the missing entries, we can ignore (id==4519) as that was clearly well past the time of interest. That leaves (id==4516)... where did it go?

Let's see what we can learn from looking at the journals_text table.

mysql> SELECT id, substr(introtext, 1, 80) FROM journals_text WHERE 4512 <= id AND id <= 4521 ORDER BY id ;
| id   | substr(introtext, 1, 80)                                                         |
| 4512 | <p>This morning, I took a quick look at the website of the Church of the Flying  |
| 4513 | <p>You probably saw who said that, so I don't have to mention his name.</p><p>Bu |
| 4514 | <tt>I've mentioned before that I perceive hip-hop as more vibrant, more alive, t |
| 4515 | <tt><br><br>Please, anyone, contribut |
| 4517 | Recently one of our frothier, crazier members, who is in my opinion a fascinatin |
| 4518 | <p>The historically racist origins of gun control are hardly a topic for debate. |
| 4520 | <p>This will be my final interaction with this site, which I have supported for  |
| 4521 | <p><b>Rejected submission by RandomFactor at 2019-07-25 21:17:28 from the tSUNam |
8 rows in set (0.01 sec)

Hmmm, same missing entries as before; no joy here.

Attentive readers will have noticed the discussion column in the journals table. Further, for (id==4515) we have (discussion==33266) and for (id==4517) we have (discussion==33274). We will come back to those presently.

Finding: We have failed to find the sought-for Journal Entry, but have found two tuples of id and discussion that warrant further investigation.

The Discussion-Related Tables:

Stepping back for a moment, since comments can be posted to both Journal Entries and stories, it makes sense they may share some commonality. There are two tables related to discussions in our database: discussion_kinds and discussions.

mysql> DESCRIBE discussion_kinds ;
| Field | Type                | Null | Key | Default | Extra          |
| dkid  | tinyint(3) unsigned | NO   | PRI | NULL    | auto_increment |
| name  | varchar(30)         | NO   | UNI |         |                |
2 rows in set (0.00 sec)

mysql> SELECT * FROM discussion_kinds ORDER BY dkid ;
| dkid | name          |
|    1 | story         |
|    2 | user_created  |
|    3 | journal       |
|    4 | journal-story |
|    5 | poll          |
|    6 | submission    |
|    7 | feed          |
|    8 | project       |
8 rows in set (0.00 sec)

Now we are getting somewhere! There are entries for story and for journal, just as we had hoped!

The other table of interest was discussions. Let's see what we can find there.

mysql> DESCRIBE discussions ;
| Field         | Type                                                                                           | Null | Key | Default             | Extra                       |
| id            | mediumint(8) unsigned                                                                          | NO   | PRI | NULL                | auto_increment              |
| dkid          | tinyint(3) unsigned                                                                            | NO   |     | 1                   |                             |
| stoid         | mediumint(8) unsigned                                                                          | NO   | MUL | 0                   |                             |
| sid           | char(16)                                                                                       | NO   | MUL |                     |                             |
| title         | varchar(128)                                                                                   | NO   |     | NULL                |                             |
| url           | varchar(255)                                                                                   | NO   |     | NULL                |                             |
| topic         | int(10) unsigned                                                                               | NO   | MUL | NULL                |                             |
| ts            | datetime                                                                                       | NO   |     | 1970-01-01 00:00:00 |                             |
| type          | enum('open','recycle','archived')                                                              | NO   | MUL | open                |                             |
| uid           | mediumint(8) unsigned                                                                          | NO   |     | NULL                |                             |
| commentcount  | smallint(5) unsigned                                                                           | NO   |     | 0                   |                             |
| flags         | enum('ok','delete','dirty')                                                                    | NO   |     | ok                  |                             |
| primaryskid   | smallint(5) unsigned                                                                           | YES  | MUL | NULL                |                             |
| last_update   | timestamp                                                                                      | NO   |     | CURRENT_TIMESTAMP   | on update CURRENT_TIMESTAMP |
| approved      | tinyint(3) unsigned                                                                            | NO   |     | 0                   |                             |
| commentstatus | enum('disabled','enabled','friends_only','friends_fof_only','no_foe','no_foe_eof','logged_in') | NO   |     | enabled             |                             |
| archivable    | enum('no','yes')                                                                               | NO   |     | yes                 |                             |
| legacy        | enum('no','yes')                                                                               | NO   |     | yes                 |                             |
18 rows in set (0.01 sec)

So far, so good. Let's take a look inside.

mysql> SELECT id, dkid, stoid, sid, substr(title,1,20), url, ts, uid, last_update FROM discussions WHERE "2019-08-23 00:00:01" < ts AND ts < "2019-08-24 23:59:59" ORDER BY id ;
| id    | dkid | stoid  | sid              | substr(title,1,20)   | url                                               | ts                  | uid  | last_update         |
| 33258 |    1 | 370766 | 19/08/23/2359248 | Billionaire David Ko | // | 2019-08-24 00:28:00 |   76 | 2019-08-29 01:41:03 |
| 33259 |    1 | 370767 | 19/08/24/0012203 | Study Corroborates t | // | 2019-08-24 02:49:00 |   76 | 2019-08-27 04:52:14 |
| 33260 |    1 | 370768 | 19/08/24/0021202 | Alleged "Snake Oil"  | // | 2019-08-24 05:12:00 |   76 | 2019-08-25 06:48:09 |
| 33261 |    1 | 370769 | 19/08/24/0027221 | Federal Agency Turf  | // | 2019-08-24 07:32:00 |   76 | 2019-08-25 23:40:34 |
| 33262 |    1 | 370770 | 19/08/24/0038241 | Phone Companies and  | // | 2019-08-24 09:54:00 |   76 | 2019-08-27 02:50:36 |
| 33263 |    1 | 370771 | 19/08/24/0133226 | EU Officials Draft P | // | 2019-08-24 12:19:00 |   76 | 2019-08-25 11:48:20 |
| 33264 |    1 | 370772 | 19/08/24/0135254 | YouTube Banning Comb | // | 2019-08-24 14:37:00 |   76 | 2019-08-25 04:10:23 |
| 33265 |    3 |      0 |                  | Perdurabo.           | //              | 2019-08-24 04:38:24 | 4543 | 2019-08-27 08:28:48 |
| 33266 |    3 |      0 |                  | By request, The HU.  | //              | 2019-08-24 08:17:01 | 4543 | 2019-08-27 15:02:04 |
| 33267 |    1 | 370773 | 19/08/24/128219  | NASA Astronaut's Div | //  | 2019-08-24 16:55:00 |   76 | 2019-08-29 18:13:08 |
| 33268 |    1 | 370774 | 19/08/24/1216253 | Neil Young on Sound  | // | 2019-08-24 19:17:00 |   76 | 2019-08-26 18:42:21 |
| 33269 |    3 |      0 |                  | Test                 | // | 2019-08-24 12:55:02 | 5291 | 2019-08-24 17:17:12 |
| 33270 |    1 | 370775 | 19/08/24/1341250 | Deadly Superbug Outb | // | 2019-08-24 21:34:00 |   52 | 2019-08-26 04:19:22 |
| 33271 |    1 | 370776 | 19/08/24/1347258 | Capsule Containing R | // | 2019-08-24 23:53:00 |   52 | 2019-08-26 18:27:55 |
| 33274 |    3 |      0 |                  | No, Virginia, "conse | //      | 2019-08-24 20:09:11 | 5086 | 2019-09-09 06:29:43 |
15 rows in set (0.62 sec)

And how about that! There — at (id==33269) — is our missing Journal Entry number 4516.

Or is it?

Let's make no assumptions. Let's try to load it in a browser

Sorry, the requested journal entries were not found.

Now that is some progress. Really! We found something that was posted to a journal, but was deleted. The very one that the user stated they had created after noticing the apparent disappearance of his journal article.

So, umm, where is the other Journal Entry? You know, the missing one? Yeah, we're still working on that one, but at least our theory of what happens in the site code for an intentional deletion of a Journal Entry is validated.

Looking back, remember the discussion column from the journals table and those we were going to come back to. Now is the time; here is what we had:

for (id==4515) we have (discussion==33266) and for (id==4517) we have (discussion==33274)

Here, in the discussions table, we see that discussion 33266 corresponds to and discussion 33274 corresponds to so we have figured out where they went. Unfortunately, this does not provide any closure on the situation, but it does close off one potential avenue of explanation.

Finding: We have found evidence that when a user deletes a Journal Entry, any comments that had been posted to it persist in the database. None have yet been found.

The Comment-Related Tables:

As was the case for the journals table, SoylentNews' comments are split up as well. Here are all of the comment-related tables:

  • comment_log
  • comment_promote_log
  • comment_text
  • commentmodes
  • comments

I do not know why they still exist, but both comment_log and comment_promote_log are empty. As for commentmodes, that is the source for the pull-down list of how one chooses to see comments displayed:

mysql> SELECT * FROM commentmodes ORDER BY mode ;
| mode      | name         | description |
| flat      | Flat         |             |
| nocomment | No Comments  |             |
| threadtng | Threaded-TNG | NULL        |
| threadtos | Threaded-TOS | NULL        |
4 rows in set (0.00 sec)

So now we are down to the comments and comment_text tables. Last things first, the comment_text table is exactly what it says on the tin, where the text of each comment is stored:

mysql> DESCRIBE comment_text ;
| Field   | Type             | Null | Key | Default | Extra |
| cid     | int(10) unsigned | NO   | PRI | NULL    |       |
| comment | mediumtext       | NO   |     | NULL    |       |
2 rows in set (0.01 sec)

The key thing here is the cid (Comment ID) field which is a pointer (key) into the comments table. Having gotten all that out of the way, what can the actual comments table tell us?

mysql> DESCRIBE comments ;
| Field            | Type                  | Null | Key | Default             | Extra                       |
| sid              | mediumint(8) unsigned | NO   | MUL | NULL                |                             |
| cid              | int(10) unsigned      | NO   | PRI | NULL                | auto_increment              |
| pid              | int(10) unsigned      | NO   | MUL | 0                   |                             |
| opid             | int(10) unsigned      | NO   | MUL | 0                   |                             |
| children         | int(10) unsigned      | NO   |     | 0                   |                             |
| date             | datetime              | NO   | MUL | 1970-01-01 00:00:00 |                             |
| last_update      | timestamp             | NO   |     | CURRENT_TIMESTAMP   | on update CURRENT_TIMESTAMP |
| ipid             | char(32)              | NO   | MUL |                     |                             |
| subnetid         | char(32)              | NO   | MUL |                     |                             |
| subject          | varchar(50)           | NO   |     | NULL                |                             |
| subject_orig     | enum('no','yes')      | NO   |     | yes                 |                             |
| uid              | mediumint(8) unsigned | NO   | MUL | NULL                |                             |
| points           | tinyint(4)            | NO   |     | 0                   |                             |
| pointsorig       | tinyint(4)            | NO   |     | 0                   |                             |
| pointsmax        | tinyint(4)            | NO   |     | 0                   |                             |
| lastmod          | mediumint(8) unsigned | NO   |     | 0                   |                             |
| reason           | tinyint(3) unsigned   | NO   |     | 0                   |                             |
| signature        | char(32)              | NO   |     |                     |                             |
| karma_bonus      | enum('yes','no')      | NO   |     | no                  |                             |
| subscriber_bonus | enum('no','yes')      | NO   |     | no                  |                             |
| len              | smallint(5) unsigned  | NO   |     | 0                   |                             |
| karma            | smallint(6)           | NO   |     | 0                   |                             |
| karma_abs        | smallint(5) unsigned  | NO   |     | 0                   |                             |
| tweak_orig       | tinyint(4)            | NO   |     | 0                   |                             |
| tweak            | tinyint(4)            | NO   |     | 0                   |                             |
| badge_id         | tinyint(4)            | NO   |     | 0                   |                             |
26 rows in set (0.00 sec)

Zoinks! That's a lot, but we can ignore much of it for our purposes. We just want to know if any comments were made to the missing Journal Entry. Let's look at a subset:

  mysql> SELECT cid, sid, pid, opid, uid, children, date, last_update, subject FROM comments WHERE pid = 0 AND "2019-08-24 06:00:00" <= date AND date <= "2019-08-24 08:00:00" ORDER BY cid;
| cid    | sid   | pid    | opid   | uid  | children | date                | last_update         | subject                                          |
| 884595 | 33250 | 884485 | 884479 |    1 |        0 | 2019-08-24 06:00:50 | 2019-09-23 07:02:01 | Re:so?                                           |
| 884596 | 33244 | 884418 | 883941 |    1 |       19 | 2019-08-24 06:01:46 | 2019-09-23 07:02:01 | Re:Sign of the times                             |
| 884597 | 33265 |      0 |      0 |   18 |       15 | 2019-08-24 06:03:10 | 2019-09-23 07:02:01 | Wait, wut?                                       |
| 884598 | 33259 | 884585 | 884531 |    1 |        1 | 2019-08-24 06:08:34 | 2019-09-23 07:02:01 | Re:Wow! Then radiation hits fetus...             |
| 884599 | 33244 | 884520 | 883955 |    1 |        2 | 2019-08-24 06:10:36 | 2019-09-23 07:02:01 | Re:It's exaeta again, isn't it?                  |
| 884600 | 33244 | 884596 | 883941 |   18 |       18 | 2019-08-24 06:11:13 | 2019-09-23 07:02:01 | Re:Sign of the times                             |
| 884601 | 33252 | 884400 | 884336 | 2828 |        0 | 2019-08-24 06:11:29 | 2019-09-23 07:02:01 | Re:What's the appeal of mercedes?                |
| 884602 | 33265 | 884581 | 884581 | 2489 |        2 | 2019-08-24 06:18:09 | 2019-09-23 07:02:01 | Re:Is a rapper a singer?                         |
| 884603 | 33259 | 884598 | 884531 |    1 |        0 | 2019-08-24 06:27:33 | 2019-09-23 07:02:01 | Re:Wow! Then radiation hits fetus...             |
| 884604 | 33259 | 884530 | 884530 | 6836 |        6 | 2019-08-24 06:30:48 | 2019-09-23 07:02:01 | Re:If so, then...                                |
| 884605 | 33265 | 884597 | 884597 | 4543 |        9 | 2019-08-24 06:31:30 | 2019-09-23 07:02:01 | Re:Wait, wut?                                    |
| 884606 | 33265 | 884597 | 884597 | 2489 |        4 | 2019-08-24 06:41:07 | 2019-09-23 07:02:01 | Re:Wait, wut?                                    |
| 884607 | 33265 | 884605 | 884597 | 2489 |        8 | 2019-08-24 06:44:10 | 2019-09-23 07:02:01 | Re:Wait, wut?                                    |
| 884608 | 33244 | 884457 | 883942 |   52 |        0 | 2019-08-24 06:47:11 | 2019-09-23 07:02:01 | Re:WTF?                                          |
| 884609 | 33259 | 884604 | 884530 |  881 |        5 | 2019-08-24 06:48:32 | 2019-09-23 07:02:01 | Re:If so, then...                                |
| 884610 | 33265 | 884607 | 884597 | 4543 |        7 | 2019-08-24 06:55:40 | 2019-09-23 07:02:01 | Re:Wait, wut?                                    |
| 884611 | 33244 | 884461 | 883966 |   52 |        1 | 2019-08-24 06:58:54 | 2019-09-23 07:02:01 | Re:Why do we moderate? - Because it is important |
| 884612 | 33259 | 884585 | 884531 |    1 |        0 | 2019-08-24 07:00:14 | 2019-09-23 07:02:01 | Re:Wow! Then radiation hits fetus...             |
| 884613 | 33255 |      0 |      0 | 3272 |        1 | 2019-08-24 07:07:03 | 2019-09-24 07:02:02 | Works on reload.                                 |
| 884614 | 33244 | 884435 | 883966 |   52 |        3 | 2019-08-24 07:08:08 | 2019-09-24 07:02:02 | Re:Why do we moderate? - Because it is important |
| 884615 | 33222 | 883944 | 883944 |    1 |        0 | 2019-08-24 07:39:07 | 2019-09-24 07:02:02 | Re:Come out, exaeta, come out!                   |
| 884616 | 33244 | 884337 | 883966 |   52 |        2 | 2019-08-24 07:39:07 | 2019-09-24 07:02:02 | Re:Why do we moderate? - Because it is important |
| 884617 | 33261 |      0 |      0 | 2452 |       16 | 2019-08-24 07:40:06 | 2019-09-24 07:02:02 | 5G is stupid                                     |
| 884618 | 33265 | 884602 | 884581 | 4543 |        1 | 2019-08-24 07:42:09 | 2019-09-24 07:02:02 | Re:Is a rapper a singer?                         |
| 884619 | 33222 | 884211 | 883718 | 2645 |        0 | 2019-08-24 07:49:42 | 2019-09-24 07:02:02 | Re:Need specifics, actually.                     |
| 884620 | 33244 | 884534 | 883966 |   52 |       13 | 2019-08-24 07:54:11 | 2019-09-24 07:02:02 | Re:Why do we moderate? - Because it is important |
26 rows in set (0.00 sec)


The pid field is the comment's "Parent ID", i.e. the comment ID of the comment that is the parent of this comment. Let's look at only those comments which have NO parent id:

mysql> SELECT cid, sid, pid, opid, uid, children, date, last_update, subject FROM comments WHERE pid = 0 AND "2019-08-24 06:00:00" <= date AND date <= "2019-08-24 08:00:00" ORDER BY cid;
| cid    | sid   | pid | opid | uid  | children | date                | last_update         | subject          |
| 884597 | 33265 |   0 |    0 |   18 |       15 | 2019-08-24 06:03:10 | 2019-09-23 07:02:01 | Wait, wut?       |
| 884613 | 33255 |   0 |    0 | 3272 |        1 | 2019-08-24 07:07:03 | 2019-09-24 07:02:02 | Works on reload. |
| 884617 | 33261 |   0 |    0 | 2452 |       16 | 2019-08-24 07:40:06 | 2019-09-24 07:02:02 | 5G is stupid     |
3 rows in set (0.78 sec)


All of these comments pertain to the story they are attached to:

Finding: we found no evidence of a comment having been made to a Journal Entry, during the time in question.

Trying Another Perspective:

In this comment, it was suggested:

All your fans would have received a message about your journal entry instantly. I for one didn't. You can't invisibly delete a journal entry without:
a) resetting the journal auto-incrementing counter in the database (and there's no programattic way to do that, you can only move it forwards)
but also:
b) invisibly deleting *all* the messages that were sent out because of it
and therefore:
c) resetting the message auto-incrementing counter in the database (ditto)
and here's the killer:
d) on a live system
and the cherry on top:
e) with no logs

If we had someone with that level of technical competence on the administrative team, that would probably double the total amount of competence have - the above goes well beyond mere guru level.

Ockham's razor points to either human error or a bug.

We've previously shown that articles can be deleted; there is a button for it on the journal writer's edit page.

If the deleted Journal Entry was the one with the largest ID, the system just re-uses that slot.

But, the idea that a message might have been sent out announcing to a user that a friend of theirs posted a Journal Entry? that's interesting. Let's see where that leads us.

Finding: We have a new situation to investigate, but no results, yet.

The Message-Related Tables:

Here are all the message-related tables on the site:

  • message_codes
  • message_deliverymodes
  • message_drop
  • message_log
  • message_web
  • message_web_text

At this point, selections made by particular users will be discovered. In the interests of excluding 3rd-parties, please forgive me if some of this discussion is less detailed than previously.

Let's take these in order:

mysql> DESCRIBE message_codes   ;
| Field           | Type                             | Null | Key | Default | Extra |
| code            | int(11)                          | NO   | PRI | NULL    |       |
| type            | varchar(32)                      | NO   |     | NULL    |       |
| seclev          | int(11)                          | NO   |     | 1       |       |
| modes           | varchar(32)                      | NO   |     |         |       |
| send            | enum('now','defer','collective') | NO   |     | now     |       |
| subscribe       | tinyint(1)                       | NO   |     | 0       |       |
| acl             | varchar(32)                      | NO   |     |         |       |
| delivery_bvalue | int(10) unsigned                 | NO   |     | 0       |       |
8 rows in set (0.00 sec)

mysql> SELECT * FROM message_codes WHERE seclev = 1 ORDER BY code ;
| code | type                       | seclev  | modes | send       | subscribe | acl   | delivery_bvalue |
|   -2 | Registration Mail          |       1 | 0     | now        |         0 |       |               0 |
|   -1 | Unknown Message            |       1 | 0     | now        |         0 |       |               0 |
|    0 | Daily Newsletter           |       1 | 0     | now        |         0 |       |               1 |
|    1 | Daily Headlines            |       1 | 0     | now        |         0 |       |               1 |
|    2 | Metamoderation Results     |       1 |       | collective |         0 |       |               3 |
|    3 | Comment Moderation         |       1 |       | collective |         0 |       |               3 |
|    4 | Comment Reply              |       1 |       | now        |         0 |       |               3 |
|    5 | Journal Entry by Friend    |       1 |       | now        |         0 |       |               3 |
|    6 | New Submission             |     100 | 0     | now        |         0 |       |               1 |
|    7 | Journal Reply              |       1 |       | now        |         0 |       |               3 |
|    8 | New Comment                | 1000000 |       | now        |         0 |       |               0 |
|   10 | Daily Site Stats           |     100 | 0     | now        |         0 | stats |               1 |
|   11 | Email Story                |       1 | 0     | now        |         0 |       |               1 |
|   12 | Relationship Change        |       1 |       | collective |         0 |       |               3 |
|   13 | Bad login attempt warnings |       1 | 1     | now        |         0 |       |               2 |
|   14 | Daily Moderation Stats     |     100 | 0     | now        |         0 | stats |               1 |
|   15 | Subscription Running Low   |       1 |       | now        |         1 |       |               1 |
|   16 | Subscription Expired       |       1 |       | now        |         1 |       |               1 |
|   18 | Invalid HTML Input         |     100 |       | now        |         0 |       |               3 |
|   19 | Declined Submission Reason |       1 |       | now        |         0 |       |               3 |
|   20 | Admin to user message      |       1 |       | now        |         0 |       |               3 |
|   22 | Achievement                |       1 |       | now        |         0 |       |               3 |
|   25 | Remarks                    |     100 |       | now        |         0 |       |               2 |
23 rows in set (0.00 sec)


Now, this looks promising; (code==5) is "Journal Entry by Friend". Let's hold on to that and see what else we can find.

mysql> DESCRIBE message_deliverymodes ;
| Field    | Type                  | Null | Key | Default | Extra |
| code     | smallint(6)           | NO   | PRI | 0       |       |
| name     | varchar(32)           | NO   |     |         |       |
| bitvalue | mediumint(8) unsigned | NO   |     | 0       |       |
3 rows in set (0.00 sec)

mysql> SELECT * FROM message_deliverymodes ORDER BY code ;
| code | name        | bitvalue |
|   -1 | No Messages |        0 |
|    0 | E-mail      |        1 |
|    1 | Web         |        2 |
3 rows in set (0.00 sec)


A registered user (someone who has a nickname and UID on the system) can specify several things for which they wish to receive notifications, and how they want to receive them. This table is where those are defined. On to the next table.

mysql> DESCRIBE message_drop ;
| Field   | Type                             | Null | Key | Default           | Extra                       |
| id      | int(11)                          | NO   | PRI | NULL              | auto_increment              |
| user    | mediumint(8) unsigned            | NO   |     | NULL              |                             |
| fuser   | mediumint(8) unsigned            | NO   |     | NULL              |                             |
| code    | int(11)                          | NO   |     | -1                |                             |
| date    | timestamp                        | NO   |     | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
| altto   | varchar(50)                      | NO   |     |                   |                             |
| send    | enum('now','defer','collective') | NO   |     | now               |                             |
| message | mediumblob                       | NO   |     | NULL              |                             |
8 rows in set (0.00 sec)

mysql> SELECT count(*) FROM message_drop ;
| count(*) |
|      115 |
1 row in set (0.00 sec)


This table contains a listing of the messages that have been deleted today. Logged-in users can go to their messages inbox, and if they have any pending, web-based messages, will see this message at the top of the screen:

These messages will be kept in the system for only 14 days, whether they have been read or not. After 14 days, they will be deleted.

Messages marked with "*" are unread.

Unfortunately, I did not learn of this table until several days too late.

There's nothing to do at this point but to continue on. Besides, the next table is much more interesting.

mysql> DESCRIBE message_log ;
| Field | Type                  | Null | Key | Default           | Extra                       |
| id    | int(11)               | NO   |     | NULL              |                             |
| user  | mediumint(8) unsigned | NO   |     | NULL              |                             |
| fuser | mediumint(8) unsigned | NO   |     | NULL              |                             |
| code  | int(11)               | NO   |     | -1                |                             |
| mode  | int(11)               | NO   |     | NULL              |                             |
| date  | timestamp             | NO   |     | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
6 rows in set (0.00 sec)


One thing we can find out here is how often messages about friend's Journal Entries are posted to the site.

mysql> SELECT count(1) AS count, substr(date,15,2) AS mins FROM message_log WHERE code = 5 GROUP BY mins ORDER BY mins;
| count | mins |
|    39 | 05   |
|   148 | 10   |
|    77 | 15   |
|    12 | 20   |
|    22 | 25   |
|    60 | 30   |
|    46 | 35   |
|    13 | 40   |
|    25 | 45   |
|    21 | 50   |
|    62 | 55   |
11 rows in set (0.01 sec)


So, every five minutes, if someone has posted a Journal Entry, the site sends out notifications to all those who had tagged the writer as a friend.

As mentioned earlier, so as to avoid unnecessarily involving innocent third parties, a quick look through the tables containing relationships (friends) and notification requests reveals:

  • there are 14 people who flagged the journal-entry-writer as a friend,
  • of these, 12 have requested notification of when they have posted a Journal Entry
  • all 12 of these have requested notification by 'the web' instead of by 'e-mail'.

A search through the message_log table for any messages sent to any of these people for journal posts by the accuser came up empty.

Finding: No indication has been found that a message was sent to a user to notify them of a friend having posted an entry to their journal.


Multiple sources were investigated to try and find a Journal Entry that was posted to the site and subsequently deleted. External sources were examined and no copy of the alleged Journal Entry were found. A user who has submited a Journal Entry can subsequently delete it, but any comments made to that Journal Entry would persist after such a deletion. No such comments were found. Further, twelve users flagged this user as a friend and requested notification of their having posted a Journal Entry. No such notification was found to have been generated.

A staff member with sufficient privileges and knowledge could, potentially, have deleted a Journal Entry directly through the database. The window of opportunity is at most five minutes, and possibly as short as a few seconds. Such action would have to be performed on a live system and leave no trace of it having been done. This would, also, require the staff memeber to have noticed the posting very shortly after it was saved. Though not impossible, the likelihood of this being the case is vanishingly small.

The user could have written the Journal Entry and, somehow, forgotten to save it. Further discussion with the user revealed their having taken extreme care in updating and saving their Journal Entry.

Given these, and the fact that yet another bug in the code was found while writing this report, combined with the frantic pace of the site's initial development and the many bugs found subsequent to its initial deployment, it appears that the user encountered a bug in the site's handling of posting a new Journal Entry.

Recommendation: It is suggested that the site be updated so that upon successful receipt of a Journal Entry, said entry be immediately displayed to the user by employing the same means that the site uses to display any Journal Entry.

posted by martyb on Tuesday September 03, @06:00PM   Printer-friendly
from the still-looking dept.

An accusation has been made that a Journal Entry made by a member of the community was deleted.

tl;dr No "smoking gun" has been found so far and no conclusion has been made one way or another. The investigation will continue. I have the next two days (Wed. and Thurs.) off from work and intend to dig further into this, then.

When I first learned of this, I immediately started looking around as I found the accusation to be credible and, if true, unacceptable behavior.

Nothing I found that morning could corroborate the claim. I posted my findings and promised that when I got home from work that I would look into it. And I did exactly that spending several hours looking for data through both the UI tools presented within the site and with ad hoc queries to our DB. I dumped sections of tables of interest and brought them local to my home machine for later investigation.

Of course, as life has a way of doing, I was also in the midst of a crunch period at work and my spare time has been scarce of late (e.g. I had a training class to attend that was over 100 miles away and worked the entire Labor Day weekend). Further, we have two editors who are on a leave of absence and another who is on vacation, leaving a greater burden on the remaining editors to keep stories coming to the community. This has been exacerbated by the Labor Day holiday weekend in the US meaning fewer substantial stories get posted to the web at this time. In a nutshell that left fewer editors having to work even harder to find stories that could be brought to the community.

Do remember, also, that we had a rare site outage a couple weeks ago which necessitated restoring the site DB from backups. This was a manual process. There have been a few hiccups since then that were attributed to an unclean shutdown/restart as well as needing to restart various services besides just the database: the web server, load balancer caches (nginx), slashd (a cron-like daemon which periodically starts still other background processes and services), and ... you get the idea.

Still, in what little free time I had, I continued to look for information that could corroborate or refute the accusation.

It bears mentioning that there may well be no "smoking gun". But whether there is or is not, I firmly believe the community nonetheless deserves an investigation. It is a very different thing to toss one's hands up in the air and not even look as opposed to look deeply and diligently and find nothing.

I have decades' experience testing computer software on a wide range of platforms. I have worked at startups and fortune-100 blue chip companies. I once was brought in to a company to test a compiler when my only prior experience was a single college course. Another place I worked at had an automated test harness written in a language in which my only prior experience was it being one of 4 languages covered in a language-survey course in college. Within a week I had started refactoring the code base and by the time I left 9 months later, had removed and/or isolated hardware-based dependencies and rewritten the test harness to run in parallel. A full test run across several different hardware platforms and OS versions, which used to take 8-9 hours, now completed in just 1 hour. These are meant only to illustrate that I have no difficulties digging into things in which I had limited or no prior experience. When SoylentNews implemented Unicode character support using UTF-8 (Unicode Transfer Format - 8bit), I searched for the relevant RFCs (Request For Comment, aka Internet "standards"). One led to another and that to others and when I was done had found about a half-dozen of those. Further, there were different versions of the Unicode standard published by the Unicode Consortium, so I dug into those, as well. Needing test data, I wrote tools to extract/generate all the NCEs (Named Character Entities) as well as both hex- and decimal- numeric character entities. Oh, and then there was the matter of IDNs (Internationalized Domain Names; like "" but "foo" could be in any Unicode supported language!) These then were used in all user-facing, text-entry fields that I could discover and the results were reviewed. I am pleased to say that in the subsequent 5 years' time, I am aware of only two bugs that got through.

(Anecdote time. During the Slashcott, I got wind of some "alternate Slashdot" being developed, but could find no announcement of where it was. I did a bunch of searches. They came up empty. Tried a bunch more. No joy. Then I found what looked to be a bug report for this new site with a screen capture of the main page. And, it had this really strange URL in the address bar. Some domain name like "". Strange! But, I tried it and found it was just what I was looking for. Not knowing who was behind this or how to contact them, I was a bit nervous. Still, after enduring several formkey errors that day and into the next morning, I was finally able to create new account. I then kept a low profile until SoylentNews went live. And that, my friends, is how I got a coveted two-digit UID.)

I present the forgoing not to brag, but to give examples of the diligence and thoroughness that I am bringing to this investigation.

Further, I have somehow become the spokesperson for site-related "stuff" be it an annual summary of where we are at, announcements of server outages due to our hosting provider, Linode, providing updated hardware, different virtual hosting software, bug patches for Meltdown and Spectre, etc. When I have made mistakes I have owned them openly and on the front page no less. I cannot persuade you to believe me, and feel free to believe as you wish, but I can only trust that my integrity in all my actions on this site speak more forcefully and firmly than any words I may bring you in support of it.

My investigation is not complete, but given the amount of time that has passed, I felt it important to give an interim summary of what was planned and any results found so far.

Original Submission

posted by martyb on Friday August 23, @06:17AM   Printer-friendly
from the tragedy-of-the-commons dept.

I was going to post this to a particular story, but thought this might generate more attention and discussion as a general submission.

Seriously, what is going on with all these troll mods? Just because you disagree with someone, thus earning a "disagree" mod, does not mean that person is a "troll." To steal a definition from Urban Dictionary:

An Internet troll, or simply troll in Internet slang, is someone who posts controversial, inflammatory, irrelevant or off-topic messages in an online community, such as an online discussion forum or chat room, with the primary intent of provoking other users into an emotional response or to generally disrupt normal on-topic discussion.

Just because you disagree with someone, does not mean they are trying to do the above. Be faster on the "disagree" and slower on the "troll." Under such abuse, it is hard to have a good discussion and, in itself, is trollish behavior by "generally disrupt[ing] normal on-topic discussion." Other than people disciplining themselves, a concerted effort to police such abuses, or making moderation logs public on the bottom of a comment where the score is shown now, I'm unsure of what to do about. As it stands, it is getting increasingly ridiculous to read what discussion is here on any topic remotely controversial, and is expanding outside of even those. It is starting to drive me away from the site, and I'm somewhat confident it is doing the same for others. I'd be interested to see what others think about the depth of the problem, if they even believe it even exists at all, and what solutions you all have for it.

[Ed note. This story is published exactly as received. First off, it bears repeating that complaining about moderations in the comments often leads (rightly) to an off-topic moderation. That is a contributing factor to my decision to run this story. Secondly, moderation is something that I on occasion have found I've fat-fingered and given a different moderation than expected. Thirdly, in the grand scheme of things, a comment's moderation is — relatively speaking — small potatoes. It is NOT a measure of your IQ or value as a human being. or standing in the community. Just accept that stuff happens and that as likely as not, someone will be along to moderate it the other way. Which is a good opportunity to say: PLEASE USE YOUR MOD POINTS! Lastly, if you think a comment was moderated in error, then send the CID (Comment ID) link e.g. "(#876543)" in an email to admin (at) soylentnews (dot) org. Keep in mind however that we are all volunteers here and there most likely will be a delay between when you send out an email and when we can get around to it. --martyb]

[Updated: 20190823_111312 UTC See comment from JR who far more precisely and eloquently expressed the idea I was attempting to. I concur with his assessment. If I want people to upmod a comment of mine that I believe was unfairly downmodded, then I need to be willing to upmod other's mis-modded comments. For perspective, so far this month, anywhere from ~150-~350 mod points were used in any given day. It bears repeating: use your mod points!]

Original Submission

posted by martyb on Thursday August 22, @12:00PM   Printer-friendly
from the say-that-five-times-fast dept.

I just finished updating the certs for SoylentNews.

We get our certs through Let's Encrypt. Yes, we could automate the whole process, but it has been discussed and decided that given our... unique configuration, it is best to have a human in the loop than to let a script somehow run amok and then try to restore things when who-all-knows-what got deployed and things have gone sideways.

I have checked our web sites for production, dev, and staff as well as sending and retrieving e-mail; all seemed to be okay.

More than anything else, this is a check on us to see if we (well, me, actually) overlooked anything. If you do detect any issues, please post a comment to this story.

(Hat tip to The Mighty Buzzard for standing by in case I bollixed up something.)

[Update: Unless, of course, you cannot post a comment to this story! Then pop onto the #Soylent channel on our Internet Relay Chat (IRC) server and let us know over there. --martyb]

Original Submission

posted by janrinok on Thursday August 15, @02:00PM   Printer-friendly
from the stuff-happens dept.

Earlier this morning (around 0520 UTC) we experienced a sequence of faults which we do not, yet, fully understand. Unfortunately this coincided with several key admins being asleep as they should be.

Several of the team started to try to recover the site but all the obvious efforts failed to have the desired results (e.g. restarting the failed software and switching it all off and back on again). Chromas took the lead in this but he was unable to resolve the problems. SemperOSS showed up next and made a mighty effort but things failed to cooperate. Martyb was next in and checked on status of our servers and a few other things but ultimately could not provide much assistance.

TheMightyBuzzard turned up at his usual time and he and SemperOSS set to work trying to recover the site. After a long slog it became apparent that the existing software had suffered some form of corruption and that a simple recovery was not going to be possible. The decision was finally made to restore to yesterday's snapshot, but this comes with a downside. All the stories, comments, and journal entries since that snapshot was taken (approximately 2019-08-14 22:02:36 UTC) have been lost. This is regrettable but under the circumstances unavoidable.

Some of you joined us on IRC and provided real or moral support to those carrying out the recovery task. Others sent us emails informing us of the outage and we also received offers of help. For this we thank you. We also thank our community at large for your patience while we carried out the work that needed to be done. The investigation into what happened has still to be done and will take some time to do. The site is now up again, but as we are not 100% sure what caused the problem we cannot guarantee that the site will function properly and we will have to wait a day or two to be confident that we have resolved the issue.

We tried to keep everyone updated on our progress on our IRC channel but, if you could not get there, then there was little else we could do to update you on our progress. Displaying a short 'Site Down' message actually exacerbated the problems so we decided that it was better to leave the database alone until the problems had been resolved.

Finally, I would like to say thank you to the guys who did all the work under difficult circumstances. Their efforts are appreciated. We used the usual piece of software on IRC to find out who should bear the blame for this debacle (~blame) — it responded with the only name it has in its choice of staff. Bytram!

posted by martyb on Friday July 05, @03:30PM   Printer-friendly
from the Buck-Feta! dept.

On July 5th of 2014, SoylentNews was officially given approvals for its letters of incorporation as a Public Benefit Corporation.

This is a good opportunity to take stock of where we are. Read on past the fold for coverage of:

  • Fundraising
  • Site activity
  • Folding@Home
  • Journals of Spam Accounts
  • Submit Stories
  • Thank YOU

Fundraising: For the first half of 2019, the community came through with room to spare! Not only did we attain our goal of $2000, but we also received approximately $549.38 towards our stretch goal of $1000 more. I'll leave the current totals in the Site News slashbox for a few more days before resetting for fundraising for the second half of the year. Please feel free to Subscribe as any new amounts received will go toward our next funding period. In case you were wondering, we had about 100 subscriptions during this period with the largest amounts being $200(x2), $120, $100(x3), and $80; all the rest were for $50 or less. Though this is an all-volunteer organization, we do have to pay for servers, domain registrations, and a CPA to file our taxes, so these subscriptions are key to our being able to "keep the lights on". THANK-YOU!

Site activity: A quick look around the site reveals some interesting statistics. As of my writing this, our story count stands at 27,861 stories. To that, add the 4,178 journal entries that have been posted by the community. We currently have 858,117 comments posted. We have had, however, comparatively few comment moderations: 586,164. We have 8,225 registered users, too!

In case you did not see the follow-up story, we have not performed moderation bans for excess up/down moderations in well over a year. Any moderations by user1 of comments by user2 that exceed 4 per day are reviewed and are simply reversed if it is determined than a mod bomb or sockpuppet moderation is happening. This is in lieu of our implementing code that will automatically handle this in the future when our dev team can scrounge up some spare time to implement it and merge it in. I had my information incorrect in a story posted a while back and want to set the record straight. Remember, there are a few thousand other users on this site, it is not necessary for you to single-handedly moderate all the comments that need it; if it really is that bad or good, then surely someone else will share your perspective and be along before too long to do the same. And, if not, it is not the end of the world; this is a place to discuss stories; not to set the entire world on the right path. In other words: Login to the site and Use your mod points!

Folding@Home: This has not been mentioned in a while, but we do have a SoylentNews Folding@Home (F@H) team... currently ranked #231 in the world! (Based on current trends, we should be #230 within a couple days.) This is purely voluntary by members of the community who saw an opportunity, banded together, and just got things started. Hat tip to SirFinkus for the original impetus and early organizing that has helped us get to where we are today. The F@H client automatically takes advantage of idle time on your CPU/GPU. Once installed and set up, it requests a work unit, processes it, and return the results. These calculations help support research into fighting maladies such as Parkinson's disease.

Journals of Spam Accounts: We have observed accounts being created which have posted no comments, made no story submissions, performed no moderations, but have made postings of a commercial nature to their journal. These are currently blocked from appearing in the "Most Recent Journal Entries" slashbox that appears on the main page until the user has accumulated at least 10 Karma. This reduces their visibility to the casual reader of the site, but could be leveraged to help boost the search engine rankings of the affiliated enterprise.

Some numbers: Out of 4178 journal entries posted to SoylentNews since the beginning, 528 were posted by users who have less than 10 Karma. Of those 528 journal entries, only 26 have a Karma greater than zero. Further, the most recent journal posting from those 26 was in March of 2018 — well over a year ago.

My take on this is that the good name of SoylentNews is being used to advance the standing of commercial enterprises. We strive to be impartial in our coverage and have never taken any remuneration from any outside enterprise lest it give even the appearance of favoritism to what we publish on this site.

Rather than make an edict from on high, I prefer to hear what the community thinks. What, if anything, have I forgotten to look at? Is this even a problem? If it is a problem, what, if anything should be done about it? Require a minimum Karma of 10 to even post in a Journal?

Submit Stories: It is summer time in the northern hemisphere and most research facilities are running with reduced staffing so folks can take advantage of the weather. Also, most schools are on summer vacation, as well. The result is that much less research is performed now than would be during the normal school year. This period is sometimes referred to as the "Silly Season" where on-line publications (many of whom are also running with fewer staff) tend to run more fluff pieces. Please bear with us as we try to find and provide the community with stories of interest with an emphasis on STEM.

This is also a good opportunity to try your hand at submittting a story to SoylentNews. Please see the Submission Guidelines. If you have any questions, feel free to pop into the "#editorial" channel on IRC (Internet Relay Chat) and ask for help. As some of our staff may also be taking a vacation, there may not be someone there to immediately respond, so please bear with us. I well remember the thrill I felt when I first saw a story submission of mine accepted and posted to the site!

If the best you can do is submit a link, then do that. If you quote any text directly from the story, please enclose it in <blockquote> ... </blockquote> tags. Of course the closer the submission is to publication-ready, the less work is required of an editor, and the better chance that one of us will run with it. Most importantly, have fun and don't be afraid to try. We all had the experience of making our first story submission and are willing to help you get comfortable with the process, as others did with us.

Thank YOU: We are the little site that could. Splintered off from the non-listening corporate overlords at Slashdot, a bunch of PO'd folk got together and bludgeoned an out-of-date code base into a solid and responsive system. The community that has formed here... I regularly see people looking to find what they can give to the community rather than what they can take from it. We could not have done it without YOU. Providing story submissions. Writing comments and journal articles. Moderating comments. It is a group effort, and this is a group I am grateful to be a part of.

In addition, there is the unheralded work of a small staff who volunteer their time to (try) and keep everything up and running smoothly. For example, Linode (our webhosting provider) recently rebooted 5 of our servers to handle an exploit that was making the rounds. Thanks to the excellent design and implementation, the site kept right on running uninterrupted. There was a minor glitch this morning that a restart of varnishd took care of. And that was it. No muss, no fuss. I count myself most fortunate to be working with such capable and talented people who give so selflessly of themselves to keep this all running so smoothly.

It has been a privilege to serve this community over the past 5 years and it is my fervent hope that we have many more years to come!

Original Submission

posted by martyb on Thursday July 04, @03:47AM   Printer-friendly
from the forewarned-is-forearmed dept.

Updated (20190704_023935 UTC) Server reboots have completed. I am unaware of any issues from these reboots; please reply here and post to the #dev channel on IRC if anything is amiss. Original story follows with minor updates to mark all servers have been rebooted. --martyb

For those who might not be aware, SoylentNews operations run on servers from Linode. I have recently become aware of their plans to reboot servers:

To complete our mitigations against the recent MDS (ZombieLoad) CPU vulnerability, we will be performing maintenance on a subset of Linode’s host machines. This maintenance will update the underlying infrastructure that Linodes reside on and will not affect the data stored within them.

Here is the schedule for our affected systems:

fluorine (*)2019-06-25 05:00 AM UTC
beryllium (*)2019-06-27 09:00 AM UTC
helium (*)2019-06-28 03:00 AM UTC
boron (*)2019-06-28 04:00 AM UTC
hydrogen (*)2019-07-02 09:00 AM UTC
sodium (*)2019-07-03 02:00 AM UTC

(*) Completed.

Historically, there is a two-hour window for reboots to occur, but it usually takes far less time than that.

We will attempt to minimize any impact on site operations, but want to let the community know what was coming up.

posted by martyb on Monday June 24, @12:37PM   Printer-friendly
We are aware of issues when trying to access the site. First noticed at approx. 0300 UTC. Our servers look okay. It appears there may be issues with upstream connectivity.

Also, Linode is planning some server reboots over the next week or so. We will try to give advance notice and keep downtime to a minimum.

Update: Everything seems to have quieted down. Many many thanks to NotSanguine for jumping in and lending his expertise to help identify and isolate where things were borked.

Indications are that a bad BGP (Border Gateway Protocol) route was published causing a relatively small AS (Autonomous System) to have all traffic to/from a large fraction of the internet attempt to go through its routers.

posted by martyb on Tuesday June 11, @04:00PM   Printer-friendly
from the passing-the-word dept.

As many of you are aware, SoylentNews uses Let's Encrypt certificates to protect the vast majority (all?) of our networking connections.

Under the watchful eyes of The Mighty Buzzard and SemperOSS I have updated our certs and deployed them across our servers and services. At this point, all seems to be working fine. That said, I have a well-earned reputation of being able to break nearly anything, so it would not entirely surprise me if you find something awry. If so, please let us know! You can comment on this story and/or jump over to the "#dev" channel on IRC and let us know there.

For completeness' sake, the updated certs were rolled out at 20190611_140630 UTC.

NOTE: Do be aware it takes time for updated DNS records to work their way across the internet, so if you do encounter a problem, try clearing your cache and trying again before assuming things are borked.

Original Submission

posted by janrinok on Monday May 27, @03:45AM   Printer-friendly
from the easy-now dept.

Holiday Weekend: For our international readers who may be unaware, Monday 27 May marks Memorial Day in the United States, a remembrance of members of the military who have passed in the line of service. In reality, if has become the de facto start of summer which runs past Independence Day (July 4) until Labor Day.

Over this 3-day-long holiday weekend, a great number of establishments are closed or run at reduced hours and staffing. Stories appear on the internet at a reduced rate, and many more "fluff pieces" appear as filler.

I have heard quite clearly that, given a choice, the community generally prefers quality over quantity.

SoylentNews runs with approximate story spacings of a bit over 1.5 hours on weekdays and a little under 2.5 hours on weekends. We've been "feeding the queue" 24 hours per day, every single day, for over 5 years (still amazes me it has been that long!)

So, based primarily on those factors, I have suggested the eds adopt weekend story spacing for Monday (UTC) so they, too, can have a bit of a break.

Read on past the break for the rest of the site news.

Staffing: I would like to take this opportunity to formally welcome back fnord666 who had been on leave from the end of last year into early this year. Good to have you back! On the other hand, mrpg has recently moved and will be incommunicado until he is able to get situated again. Please join me in wishing him well! Our Editor-in-Chief emeritus is back in the saddle, too, though his circumstances may change at any time. He has many outside obligations and yet willingly gives of his very limited free time to SoylentNews. We are indeed fortunate to have him with us; thanks JR!

Outage: Well, it was not actually an outage, but on May 22, 2019 at approximately 2300 UTC, people started reporting issues with the site. Symptoms included some of the slashboxes on the side of the main page were not loading as well as some reports that CSS (Cascading Style Sheets) were failing to load causing malformed layout. I got wind of the issue just before midnight. It took me a couple tries to invoke the correct script to "bounce" the front-end servers (which basically just restarts apache on both our front-end servers and restarts the single slashd daemon). I promptly reported the outage and corrective measures to the community and it seems that all is functioning properly again. Please speak up in the comments if you are still experiencing any issues. But... be sure to clear your browser cache and do a hard reload of the site (e.g. Ctrl+F5 works on Firefox and derivatives) to make sure you are getting a fresh copy of all the files.

Finances: We have reached our funding goal of $2000 for the first half of 2019! Better still, we still have the rest of May and all of June remaining. I took the liberty of updating the "beg-o-meter" in the site news slashbox with a stretch goal of $1000. I am happy to report that we have already received $168.93 towards the stretch goal. I am so grateful for the support of the community for our efforts. For those who may be new or unawares, this is a purely volunteer organization. Nobody on staff has ever received any remuneration ($5 word for pay) for their efforts on this site. As I understand it, both NCommander and Matt_ incurred large monetary expenses in getting the site going and established legally as a Private Benefit Corporation based out of Maryland — expenses for which they have never been repaid. They have been more than patient and gracious in their insistence that SoylentNews become well enough established before getting repaid.

When you subscribe, some of the site limits are relaxed and you get a shiny star next to any comments you post. For the humble, you can turn that star's display off in your preferences.

If you wish to help out, click on Subscribe and select whether you want the subscription to start/extend your own subscription or you wish to make a gift subscription. If it is a gift subscription, specify the UID for the recipient. The default of UID==6 is that of Michael Casadevall (another nick NCommander used when setting up the site) or you may replace the UID with 2339 in memory of Michael David Crawford, or any other UID that you want.

Penultimately, select a subscription duration and amount (the suggested amount is a minimum; any in excess of that is greatly appreciated), and click on the Continue button. NOTE: Javascript needs to have been enabled for the following step to work. Lastly, choose your payment method of either Paypal or Stripe, fill in the requested details, and submit.

As I have mentioned before, this site has real expenses with server hosting fees, domain name registration, paying for a CPA to file our taxes, and the like. Those who support us financially help "keep the lights on." Thank You!

Submissions: This is not to take away from those who contribute to the site in other ways. Please accept my thanks to all of you who submit stories for the editors to poke at and bring to the community. Some of the community post stories to their journal which has led to many an interesting conversation. Above all, thank you to those who weigh in by posting comments. It is the primary reason for this site's existence, so please keep those comments coming!

Moderations: All comments are visible, but some have more visibility than others, thanks to moderation. For those who wish to see just the crème de la crème, set your comment threshold higher and give thanks to those who have performed moderations.

With respect to moderations, a user may moderate anywhere from 0 to 4 comments by another user. Any moderations in excess of 4 upmods or 4 downmods in a 24-hour period starting at 00:10 UTC will be reversed. It is our intention at some point to have the site impose that limitation automatically so no penalty is currently being imposed at this time. That means we need to monitor moderations and manually reverse those in excess of the limits. The short form of the moderation guidelines is "Don't be a jerk." On the other hand, 10 mod points are issued to every registered user at 00:10 UTC each day... please use them! Please try to focus on upmodding good (e.g. Funny, Interesting, Insightful, Informative) comments but don't be afraid to downmod (e.g. Offtopic, Troll, Flamebait) where appropriate.

When I moderate a comment, my thoughts are for the next person to come along; how can I help curate the discussion to best help them out?

Thank You! Thanks to all the rest of the staff who humbly trudge along behind the scenes to make it happen. We have posted over 27,000 stories and have received nearly 850,000 comments in the just over 5 years since SoylentNews was launched... here's looking forward to many more years to come!!

Original Submission

posted by martyb on Thursday May 23, @12:30AM   Printer-friendly
from the insert-picure-of-big-haired-guy-saying-it-was-aliens.jpg dept.

We recently had reports of some 'wonkiness' on the site.

Reports started appearing in our #Soylent channel on IRC about 90 minutes ago. For example, the Main Page would load, but parts of the page (such as the slashboxes on the left-hand side) were missing.

I borked my first attempt, but on the second try successfully restarted apache and slashd on both of our front-end servers: hydrogen and fluorine.

All seems to be working correctly now. If this is not the case for you, you may need to clear the browser cache on your system and/or do a hard reload of the page (e.g. Ctrl+F5). If things are still not right, please reply in the comments and/or pop into channel #dev on IRC and let us know.

I'm not sure what precipitated the behavior; I'll leave that to TheMightyBuzzard or one of the sysadmin folk to investigate. I'm sure they'll fill us in on whatever root cause was found.

OTOH, one cannot entirely rule out a spurious gamma ray that flipped a key bit in memory that caused an avalanche of untrapped events.

You can now return to your usual, spirited discussions.

Original Submission