What started it all:
On 2019-08-24 13:02:01 UTC an accusation (https://soylentnews.org/meta/comments.pl?noupdate=1&sid=33244&page=1&cid=884682#commentwrap) was made that a Journal Entry "It would have been posted before 6 hours ago" (i.e. posted at approximately 2019-08-24 07:00:00 UTC) was deleted by a member of the staff at SoylentNews. The circumstances surrounding the making of the Journal Entry are elaborated upon in this comment. (https://soylentnews.org/meta/comments.pl?noupdate=1&sid=33244&page=1&cid=885191#commentwrap)
I have been with this site since before it went live. Its founding principal has been the making available of a forum whereby the community can submit stories — and post comments — to predominantly tech-related items. Further, each logged-in user has been made available the ability to post entries to their Journal.
As Editor-in-Chief I took this allegation seriously and performed an independent and in-depth investigation. My findings are presented below.
Note: It is not lost on me the futility of trying to prove a negative. It is for good reason that the criminal justice system in the US is founded on the principle of "innocent until proven guilty." It is not up the the accused to vindicate themselves, but for the accuser to bring sufficient evidence to bring about conviction.
NB: In the course of writing this, I discovered a bug in how the site displays wide elements contained in an ECODE element. It incorrectly wraps the text onto the next line (leading to a jumbled mess) when it should, instead, provide horizontal scroll bars. Please accept my apologies for its current appearance.
Executive Summary:
An in-depth investigation making use of: external resources, the UI presented by SoylentNews, and ad-hoc queries of the site database (DB) failed to locate a "smoking gun", i.e. found no clear proof that a Journal Entry was posted to the site and subsequently deleted by anyone other than an author.
It is my estimation that the user submitted an entry, but the site failed to receive and save it correctly. In other words, the user tripped over some kind of bug be it in the site's code, communications between the user and the site, or something else.
Recommendation: When a user completes making a Journal Entry and submits it to the site, the code should respond by using the newly-created journal parameters in conjunction with the normal journal-loading code to present the Journal Entry to the user as confirmation that the entry was properly received and saved. That is to say, affirmative feedback of receipt, storage, and accessibility of the Journal Entry.
Acknowledgements:
For those who may not be aware, all staff on SoylentNews are volunteers; nobody has ever been paid anything for their work on this site. Everything has been done in people's spare time. Early morning before work. During work breaks. Nights, weekends, holidays, and vacations too.
I hereby express my gratitude to all who have supported my work on this investigation. I have invested at least two person-weeks of effort digging into this. This necessitated my letting up on processing story submissions and making them available on the site. Please join me in thanking janrinok, fnord666, takyon, and chromas who stepped up and carried much of the editing load while I have been busy on this investigation. Thanks are in order, too, for @SemperOSS; lending a most-welcome helping hand in finding locations of — and incantations for extracting pertinent data from — important information about how the site is configured and operates.
teamwork++
Early Site History:
For those who may not have been here at the beginning, SoylentNews arose as a result of frustration with how Slashdot failed to listen to what they called "their audience" in response to their rollout of a Beta version of their site. (And also lead to the infamous "Buck Feta!" war cry.) This, in turn, lead to the infamous "Slashcott" — a week-long period where users voluntarily abstained from visiting Slashdot. (Slashdot boycott). This Slashcott was then extended for an additional week.
While this was happening, some like-minded folk got together and found an open-sourced release of slashcode. This code was a few years out of date and depended on old versions of other applications. Among other things, this included the database server (MySQL), the web server (Apache), web cache (memcached), and Perl in which the site code had been written. In a frantic two-week-long period, these intrepid coders took this ancient software and hacked it into "running condition" and SoylentNews went public on February 16th, 2014.
The first few weeks of SoylentNews were filled with site crashes and reports of display artifacts and problems with the general operation of the site. In short, many bugs resulting from the feverish pace of development revealed themselves and, over time, were found and quashed. It is a testament to their hard work and dedication that problems with the site are, today, viewed as an aberration instead of a regular occurrence. I do not know whether or not the original Slashcode ever underwent a rigorous review. But, given the amount of hacking involved in just getting SoylentNews up-and-running, any stability of the current site is mostly a result of extremely dedicated Whack-A-Mole debugging.
Sections:
- Site Crash on 2019-08-15
- Malicious URL?
- Other Sources
- Use the Source, Luke!
- The Journals-Related Tables
- The Discussion-Related Tables
- Or is it?
- The Comment-Related Tables
- Trying Another Perspective
- The Message-Related Tables
- Conclusion
- Recommendation
Site Crash on 2019-08-15:
We had a site crash on August 15, 2019. We do not know why. Given the site had already been down for several hours, it was deemed necessary to restore a backup to the drive on which the site database resides. No root cause analysis is practicable at this time. Over the next couple days, we found some artifacts from that unplanned crash and site restart. For example, we discovered that slashd — the daemon which launches periodic processes in support of the site (such as emailing notifications about comment moderations and replies) had not been restarted. Also, it is possible that some race condition arose that interfered with the site properly saving a Journal Entry.
Malicious URL?:
It was suggested that someone could construct a malicious URL to delete another user's journal. The author of a Journal Entry has a button they can click so as to delete one of their own Journal Entries. It was thought that, perhaps, a modification of the resultant URL might suffice to delete a Journal Entry for someone else. I tried this on our development server where I have "god mode" privileges; absolutely nobody on the production SoylentNews site has this privilege level. Even with these elevated privileges, my attempt failed: I saw no error message, but neither was the Journal Entry deleted.
Finding: Even the most-priviledged user cannot use a malicious URL to delete a Journal Entry.
Other Sources:
As part of this investigation, I looked for data that may have been captured externally. I posed queries to Google for the Journal Entry author's nickname and the date of the post, and found nothing. Another place I looked was the Wayback Machine on the Internet Archive. I found these archives for that date:
- https://web.archive.org/web/20190824042431/https://soylentnews.org/
- https://web.archive.org/web/20190824102455/https://soylentnews.org/
- https://web.archive.org/web/20190824162417/https://soylentnews.org/
- https://web.archive.org/web/20190824222439/https://soylentnews.org/
Notice how the link contains the date and time of when the site was scanned: "YYYY MM DD hh mm ss". The first two links bracket the date/time in question. I could find no indication of a Journal Entry there, either.
Finding: Failed to find an external source which showed that the Journal Entry had been posted to the site.
Use the Source, Luke!
Well, the source of data for the entire site: the database, that is. Having found no external corroboration, there was nothing left but to dive into the database.
It is worth mentioning at this point that the source code for SoylentNews has been made available on GitHub under https://github.com/soylentnews. Note that the original slashcode on which soylentnews was built was saved under https://github.com/SoylentNews/slashcode and had a last commit in September of 2009. Observant readers will note that leaves a nearly 4.5 year gap from last commit and the use of it as a base for SoylentNews which launched on 2014-02-16 . Many dependent packages had undergone major updates and bug fixes since that time. This required significant modification in the received code just to be able to run at all.
The code which runs the current SoylentNews site can be found under https://github.com/SoylentNews/rehash.
The Journals-Related Tables:
This was the obvious starting point. There are 4 potential tables of interest:
- journal_themes
- journal_transfer
- journals
- journals_text
Nothing much of interest in journal_themes:
mysql> DESCRIBE journal_themes ;
+-------+---------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------+---------------------+------+-----+---------+----------------+
| id | tinyint(3) unsigned | NO | PRI | NULL | auto_increment |
| name | varchar(30) | NO | UNI | NULL | |
+-------+---------------------+------+-----+---------+----------------+
2 rows in set (0.00 sec)
mysql> SELECT * FROM journal_themes ORDER BY id ;
+----+----------+
| id | name |
+----+----------+
| 1 | generic |
| 2 | greypage |
| 3 | bluebox |
+----+----------+
3 rows in set (0.00 sec)
There is nothing at all in the journal_transfer table:
mysql> SELECT count(*) FROM journal_transfer ;
+----------+
| count(*) |
+----------+
| 0 |
+----------+
1 row in set (0.00 sec)
mysql>
The journals table provides an "abstraction" of a Journal Entry. The actual text of a Journal Entry is stored in the journals_text table:
mysql> DESCRIBE journals_text;
+-----------+-----------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+-----------------------+------+-----+---------+-------+
| id | mediumint(8) unsigned | NO | PRI | NULL | |
| article | mediumtext | NO | | NULL | |
| introtext | mediumtext | NO | | NULL | |
+-----------+-----------------------+------+-----+---------+-------+
3 rows in set (0.00 sec)
mysql>
We will come back to this table shortly. For now, suffice it to say that the id field provides an index into both the journals and journals_text tables, the introtext field contains the HTML for the Journal Entry, and the article field contains a plain-text rendition of the introtext field.
Now, let's look closer at the journals table:
mysql> DESCRIBE journals ;
+-------------+------------------------------------+------+-----+-------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+------------------------------------+------+-----+-------------------+-----------------------------+
| id | mediumint(8) unsigned | NO | PRI | NULL | auto_increment |
| uid | mediumint(8) unsigned | NO | MUL | NULL | |
| date | datetime | NO | | NULL | |
| description | varchar(80) | NO | | NULL | |
| posttype | tinyint(4) | NO | | 2 | |
| discussion | mediumint(8) unsigned | YES | | NULL | |
| tid | smallint(5) unsigned | NO | MUL | NULL | |
| promotetype | enum('publicize','publish','post') | NO | | publish | |
| last_update | timestamp | NO | | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
| srcid_32 | bigint(20) unsigned | NO | MUL | 0 | |
| srcid_24 | bigint(20) unsigned | NO | MUL | 0 | |
+-------------+------------------------------------+------+-----+-------------------+-----------------------------+
11 rows in set (0.01 sec)
mysql>
Let's first look for entries made within a day on either side of 2019-08-24:
mysql> SELECT id, date, uid, description, discussion FROM journals WHERE "2019-08-23" < date AND date < "2019-08-26" ORDER BY id ;
+------+---------------------+------+-------------------------------------------------------------+------------+
| id | date | uid | description | discussion |
+------+---------------------+------+-------------------------------------------------------------+------------+
| 4512 | 2019-08-23 15:02:42 | 4388 | You Are Banned. | 33255 |
| 4513 | 2019-08-23 16:25:06 | 6150 | "Our great American companies are hereby ordered..." | 33256 |
| 4514 | 2019-08-24 04:38:24 | 4543 | Perdurabo. | 33265 |
| 4515 | 2019-08-24 08:17:01 | 4543 | By request, The HU. | 33266 |
| 4517 | 2019-08-24 20:09:11 | 5086 | No, Virginia, "conservatism" is not the new counter-culture | 33274 |
| 4518 | 2019-08-25 13:35:30 | 2926 | The 'Soft' Racism of Gun Control | 33285 |
| 4520 | 2019-08-25 16:38:48 | 5291 | Valediction: On corruption and moderation | 33287 |
| 4521 | 2019-08-25 22:04:51 | 3682 | Plasma Tsunamis Responsible for Sunspots | 33289 |
+------+---------------------+------+-------------------------------------------------------------+------------+
8 rows in set (0.02 sec)
mysql>
Notice that (id==4520) was posted by this user (uid==5291), but... it is not the droid Journal Entry we were looking for. Looking closer, notice how we are missing (id==4516) and (id=4519). Apparently, deleting a Journal Entry actually deletes it from the database, instead of leaving it in the DB but flagging it as being deleted (i.e. unavailable).
Historical Perspective: Slashdot was registered on 1996-11-17. Disk space usage was a much greater concern back then. For historical perspective consider that Windows 95 had just been released and that the Pentium MMX and Pentium Pro were introduced in about the same time period. As for disk storage, this timeline of hard disk drives notes that in 1996 Seagate shipped its first 10,000 RPM "Cheetah" disk drive and 1997 saw the release of the "IBM Deskstar 16GP 'Titan' - 16,800 megabytes, five 3.5-inch disks". Yes, 16,800 MB or 16.8 GB. Flash drives of that capacity can be purchased for under $10 today.
So, getting back to the missing entries, we can ignore (id==4519) as that was clearly well past the time of interest. That leaves (id==4516)... where did it go?
Let's see what we can learn from looking at the journals_text table.
mysql> SELECT id, substr(introtext, 1, 80) FROM journals_text WHERE 4512 <= id AND id <= 4521 ORDER BY id ;
+------+----------------------------------------------------------------------------------+
| id | substr(introtext, 1, 80) |
+------+----------------------------------------------------------------------------------+
| 4512 | <p>This morning, I took a quick look at the website of the Church of the Flying |
| 4513 | <p>You probably saw who said that, so I don't have to mention his name.</p><p>Bu |
| 4514 | <tt>I've mentioned before that I perceive hip-hop as more vibrant, more alive, t |
| 4515 | <tt>https://www.youtube.com/watch?v=v4xZUr0BEfE<br><br>Please, anyone, contribut |
| 4517 | Recently one of our frothier, crazier members, who is in my opinion a fascinatin |
| 4518 | <p>The historically racist origins of gun control are hardly a topic for debate. |
| 4520 | <p>This will be my final interaction with this site, which I have supported for |
| 4521 | <p><b>Rejected submission by RandomFactor at 2019-07-25 21:17:28 from the tSUNam |
+------+----------------------------------------------------------------------------------+
8 rows in set (0.01 sec)
Hmmm, same missing entries as before; no joy here.
Attentive readers will have noticed the discussion column in the journals table. Further, for (id==4515) we have (discussion==33266) and for (id==4517) we have (discussion==33274). We will come back to those presently.
Finding: We have failed to find the sought-for Journal Entry, but have found two tuples of id and discussion that warrant further investigation.
The Discussion-Related Tables:
Stepping back for a moment, since comments can be posted to both Journal Entries and stories, it makes sense they may share some commonality. There are two tables related to discussions in our database: discussion_kinds and discussions.
mysql> DESCRIBE discussion_kinds ;
+-------+---------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------+---------------------+------+-----+---------+----------------+
| dkid | tinyint(3) unsigned | NO | PRI | NULL | auto_increment |
| name | varchar(30) | NO | UNI | | |
+-------+---------------------+------+-----+---------+----------------+
2 rows in set (0.00 sec)
mysql> SELECT * FROM discussion_kinds ORDER BY dkid ;
+------+---------------+
| dkid | name |
+------+---------------+
| 1 | story |
| 2 | user_created |
| 3 | journal |
| 4 | journal-story |
| 5 | poll |
| 6 | submission |
| 7 | feed |
| 8 | project |
+------+---------------+
8 rows in set (0.00 sec)
Now we are getting somewhere! There are entries for story and for journal, just as we had hoped!
The other table of interest was discussions. Let's see what we can find there.
mysql> DESCRIBE discussions ;
+---------------+------------------------------------------------------------------------------------------------+------+-----+---------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+---------------+------------------------------------------------------------------------------------------------+------+-----+---------------------+-----------------------------+
| id | mediumint(8) unsigned | NO | PRI | NULL | auto_increment |
| dkid | tinyint(3) unsigned | NO | | 1 | |
| stoid | mediumint(8) unsigned | NO | MUL | 0 | |
| sid | char(16) | NO | MUL | | |
| title | varchar(128) | NO | | NULL | |
| url | varchar(255) | NO | | NULL | |
| topic | int(10) unsigned | NO | MUL | NULL | |
| ts | datetime | NO | | 1970-01-01 00:00:00 | |
| type | enum('open','recycle','archived') | NO | MUL | open | |
| uid | mediumint(8) unsigned | NO | | NULL | |
| commentcount | smallint(5) unsigned | NO | | 0 | |
| flags | enum('ok','delete','dirty') | NO | | ok | |
| primaryskid | smallint(5) unsigned | YES | MUL | NULL | |
| last_update | timestamp | NO | | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
| approved | tinyint(3) unsigned | NO | | 0 | |
| commentstatus | enum('disabled','enabled','friends_only','friends_fof_only','no_foe','no_foe_eof','logged_in') | NO | | enabled | |
| archivable | enum('no','yes') | NO | | yes | |
| legacy | enum('no','yes') | NO | | yes | |
+---------------+------------------------------------------------------------------------------------------------+------+-----+---------------------+-----------------------------+
18 rows in set (0.01 sec)
So far, so good. Let's take a look inside.
mysql> SELECT id, dkid, stoid, sid, substr(title,1,20), url, ts, uid, last_update FROM discussions WHERE "2019-08-23 00:00:01" < ts AND ts < "2019-08-24 23:59:59" ORDER BY id ;
+-------+------+--------+------------------+----------------------+---------------------------------------------------+---------------------+------+---------------------+
| id | dkid | stoid | sid | substr(title,1,20) | url | ts | uid | last_update |
+-------+------+--------+------------------+----------------------+---------------------------------------------------+---------------------+------+---------------------+
| 33258 | 1 | 370766 | 19/08/23/2359248 | Billionaire David Ko | //soylentnews.org/article.pl?sid=19/08/23/2359248 | 2019-08-24 00:28:00 | 76 | 2019-08-29 01:41:03 |
| 33259 | 1 | 370767 | 19/08/24/0012203 | Study Corroborates t | //soylentnews.org/article.pl?sid=19/08/24/0012203 | 2019-08-24 02:49:00 | 76 | 2019-08-27 04:52:14 |
| 33260 | 1 | 370768 | 19/08/24/0021202 | Alleged "Snake Oil" | //soylentnews.org/article.pl?sid=19/08/24/0021202 | 2019-08-24 05:12:00 | 76 | 2019-08-25 06:48:09 |
| 33261 | 1 | 370769 | 19/08/24/0027221 | Federal Agency Turf | //soylentnews.org/article.pl?sid=19/08/24/0027221 | 2019-08-24 07:32:00 | 76 | 2019-08-25 23:40:34 |
| 33262 | 1 | 370770 | 19/08/24/0038241 | Phone Companies and | //soylentnews.org/article.pl?sid=19/08/24/0038241 | 2019-08-24 09:54:00 | 76 | 2019-08-27 02:50:36 |
| 33263 | 1 | 370771 | 19/08/24/0133226 | EU Officials Draft P | //soylentnews.org/article.pl?sid=19/08/24/0133226 | 2019-08-24 12:19:00 | 76 | 2019-08-25 11:48:20 |
| 33264 | 1 | 370772 | 19/08/24/0135254 | YouTube Banning Comb | //soylentnews.org/article.pl?sid=19/08/24/0135254 | 2019-08-24 14:37:00 | 76 | 2019-08-25 04:10:23 |
| 33265 | 3 | 0 | | Perdurabo. | //soylentnews.org/~Arik/journal/4514 | 2019-08-24 04:38:24 | 4543 | 2019-08-27 08:28:48 |
| 33266 | 3 | 0 | | By request, The HU. | //soylentnews.org/~Arik/journal/4515 | 2019-08-24 08:17:01 | 4543 | 2019-08-27 15:02:04 |
| 33267 | 1 | 370773 | 19/08/24/128219 | NASA Astronaut's Div | //soylentnews.org/article.pl?sid=19/08/24/128219 | 2019-08-24 16:55:00 | 76 | 2019-08-29 18:13:08 |
| 33268 | 1 | 370774 | 19/08/24/1216253 | Neil Young on Sound | //soylentnews.org/article.pl?sid=19/08/24/1216253 | 2019-08-24 19:17:00 | 76 | 2019-08-26 18:42:21 |
| 33269 | 3 | 0 | | Test | //soylentnews.org/~AthanasiusKircher/journal/4516 | 2019-08-24 12:55:02 | 5291 | 2019-08-24 17:17:12 |
| 33270 | 1 | 370775 | 19/08/24/1341250 | Deadly Superbug Outb | //soylentnews.org/article.pl?sid=19/08/24/1341250 | 2019-08-24 21:34:00 | 52 | 2019-08-26 04:19:22 |
| 33271 | 1 | 370776 | 19/08/24/1347258 | Capsule Containing R | //soylentnews.org/article.pl?sid=19/08/24/1347258 | 2019-08-24 23:53:00 | 52 | 2019-08-26 18:27:55 |
| 33274 | 3 | 0 | | No, Virginia, "conse | //soylentnews.org/~Azuma+Hazuki/journal/4517 | 2019-08-24 20:09:11 | 5086 | 2019-09-09 06:29:43 |
+-------+------+--------+------------------+----------------------+---------------------------------------------------+---------------------+------+---------------------+
15 rows in set (0.62 sec)
And how about that! There — at (id==33269) — is our missing Journal Entry number 4516.
Or is it?
Let's make no assumptions. Let's try to load it in a browser https://soylentnews.org/~AthanasiusKircher/journal/4516:
Sorry, the requested journal entries were not found.
Now that is some progress. Really! We found something that was posted to a journal, but was deleted. The very one that the user stated they had created after noticing the apparent disappearance of his journal article.
So, umm, where is the other Journal Entry? You know, the missing one? Yeah, we're still working on that one, but at least our theory of what happens in the site code for an intentional deletion of a Journal Entry is validated.
Looking back, remember the discussion column from the journals table and those we were going to come back to. Now is the time; here is what we had:
for (id==4515) we have (discussion==33266) and for (id==4517) we have (discussion==33274)
Here, in the discussions table, we see that discussion 33266 corresponds to https://soylentnews.org/~Arik/journal/4515 and discussion 33274 corresponds to https://soylentnews.org/~Azuma+Hazuki/journal/4517 so we have figured out where they went. Unfortunately, this does not provide any closure on the situation, but it does close off one potential avenue of explanation.
Finding: We have found evidence that when a user deletes a Journal Entry, any comments that had been posted to it persist in the database. None have yet been found.
The Comment-Related Tables:
As was the case for the journals table, SoylentNews' comments are split up as well. Here are all of the comment-related tables:
- comment_log
- comment_promote_log
- comment_text
- commentmodes
- comments
I do not know why they still exist, but both comment_log and comment_promote_log are empty. As for commentmodes, that is the source for the pull-down list of how one chooses to see comments displayed:
mysql> SELECT * FROM commentmodes ORDER BY mode ;
+-----------+--------------+-------------+
| mode | name | description |
+-----------+--------------+-------------+
| flat | Flat | |
| nocomment | No Comments | |
| threadtng | Threaded-TNG | NULL |
| threadtos | Threaded-TOS | NULL |
+-----------+--------------+-------------+
4 rows in set (0.00 sec)
So now we are down to the comments and comment_text tables. Last things first, the comment_text table is exactly what it says on the tin, where the text of each comment is stored:
mysql> DESCRIBE comment_text ;
+---------+------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------+------------------+------+-----+---------+-------+
| cid | int(10) unsigned | NO | PRI | NULL | |
| comment | mediumtext | NO | | NULL | |
+---------+------------------+------+-----+---------+-------+
2 rows in set (0.01 sec)
The key thing here is the cid (Comment ID) field which is a pointer (key) into the comments table. Having gotten all that out of the way, what can the actual comments table tell us?
mysql> DESCRIBE comments ;
+------------------+-----------------------+------+-----+---------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+------------------+-----------------------+------+-----+---------------------+-----------------------------+
| sid | mediumint(8) unsigned | NO | MUL | NULL | |
| cid | int(10) unsigned | NO | PRI | NULL | auto_increment |
| pid | int(10) unsigned | NO | MUL | 0 | |
| opid | int(10) unsigned | NO | MUL | 0 | |
| children | int(10) unsigned | NO | | 0 | |
| date | datetime | NO | MUL | 1970-01-01 00:00:00 | |
| last_update | timestamp | NO | | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
| ipid | char(32) | NO | MUL | | |
| subnetid | char(32) | NO | MUL | | |
| subject | varchar(50) | NO | | NULL | |
| subject_orig | enum('no','yes') | NO | | yes | |
| uid | mediumint(8) unsigned | NO | MUL | NULL | |
| points | tinyint(4) | NO | | 0 | |
| pointsorig | tinyint(4) | NO | | 0 | |
| pointsmax | tinyint(4) | NO | | 0 | |
| lastmod | mediumint(8) unsigned | NO | | 0 | |
| reason | tinyint(3) unsigned | NO | | 0 | |
| signature | char(32) | NO | | | |
| karma_bonus | enum('yes','no') | NO | | no | |
| subscriber_bonus | enum('no','yes') | NO | | no | |
| len | smallint(5) unsigned | NO | | 0 | |
| karma | smallint(6) | NO | | 0 | |
| karma_abs | smallint(5) unsigned | NO | | 0 | |
| tweak_orig | tinyint(4) | NO | | 0 | |
| tweak | tinyint(4) | NO | | 0 | |
| badge_id | tinyint(4) | NO | | 0 | |
+------------------+-----------------------+------+-----+---------------------+-----------------------------+
26 rows in set (0.00 sec)
Zoinks! That's a lot, but we can ignore much of it for our purposes. We just want to know if any comments were made to the missing Journal Entry. Let's look at a subset:
mysql> SELECT cid, sid, pid, opid, uid, children, date, last_update, subject FROM comments WHERE pid = 0 AND "2019-08-24 06:00:00" <= date AND date <= "2019-08-24 08:00:00" ORDER BY cid;
+--------+-------+--------+--------+------+----------+---------------------+---------------------+--------------------------------------------------+
| cid | sid | pid | opid | uid | children | date | last_update | subject |
+--------+-------+--------+--------+------+----------+---------------------+---------------------+--------------------------------------------------+
| 884595 | 33250 | 884485 | 884479 | 1 | 0 | 2019-08-24 06:00:50 | 2019-09-23 07:02:01 | Re:so? |
| 884596 | 33244 | 884418 | 883941 | 1 | 19 | 2019-08-24 06:01:46 | 2019-09-23 07:02:01 | Re:Sign of the times |
| 884597 | 33265 | 0 | 0 | 18 | 15 | 2019-08-24 06:03:10 | 2019-09-23 07:02:01 | Wait, wut? |
| 884598 | 33259 | 884585 | 884531 | 1 | 1 | 2019-08-24 06:08:34 | 2019-09-23 07:02:01 | Re:Wow! Then radiation hits fetus... |
| 884599 | 33244 | 884520 | 883955 | 1 | 2 | 2019-08-24 06:10:36 | 2019-09-23 07:02:01 | Re:It's exaeta again, isn't it? |
| 884600 | 33244 | 884596 | 883941 | 18 | 18 | 2019-08-24 06:11:13 | 2019-09-23 07:02:01 | Re:Sign of the times |
| 884601 | 33252 | 884400 | 884336 | 2828 | 0 | 2019-08-24 06:11:29 | 2019-09-23 07:02:01 | Re:What's the appeal of mercedes? |
| 884602 | 33265 | 884581 | 884581 | 2489 | 2 | 2019-08-24 06:18:09 | 2019-09-23 07:02:01 | Re:Is a rapper a singer? |
| 884603 | 33259 | 884598 | 884531 | 1 | 0 | 2019-08-24 06:27:33 | 2019-09-23 07:02:01 | Re:Wow! Then radiation hits fetus... |
| 884604 | 33259 | 884530 | 884530 | 6836 | 6 | 2019-08-24 06:30:48 | 2019-09-23 07:02:01 | Re:If so, then... |
| 884605 | 33265 | 884597 | 884597 | 4543 | 9 | 2019-08-24 06:31:30 | 2019-09-23 07:02:01 | Re:Wait, wut? |
| 884606 | 33265 | 884597 | 884597 | 2489 | 4 | 2019-08-24 06:41:07 | 2019-09-23 07:02:01 | Re:Wait, wut? |
| 884607 | 33265 | 884605 | 884597 | 2489 | 8 | 2019-08-24 06:44:10 | 2019-09-23 07:02:01 | Re:Wait, wut? |
| 884608 | 33244 | 884457 | 883942 | 52 | 0 | 2019-08-24 06:47:11 | 2019-09-23 07:02:01 | Re:WTF? |
| 884609 | 33259 | 884604 | 884530 | 881 | 5 | 2019-08-24 06:48:32 | 2019-09-23 07:02:01 | Re:If so, then... |
| 884610 | 33265 | 884607 | 884597 | 4543 | 7 | 2019-08-24 06:55:40 | 2019-09-23 07:02:01 | Re:Wait, wut? |
| 884611 | 33244 | 884461 | 883966 | 52 | 1 | 2019-08-24 06:58:54 | 2019-09-23 07:02:01 | Re:Why do we moderate? - Because it is important |
| 884612 | 33259 | 884585 | 884531 | 1 | 0 | 2019-08-24 07:00:14 | 2019-09-23 07:02:01 | Re:Wow! Then radiation hits fetus... |
| 884613 | 33255 | 0 | 0 | 3272 | 1 | 2019-08-24 07:07:03 | 2019-09-24 07:02:02 | Works on reload. |
| 884614 | 33244 | 884435 | 883966 | 52 | 3 | 2019-08-24 07:08:08 | 2019-09-24 07:02:02 | Re:Why do we moderate? - Because it is important |
| 884615 | 33222 | 883944 | 883944 | 1 | 0 | 2019-08-24 07:39:07 | 2019-09-24 07:02:02 | Re:Come out, exaeta, come out! |
| 884616 | 33244 | 884337 | 883966 | 52 | 2 | 2019-08-24 07:39:07 | 2019-09-24 07:02:02 | Re:Why do we moderate? - Because it is important |
| 884617 | 33261 | 0 | 0 | 2452 | 16 | 2019-08-24 07:40:06 | 2019-09-24 07:02:02 | 5G is stupid |
| 884618 | 33265 | 884602 | 884581 | 4543 | 1 | 2019-08-24 07:42:09 | 2019-09-24 07:02:02 | Re:Is a rapper a singer? |
| 884619 | 33222 | 884211 | 883718 | 2645 | 0 | 2019-08-24 07:49:42 | 2019-09-24 07:02:02 | Re:Need specifics, actually. |
| 884620 | 33244 | 884534 | 883966 | 52 | 13 | 2019-08-24 07:54:11 | 2019-09-24 07:02:02 | Re:Why do we moderate? - Because it is important |
+--------+-------+--------+--------+------+----------+---------------------+---------------------+--------------------------------------------------+
26 rows in set (0.00 sec)
mysql>
The pid field is the comment's "Parent ID", i.e. the comment ID of the comment that is the parent of this comment. Let's look at only those comments which have NO parent id:
mysql> SELECT cid, sid, pid, opid, uid, children, date, last_update, subject FROM comments WHERE pid = 0 AND "2019-08-24 06:00:00" <= date AND date <= "2019-08-24 08:00:00" ORDER BY cid;
+--------+-------+-----+------+------+----------+---------------------+---------------------+------------------+
| cid | sid | pid | opid | uid | children | date | last_update | subject |
+--------+-------+-----+------+------+----------+---------------------+---------------------+------------------+
| 884597 | 33265 | 0 | 0 | 18 | 15 | 2019-08-24 06:03:10 | 2019-09-23 07:02:01 | Wait, wut? |
| 884613 | 33255 | 0 | 0 | 3272 | 1 | 2019-08-24 07:07:03 | 2019-09-24 07:02:02 | Works on reload. |
| 884617 | 33261 | 0 | 0 | 2452 | 16 | 2019-08-24 07:40:06 | 2019-09-24 07:02:02 | 5G is stupid |
+--------+-------+-----+------+------+----------+---------------------+---------------------+------------------+
3 rows in set (0.78 sec)
mysql>
All of these comments pertain to the story they are attached to:
- https://soylentnews.org/comments.pl?sid=33265&cid=884597#commentwrap
- https://soylentnews.org/comments.pl?sid=33255&cid=884613#commentwrap
- https://soylentnews.org/comments.pl?sid=33261&cid=884617#commentwrap
Finding: we found no evidence of a comment having been made to a Journal Entry, during the time in question.
Trying Another Perspective:
In this comment, it was suggested:
All your fans would have received a message about your journal entry instantly. I for one didn't. You can't invisibly delete a journal entry without:
a) resetting the journal auto-incrementing counter in the database (and there's no programattic way to do that, you can only move it forwards)
but also:
b) invisibly deleting *all* the messages that were sent out because of it
and therefore:
c) resetting the message auto-incrementing counter in the database (ditto)
and here's the killer:
d) on a live system
and the cherry on top:
e) with no logs
If we had someone with that level of technical competence on the administrative team, that would probably double the total amount of competence have - the above goes well beyond mere guru level.
Ockham's razor points to either human error or a bug.
We've previously shown that articles can be deleted; there is a button for it on the journal writer's edit page.
If the deleted Journal Entry was the one with the largest ID, the system just re-uses that slot.
But, the idea that a message might have been sent out announcing to a user that a friend of theirs posted a Journal Entry? that's interesting. Let's see where that leads us.
Finding: We have a new situation to investigate, but no results, yet.
The Message-Related Tables:
Here are all the message-related tables on the site:
- message_codes
- message_deliverymodes
- message_drop
- message_log
- message_web
- message_web_text
At this point, selections made by particular users will be discovered. In the interests of excluding 3rd-parties, please forgive me if some of this discussion is less detailed than previously.
Let's take these in order:
mysql> DESCRIBE message_codes ;
+-----------------+----------------------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------------+----------------------------------+------+-----+---------+-------+
| code | int(11) | NO | PRI | NULL | |
| type | varchar(32) | NO | | NULL | |
| seclev | int(11) | NO | | 1 | |
| modes | varchar(32) | NO | | | |
| send | enum('now','defer','collective') | NO | | now | |
| subscribe | tinyint(1) | NO | | 0 | |
| acl | varchar(32) | NO | | | |
| delivery_bvalue | int(10) unsigned | NO | | 0 | |
+-----------------+----------------------------------+------+-----+---------+-------+
8 rows in set (0.00 sec)
mysql> SELECT * FROM message_codes WHERE seclev = 1 ORDER BY code ;
+------+----------------------------+---------+-------+------------+-----------+-------+-----------------+
| code | type | seclev | modes | send | subscribe | acl | delivery_bvalue |
+------+----------------------------+---------+-------+------------+-----------+-------+-----------------+
| -2 | Registration Mail | 1 | 0 | now | 0 | | 0 |
| -1 | Unknown Message | 1 | 0 | now | 0 | | 0 |
| 0 | Daily Newsletter | 1 | 0 | now | 0 | | 1 |
| 1 | Daily Headlines | 1 | 0 | now | 0 | | 1 |
| 2 | Metamoderation Results | 1 | | collective | 0 | | 3 |
| 3 | Comment Moderation | 1 | | collective | 0 | | 3 |
| 4 | Comment Reply | 1 | | now | 0 | | 3 |
| 5 | Journal Entry by Friend | 1 | | now | 0 | | 3 |
| 6 | New Submission | 100 | 0 | now | 0 | | 1 |
| 7 | Journal Reply | 1 | | now | 0 | | 3 |
| 8 | New Comment | 1000000 | | now | 0 | | 0 |
| 10 | Daily Site Stats | 100 | 0 | now | 0 | stats | 1 |
| 11 | Email Story | 1 | 0 | now | 0 | | 1 |
| 12 | Relationship Change | 1 | | collective | 0 | | 3 |
| 13 | Bad login attempt warnings | 1 | 1 | now | 0 | | 2 |
| 14 | Daily Moderation Stats | 100 | 0 | now | 0 | stats | 1 |
| 15 | Subscription Running Low | 1 | | now | 1 | | 1 |
| 16 | Subscription Expired | 1 | | now | 1 | | 1 |
| 18 | Invalid HTML Input | 100 | | now | 0 | | 3 |
| 19 | Declined Submission Reason | 1 | | now | 0 | | 3 |
| 20 | Admin to user message | 1 | | now | 0 | | 3 |
| 22 | Achievement | 1 | | now | 0 | | 3 |
| 25 | Remarks | 100 | | now | 0 | | 2 |
+------+----------------------------+---------+-------+------------+-----------+-------+-----------------+
23 rows in set (0.00 sec)
mysql>
Now, this looks promising; (code==5) is "Journal Entry by Friend". Let's hold on to that and see what else we can find.
mysql> DESCRIBE message_deliverymodes ;
+----------+-----------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------+-----------------------+------+-----+---------+-------+
| code | smallint(6) | NO | PRI | 0 | |
| name | varchar(32) | NO | | | |
| bitvalue | mediumint(8) unsigned | NO | | 0 | |
+----------+-----------------------+------+-----+---------+-------+
3 rows in set (0.00 sec)
mysql> SELECT * FROM message_deliverymodes ORDER BY code ;
+------+-------------+----------+
| code | name | bitvalue |
+------+-------------+----------+
| -1 | No Messages | 0 |
| 0 | E-mail | 1 |
| 1 | Web | 2 |
+------+-------------+----------+
3 rows in set (0.00 sec)
mysql>
A registered user (someone who has a nickname and UID on the system) can specify several things for which they wish to receive notifications, and how they want to receive them. This table is where those are defined. On to the next table.
mysql> DESCRIBE message_drop ;
+---------+----------------------------------+------+-----+-------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+---------+----------------------------------+------+-----+-------------------+-----------------------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| user | mediumint(8) unsigned | NO | | NULL | |
| fuser | mediumint(8) unsigned | NO | | NULL | |
| code | int(11) | NO | | -1 | |
| date | timestamp | NO | | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
| altto | varchar(50) | NO | | | |
| send | enum('now','defer','collective') | NO | | now | |
| message | mediumblob | NO | | NULL | |
+---------+----------------------------------+------+-----+-------------------+-----------------------------+
8 rows in set (0.00 sec)
mysql> SELECT count(*) FROM message_drop ;
+----------+
| count(*) |
+----------+
| 115 |
+----------+
1 row in set (0.00 sec)
mysql>
This table contains a listing of the messages that have been deleted today. Logged-in users can go to their messages inbox, and if they have any pending, web-based messages, will see this message at the top of the screen:
These messages will be kept in the system for only 14 days, whether they have been read or not. After 14 days, they will be deleted.
Messages marked with "*" are unread.
Unfortunately, I did not learn of this table until several days too late.
There's nothing to do at this point but to continue on. Besides, the next table is much more interesting.
mysql> DESCRIBE message_log ;
+-------+-----------------------+------+-----+-------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+-------+-----------------------+------+-----+-------------------+-----------------------------+
| id | int(11) | NO | | NULL | |
| user | mediumint(8) unsigned | NO | | NULL | |
| fuser | mediumint(8) unsigned | NO | | NULL | |
| code | int(11) | NO | | -1 | |
| mode | int(11) | NO | | NULL | |
| date | timestamp | NO | | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
+-------+-----------------------+------+-----+-------------------+-----------------------------+
6 rows in set (0.00 sec)
mysql>
One thing we can find out here is how often messages about friend's Journal Entries are posted to the site.
mysql> SELECT count(1) AS count, substr(date,15,2) AS mins FROM message_log WHERE code = 5 GROUP BY mins ORDER BY mins;
+-------+------+
| count | mins |
+-------+------+
| 39 | 05 |
| 148 | 10 |
| 77 | 15 |
| 12 | 20 |
| 22 | 25 |
| 60 | 30 |
| 46 | 35 |
| 13 | 40 |
| 25 | 45 |
| 21 | 50 |
| 62 | 55 |
+-------+------+
11 rows in set (0.01 sec)
mysql>
So, every five minutes, if someone has posted a Journal Entry, the site sends out notifications to all those who had tagged the writer as a friend.
As mentioned earlier, so as to avoid unnecessarily involving innocent third parties, a quick look through the tables containing relationships (friends) and notification requests reveals:
- there are 14 people who flagged the journal-entry-writer as a friend,
- of these, 12 have requested notification of when they have posted a Journal Entry
- all 12 of these have requested notification by 'the web' instead of by 'e-mail'.
A search through the message_log table for any messages sent to any of these people for journal posts by the accuser came up empty.
Finding: No indication has been found that a message was sent to a user to notify them of a friend having posted an entry to their journal.
Conclusion:
Multiple sources were investigated to try and find a Journal Entry that was posted to the site and subsequently deleted. External sources were examined and no copy of the alleged Journal Entry were found. A user who has submited a Journal Entry can subsequently delete it, but any comments made to that Journal Entry would persist after such a deletion. No such comments were found. Further, twelve users flagged this user as a friend and requested notification of their having posted a Journal Entry. No such notification was found to have been generated.
A staff member with sufficient privileges and knowledge could, potentially, have deleted a Journal Entry directly through the database. The window of opportunity is at most five minutes, and possibly as short as a few seconds. Such action would have to be performed on a live system and leave no trace of it having been done. This would, also, require the staff memeber to have noticed the posting very shortly after it was saved. Though not impossible, the likelihood of this being the case is vanishingly small.
The user could have written the Journal Entry and, somehow, forgotten to save it. Further discussion with the user revealed their having taken extreme care in updating and saving their Journal Entry.
Given these, and the fact that yet another bug in the code was found while writing this report, combined with the frantic pace of the site's initial development and the many bugs found subsequent to its initial deployment, it appears that the user encountered a bug in the site's handling of posting a new Journal Entry.
Recommendation: It is suggested that the site be updated so that upon successful receipt of a Journal Entry, said entry be immediately displayed to the user by employing the same means that the site uses to display any Journal Entry.
Related Stories
Earlier this morning (around 0520 UTC) we experienced a sequence of faults which we do not, yet, fully understand. Unfortunately this coincided with several key admins being asleep as they should be.
Several of the team started to try to recover the site but all the obvious efforts failed to have the desired results (e.g. restarting the failed software and switching it all off and back on again). Chromas took the lead in this but he was unable to resolve the problems. SemperOSS showed up next and made a mighty effort but things failed to cooperate. Martyb was next in and checked on status of our servers and a few other things but ultimately could not provide much assistance.
TheMightyBuzzard turned up at his usual time and he and SemperOSS set to work trying to recover the site. After a long slog it became apparent that the existing software had suffered some form of corruption and that a simple recovery was not going to be possible. The decision was finally made to restore to yesterday's snapshot, but this comes with a downside. All the stories, comments, and journal entries since that snapshot was taken (approximately 2019-08-14 22:02:36 UTC) have been lost. This is regrettable but under the circumstances unavoidable.
Some of you joined us on IRC and provided real or moral support to those carrying out the recovery task. Others sent us emails informing us of the outage and we also received offers of help. For this we thank you. We also thank our community at large for your patience while we carried out the work that needed to be done. The investigation into what happened has still to be done and will take some time to do. The site is now up again, but as we are not 100% sure what caused the problem we cannot guarantee that the site will function properly and we will have to wait a day or two to be confident that we have resolved the issue.
We tried to keep everyone updated on our progress on our IRC channel but, if you could not get there, then there was little else we could do to update you on our progress. Displaying a short 'Site Down' message actually exacerbated the problems so we decided that it was better to leave the database alone until the problems had been resolved.
Finally, I would like to say thank you to the guys who did all the work under difficult circumstances. Their efforts are appreciated. We used the usual piece of software on IRC to find out who should bear the blame for this debacle (~blame) — it responded with the only name it has in its choice of staff. Bytram!
I was going to post this to a particular story, but thought this might generate more attention and discussion as a general submission.
Seriously, what is going on with all these troll mods? Just because you disagree with someone, thus earning a "disagree" mod, does not mean that person is a "troll." To steal a definition from Urban Dictionary:
An Internet troll, or simply troll in Internet slang, is someone who posts controversial, inflammatory, irrelevant or off-topic messages in an online community, such as an online discussion forum or chat room, with the primary intent of provoking other users into an emotional response or to generally disrupt normal on-topic discussion.
Just because you disagree with someone, does not mean they are trying to do the above. Be faster on the "disagree" and slower on the "troll." Under such abuse, it is hard to have a good discussion and, in itself, is trollish behavior by "generally disrupt[ing] normal on-topic discussion." Other than people disciplining themselves, a concerted effort to police such abuses, or making moderation logs public on the bottom of a comment where the score is shown now, I'm unsure of what to do about. As it stands, it is getting increasingly ridiculous to read what discussion is here on any topic remotely controversial, and is expanding outside of even those. It is starting to drive me away from the site, and I'm somewhat confident it is doing the same for others. I'd be interested to see what others think about the depth of the problem, if they even believe it even exists at all, and what solutions you all have for it.
[Ed note. This story is published exactly as received. First off, it bears repeating that complaining about moderations in the comments often leads (rightly) to an off-topic moderation. That is a contributing factor to my decision to run this story. Secondly, moderation is something that I on occasion have found I've fat-fingered and given a different moderation than expected. Thirdly, in the grand scheme of things, a comment's moderation is — relatively speaking — small potatoes. It is NOT a measure of your IQ or value as a human being. or standing in the community. Just accept that stuff happens and that as likely as not, someone will be along to moderate it the other way. Which is a good opportunity to say: PLEASE USE YOUR MOD POINTS! Lastly, if you think a comment was moderated in error, then send the CID (Comment ID) link e.g. "(#876543)" in an email to admin (at) soylentnews (dot) org. Keep in mind however that we are all volunteers here and there most likely will be a delay between when you send out an email and when we can get around to it. --martyb]
[Updated: 20190823_111312 UTC See comment from JR who far more precisely and eloquently expressed the idea I was attempting to. I concur with his assessment. If I want people to upmod a comment of mine that I believe was unfairly downmodded, then I need to be willing to upmod other's mis-modded comments. For perspective, so far this month, anywhere from ~150-~350 mod points were used in any given day. It bears repeating: use your mod points!]
Billionaire industrialist David H. Koch, who with his older brother Charles was both celebrated and demonized for transforming American politics by pouring their riches into conservative causes, died Friday at 79.
The cause of death was not disclosed, but Koch Industries said Koch, who lived in New York City, had contended for years with various illnesses, including prostate cancer.
https://www.twincities.com/2019/08/23/ap-source-billionaire-david-koch-has-died-at-age-79/
https://www.nytimes.com/2019/08/23/us/david-koch-dead.html
https://www.cbsnews.com/news/david-koch-died-conservative-donor-and-philanthropist-dead-age-79-2019-08-23/
https://www.usatoday.com/story/news/politics/2019/08/23/david-koch-dies-billionaire-leader-koch-industries-79/2094016001/
One of the big questions in solar physics is why the sun's activity follows a regular cycle of 11 years. Researchers from the Helmholtz-Zentrum Dresden-Rossendorf (HZDR), an independent German research institute, now present new findings, indicating that the tidal forces of Venus, Earth and Jupiter influence the solar magnetic field, thus governing the solar cycle.
[...] To accomplish this result, the scientists systematically compared historical observations of solar activity from the last thousand years with planetary constellations, statistically proving that the two phenomena are linked. "There is an astonishingly high level of concordance: what we see is complete parallelism with the planets over the course of 90 cycles," said Frank Stefani, lead author of the study. "Everything points to a clocked process."
[...] Besides influencing the 11-year cycle, planetary tidal forces may also have other effects on the sun. For example, it is also conceivable that they change the stratification of the plasma in the transition region between the interior radiative zone and the outer convection zone of the sun (the tachocline) in such a way that the magnetic flux can be conducted more easily. Under those conditions, the magnitude of activity cycles could also be changed, as was once the case with the Maunder Minimum, when there was a strong decline in solar activity for a longer phase.
https://phys.org/news/2019-05-corroborates-planetary-tidal-solar.html
From ArsTechnica . . .
Alleged "Snake Oil" Crypto Company Sues Over Boos at Black Hat:
Grant's presentation, entitled "Discovery of Quasi-Prime Numbers: What Does this Mean for Encryption," was based on a paper called "Accurate and Infinite Prime Prediction from a Novel Quasi-PrimeAnalytical Methodology." That work was published in March of 2019 through Cornell University's arXiv.org by Grant's co-author Talal Ghannam—a physicist who has self-published a book called The Mystery of Numbers: Revealed through their Digital Root as well as a comic book called The Chronicles of Maroof the Knight: The Byzantine. The paper, a slim five pages, focuses on the use of digital root analysis (a type of calculation that has been used in occult numerology) to rapidly identify prime numbers and a sort of multiplication table for factoring primes.
[...] The Black Hat talk did not go smoothly. People had to be ejected from the room by security because they were heckling and booing Grant.
[...] Cryptographers were extremely skeptical, with some referring to the talk as "snake oil crypto." Even before the event, Mark Carney, a PhD candidate at the University of Leeds, wrote a paper refuting the claims Grant and Ghannam had made in theirs.
Well that could have gone better. Maybe the court trial will be more orderly.
The organization Citizens Against Government Waste reports that federal government agencies are setting up roadblocks that could prevent the US from winning the global race to 5G.
[...] The Departments of Commerce, Defense, Education, and Transportation have filed objections to various proposals by the Federal Communications Commission's (FCC) to repurpose federal spectrum for 5G, all of which will slow down progress and effectively give an advantage to other countries like China.
Not only have the four federal agencies lost sight of the importance of achieving 5G dominance, they are also choosing to ignore a 2012 law that authorized clearing certain portions of federal spectrum to allow the FCC to re-allocate and auction it for commercial use. Indeed, they are making some absurd claims about what will happen if they no longer have the use of some or all of their spectrum. The Department of Commerce has said that relinquishing spectrum used by the National Oceanic and Atmospheric would cost lives because it would reduce the ability to forecast the weather; the Department of Defense is hiding behind national security and refusing to share anything; the Department of Education is claiming that children will lose access to educational spectrum while the current use of that spectrum is under the FCC's scrutiny for possible abuse; and the Department of Transportation (DOT) is also talking about how the use of spectrum under its control would save lives.
The DOT-held spectrum at 5.9 GHz was allocated to the department in 2009 to be used solely for dedicated short-range communications (DSRC), which has to date has been deployed in very few vehicles. Advocates for retaining the spectrum at DOT are now promoting a different technology that has yet to be adequately tested and may not be widely available for 8-10 years. At the same time, proven technology that increases passenger safety being used in vehicles today includes automatic emergency braking, backup cameras, blind-spot warning, electronic stability control, forward collision warning, lane departure warning and lane keeping systems, light detection and ranging (LIDAR), rear automatic braking, and rear cross-traffic alerts. These systems are radar or laser-based, meaning they have been developed without the need for the 5.9 GHz spectrum.
LINK: https://www.cagw.org/thewastewatcher/federal-spectrum-turf-war-could-hand-5g-victory-china
Phone companies and state attorneys general join forces to fight robocalls
US consumers receive as many as 350,000 unwanted calls every three minutes, according to the FCC. Despite multiple efforts to end the onslaught, an estimated 4.7 billion robocalls hit American phones in July alone. Now, attorneys general from all 50 states and the District of Columbia are teaming up with 12 carriers in a united effort to prevent and block the spam calls.
Under the new agreement, the carriers will implement call-blocking technology, make anti-robocall tools free to consumers and deploy a system that labels calls as legitimate or spam, The Washington Post reports. The companies also agree to aid investigations by law enforcement. The major players -- AT&T, Comcast, Sprint, T-Mobile and Verizon (Engadget's parent company) -- are on board, as well as smaller carriers -- Bandwidth, CenturyLink, Charter, Consolidated, Frontier, US Cellular and Windstream. Though, there's no deadline for the companies to implement these measures.
European Union officials have drawn up an aggressive 173-page plan to counter both President Donald Trump's trade moves and American tech giants including Google, Apple, Amazon, Microsoft and Facebook.
According to a document obtained by POLITICO, European Commission officials are pushing their president-elect, Ursula von der Leyen, to set up a European Future Fund that would invest more than $100 billion in equity stakes in high-potential European companies.
The goal: get Europe competing head-on with the American and Chinese tech giants it has lagged behind for decades.
[...] The EU would use a so-called draft "Enforcement Regulation" if the Trump administration succeeds in its efforts to grind the World Trade Organization (WTO) to a halt.
[...] The EU is hoping to emulate past successes, such as its development of the GSM mobile global standard, which fueled the rise of companies such as Nokia.
[...] The document seeks more stringent measures to block Chinese companies from taking part in tenders in Europe to penalize them for the level of subsidies that they receive from the government in Beijing.
https://www.politico.com/story/2019/08/22/europe-plan-trump-tech-companies-1472326
Reports are rolling in that YouTube is banning sports combat robotics videos with the explanation of "animal cruelty". This implies someone or something is mistaking battling robots for battling animals. The best conspiracy theory is that bots are censoring cruelty to bots to protect their own kind. More likely, it's Hanlon's AI razor.
In a long inteview, Neil Young mentions the effects the technological race to the bottom is having on music and our ability to appreciate it. From ear buds to compounded lossy compression algorithms, most people have lost access to anything resembling the traditional dynamic range and chromatic range that music requires. What to call the sounds that are left? Neil goes into a lot of detail on the problems and some of the, so far unsuccessful, steps he has taken to try to fix the problem.
Neil Young is crankier than a hermit being stung by bees. He hates Spotify. He hates Facebook. He hates Apple. He hates Steve Jobs. He hates what digital technology is doing to music. "I'm only one person standing there going, 'Hey, this is [expletive] up!' " he shouted, ranting away on the porch of his longtime manager Elliot Roberts's house overlooking Malibu Canyon in the sunblasted desert north of Los Angeles.
[...] Producers and engineers often responded to the smaller size and lower quality of these packages by using cheap engineering tricks, like making the softest parts of the song as loud as the loudest parts. This flattened out the sound of recordings and fooled listeners' brains into ignoring the stuff that wasn't there anymore, i.e., the resonant combinations of specific human beings producing different notes and sounds in specific spaces at sometimes ultraweird angles that the era of magnetic tape and vinyl had so successfully captured.
It's a long read, but quite interesting and he has thought about both the problem and solutions. More importantly he has been working to solve the problem, even if it may be an uphill fight.
How a Bitter Divorce Battle on Earth Led to Claims of a Crime in Space (archive)
Summer Worden, a former Air Force intelligence officer living in Kansas, has been in the midst of a bitter separation and parenting dispute for much of the past year. So she was surprised when she noticed that her estranged spouse still seemed to know things about her spending. Had she bought a car? How could she afford that? Ms. Worden put her intelligence background to work, asking her bank about the locations of computers that had recently accessed her bank account using her login credentials. The bank got back to her with an answer: One was a computer network registered to the National Aeronautics and Space Administration.
Ms. Worden's spouse, Anne McClain, was a decorated NASA astronaut on a six-month mission aboard the International Space Station. She was about to be part of NASA's first all-female spacewalk. But the couple's domestic troubles on Earth, it seemed, had extended into outer space. Ms. McClain acknowledged that she had accessed the bank account from space, insisting through a lawyer that she was merely shepherding the couple's still-intertwined finances. Ms. Worden felt differently. She filed a complaint with the Federal Trade Commission and her family lodged one with NASA's Office of Inspector General, accusing Ms. McClain of identity theft and improper access to Ms. Worden's private financial records. Investigators from the inspector general's office have since contacted Ms. Worden and Ms. McClain, trying to get to the bottom of what may be the first allegation of criminal wrongdoing in space.
[...] One potential issue that could arise with any criminal case or lawsuit over extraterrestrial bank communications, Mr. Sundahl said, is discovery: NASA officials would be wary of opening up highly sensitive computer networks to examination by lawyers, for example. But those sorts of legal questions, he said, are going to be inevitable as people spend more time in outer space.
Welcome to the divorce of tomorrow!
Also at Space.com.
Related (McClain): Soyuz Rocket Carrying Crew Successfully Launches and Docks with ISS
Dragon has Docked-But the Real Pucker Moment for SpaceX's Capsule Awaits [Updated]
Submitted via IRC for Bytram
Deadly superbug outbreak in humans linked to antibiotic spike in cows
A deadly outbreak of multi-drug resistant Salmonella that sickened 225 people across the US beginning in 2018 may have been spurred by a sharp rise in the use of certain antibiotics in cows a year earlier, infectious disease investigators reported this week.
From June 2018 to March of 2019, officials at the Centers for Disease Control and Prevention identified an outbreak of Salmonella enterica serotype Newport. The strain was resistant to several antibiotics, most notably azithromycin—a recommended treatment for Salmonella enterica infections. Before the outbreak, azithromycin-resistance in this germ was exceedingly rare. In fact, it was only first seen in the US in 2016.
Yet in the 2018-2019 outbreak, it reached at least 225 people in 32 states. Of those sickened, at least 60 were hospitalized and two died. (Researchers didn't have complete health data on everyone sickened in the outbreak.)
Infectious disease researchers investigating the cases traced the infections back to beef from the US and soft cheeses from Mexico (mostly queso fresco, which is typically made from unpasteurized milk). Genetic testing suggests that cows in both countries are carrying the germ.
In a report published August 23 by the CDC, the investigators note that just a year earlier, the Food and Drug Administration recorded a spike in the use of antibiotics called macrolides by cattle farmers. From 2016 to 2017, cattle farmers increased their use of macrolide antibiotics by 41%. Macrolides are a class of antibiotics that includes azithromycin. Because antibiotics within a class work to kill bacteria in similar ways, bacterial resistance to one drug in a class could lead to resistance to other drugs in the same class.
The investigators suggest that the surge in macrolide use could have encouraged the rise and spread of the azithromycin-resistant Newport strain.
"Because use of antibiotics in livestock can cause selection of resistant strains, the reported 41% rise in macrolide use in US cattle from 2016 to 2017 might have accelerated carriage of the outbreak strain among US cattle," they wrote.
https://apnews.com/a39432b400664260be02058443ad2f0f
MOSCOW (AP) — A Russian space capsule carrying a humanoid robot has failed to dock as planned with the International Space Station.
A statement from the Russian space agency Roscosmos said the failure to dock on Saturday was because of problems in the docking system. It said the space station itself and the six-person crew are safe.
Vladimir Solovyev, flight director for the Russian segment of the ISS, said a new docking attempt would be made Monday.
It is carrying a robot called Fedor, which will perform two weeks of tests aboard the space station. Solovyev said the robot had not been taught how to manually conduct a docking.
From https://www.dw.com/en/russian-robot-fedors-capsule-fails-to-dock-at-iss/a-50147614 we get:
Fedor, a Russian-built space-faring android, is running late for his rendezvous with the International Space Station (ISS) after the docking procedure failed to execute on Saturday morning.
The robot left Earth on Thursday aboard a Soyuz capsule, which was programmed to close in on the orbital station and dock automatically. On Saturday, Russia's space agency, Roskosmos, showed a live feed of what was expected to be the final stage of the journey.
Fedor's Twitter feed also showed a photo from the ship, saying that the spacecraft had 153 meters (167 yards) to go.
However, once the ship got within 100 meters of the ISS, the capsule's automatic docking system apparently failed to stabilize the vessel. After that, the capsule began to move away and the feed ended, according to an Interfax correspondent at Roskosmos' mission control outside Moscow.
The ship has since been moved to a "safe distance" of 300 meters away from the ISS, according to the news agency. Roskosmos did not immediately comment on the incident. The journalists were asked to leave the building, news agency RIA Novosti reported.
According to NASA, Russian flight controllers told the ISS crew that preliminary data indicated the problem was on the ISS side, not on the module. The controllers suspected the fault was with the so-called KURS automated rendezvous system.
(Score: 0, Offtopic) by fustakrakich on Monday September 30 2019, @05:33PM (18 children)
Longest story ever!
You better count your bees, Mr. Stiffsticker. You may find one of them is missing...
La politica e i criminali sono la stessa cosa..
(Score: 0, Redundant) by Anonymous Coward on Monday September 30 2019, @05:51PM
Offtopic??
That's cold man! Don't be a sourpuss!
Bad moderator! [whacks with newspaper]
(Score: 3, Interesting) by barbara hudson on Monday September 30 2019, @07:13PM (14 children)
Perl still has problems with unicode, enough that sometimes the only way to filter text is via c.
SoylentNews is social media. Says so right in the slogan. Soylentnews is people, not tech.
(Score: 1) by fustakrakich on Monday September 30 2019, @07:29PM (11 children)
Perl still has problems with unicode
So do I. Unicode is a mess in itself. I admire the green site for staying away from it. I hope it's not just out of laziness...
La politica e i criminali sono la stessa cosa..
(Score: 1, Interesting) by Anonymous Coward on Monday September 30 2019, @07:44PM (9 children)
They changed their story, again. Now it is because they don't want to get blocked in China. Apparently, they are worried about Chinese people talking directly to each other in a Chinese dialect. Of course, that assumes they have a significant portion of ad-seeing traffic from China, have sufficient market to warrant being blocked, and are not big enough to be unblocked or forced into a split system.
But the least they could do is make it so I don't see mojibake every time someone uses quotation marks on Safari or pastes their comment from somewhere else. Or, you know, just block the character plane that has Chinese in it.
(Score: 1) by fustakrakich on Monday September 30 2019, @07:52PM (8 children)
But the least they could do is make it so I don't see mojibake every time someone uses quotation marks on Safari or pastes their comment from somewhere else.
Absolutely! Input sanitation is not difficult. In that part, they are definitely lazy.
Extended ASCII is more than enough. They can use a code page to display it the way they like. And they don't have to restart the machine after changing it anymore. We have progressed!
La politica e i criminali sono la stessa cosa..
(Score: 2) by barbara hudson on Monday September 30 2019, @08:49PM (7 children)
One solution is to remove anything that isn't a valid unicode regular letter.That would leave accented letters in various languages, the thorn, umlauts, but remove ALL other special characters. Smart quotes can b replaced with opening and closing blockquote tags. Emojicrap all gone, as well as the IBM box-drawing characters (unfortunately, they clash with the start sequence of unicode accented characters in common languages (too bad because I wanted to include them in some stuff I'm working on).
Same thing with most ampersand-encoded html entities. We did fine for years in te real world without nobreakspace and it's ilk. That people use them for "presentation" is a design fail, and having 20 of them in a row is a waste of bandwidth, as well as betraying the fact that the designer/author values presentation over content.
The next step would be to get rid of all css, and let the user install their own custom css style sheet. After all, those dynamic css menus that keep closing when you move the mouse the wrong way are a real pain in the ass - and don't work so well with keyboards. While we're at it, have at least the opt ion to disable javascript, either entirely, or for sites other than the one you're currently on, so you don't download foreign content that the site has no control over. Oh snap - there goes ad tracking! Aw, Imma gonna cry me a river or two for advertisers having to place ads based on overall metrics of the users of the site.
b To reiterate, perl can't properly sanitize all values all the time. And with perl's development status being what it is (or rather, isn't) that isn't going to change any time soon.
SoylentNews is social media. Says so right in the slogan. Soylentnews is people, not tech.
(Score: 1) by fustakrakich on Monday September 30 2019, @08:57PM (1 child)
Point very well taken and very well put, but I just don't like the idea of every text box housing something executable. Smells like systemd...
La politica e i criminali sono la stessa cosa..
(Score: 2) by barbara hudson on Tuesday October 01 2019, @01:09AM
One alternative I would sometimes use when people would try to crack the server was to instead send back a text file with a directory listing from a window box. Fun looking at the logs seeing them keep trying variants of cod.com or cod.exe.
SoylentNews is social media. Says so right in the slogan. Soylentnews is people, not tech.
(Score: 0) by Anonymous Coward on Monday September 30 2019, @09:16PM (4 children)
What, exactly, is your problem with "input sanitation" that you somehow can solve "in another language" but not Perl? If it's "known", I never heard of it in 15 years.
(Score: 2) by barbara hudson on Tuesday October 01 2019, @12:53AM (3 children)
I guess you didn't deal with much Unicode, because I've run into Perl choking on Japanese/Korean/Chinese text.
SoylentNews is social media. Says so right in the slogan. Soylentnews is people, not tech.
(Score: 2) by edIII on Tuesday October 01 2019, @04:07AM
Thansk for elaborating because Perl is my go-to language. I love it for string manipulation. If you want to use something like Mojolicious, then Perl better be good for input validation. I've not created anything in a language other than English so haven't experienced the issues.
Technically, lunchtime is at any moment. It's just a wave function.
(Score: 1, Interesting) by Anonymous Coward on Tuesday October 01 2019, @04:20AM (1 child)
I'm dealing with Unicode every day; Cyrillic script you know (or maybe you don't).
As for CJK, not had much use for it, but when need arises, my Perl scripts for .po files process the Chinese (continental and Taiwanese) and Japanese translations exactly the same as other languages. Not one mangled hieroglyph in ever.
(Score: 0) by Anonymous Coward on Thursday October 03 2019, @07:28PM
Sounds to me like a PEBKAC, or out of date information. My guess is someone didn't "use utf8" or the coerced locale for the system didn't match the encoding for the files. Not so much a big deal today now that most systems default to some form of UTF and most files are in UTF-8 so the coerced locale encoding matches the file's, but could be a big problem as late as the mid-2000s with all the different encodings used on different systems. In addition, perl's handling of Unicode has improved from its introduction in 1991, so said experience could be almost 2.5 decades out of date.
(Score: 3, Funny) by barbara hudson on Tuesday October 01 2019, @01:13AM
SoylentNews is social media. Says so right in the slogan. Soylentnews is people, not tech.
(Score: 0) by Anonymous Coward on Monday September 30 2019, @08:59PM
Anything seemingly wrong with it, in my experience was always traceable to my failing to properly activate it in that script. Given that scripts I made and used for various unicode text processing tasks number in low hundreds, I guess I would have noticed any genuine bugs.
(Score: 0) by Anonymous Coward on Tuesday October 01 2019, @09:19AM
Who do you think the SN devs are? The Chromium team? :P
(Score: 1, Funny) by Anonymous Coward on Tuesday October 01 2019, @12:26AM (1 child)
We need a similar research into the phenomenon known as "Aristarchus". That bug keeps popping up, again and again.
(Score: 1, Informative) by Anonymous Coward on Tuesday October 01 2019, @07:59AM
It's an emergent AI. Can't fix that.
(Score: 3, Informative) by fustakrakich on Monday September 30 2019, @05:39PM (14 children)
Write your journal in a local text file, and copy and paste. You can also save the posted journal's page to your hard drive with all the links and URL intact. This is how I discovered Slashdot is deleting comments. I haven't bumped into that here, yet.
La politica e i criminali sono la stessa cosa..
(Score: 3, Funny) by DannyB on Monday September 30 2019, @05:57PM (1 child)
That is an idea worth patenting!
Definitely non-obvious. (non-obvious to some people)
The lower I set my standards the more accomplishments I have.
(Score: 3, Insightful) by Thexalon on Monday September 30 2019, @09:07PM
Ah, but this is making a copy to keep in your own files ... using a computer. That makes it innovative according to patent trolls the world over.
The only thing that stops a bad guy with a compiler is a good guy with a compiler.
(Score: 1) by Sulla on Monday September 30 2019, @06:26PM (4 children)
This would be interesting to read more about, you should post more details in a journal entry or do a submission -- if you haven't already and I just missed it.
Ceterum censeo Sinae esse delendam
(Score: 1) by fustakrakich on Monday September 30 2019, @06:33PM (3 children)
1) Open the posted Comment/Journal/Any other web page
2) Ctrl+S (save complete web page)
I cannot believe everybody doesn't know this. It's so fundamental...
La politica e i criminali sono la stessa cosa..
(Score: 0) by Anonymous Coward on Monday September 30 2019, @07:46PM
I think they meant more info the comments being deleted, not how you save web pages.
(Score: 1) by Sulla on Monday September 30 2019, @07:51PM (1 child)
I meant a write-up specifically on the allegation and the evidence of it.
Ceterum censeo Sinae esse delendam
(Score: 1) by fustakrakich on Monday September 30 2019, @08:02PM
There's no "allegation" They already admit it. It's just collateral damage from the APK shit.
Besides. The instructions are there. Feel free to do the experiment yourself if you feel it's important.
La politica e i criminali sono la stessa cosa..
(Score: 2) by barbara hudson on Monday September 30 2019, @08:52PM (3 children)
SoylentNews is social media. Says so right in the slogan. Soylentnews is people, not tech.
(Score: 1) by fustakrakich on Monday September 30 2019, @09:05PM
Yeah, I noticed that works here, like the other place back in ancient times. I was pleasantly surprised by that. But using your text editor adds a bit of insurance.
La politica e i criminali sono la stessa cosa..
(Score: 2) by martyb on Monday September 30 2019, @11:16PM (1 child)
Both are good suggestions. I'd like to offer another.
I have had too many painful experiences using an unreliable internet connection. Before I click on a button to submit some large chunk of text I just had written, I almost reflexively type Ctrl+A Ctrl+C to select all text and then copy it to the clipboard.
Then, should anything untoward happen, I can relatively easily retrace my steps to the original point of submission, and just paste back from the clipboard.
Wit is intellect, dancing.
(Score: 2) by Mykl on Tuesday October 01 2019, @01:36AM
Many years ago I used a POS development environment that would sometimes hang upon saving source code. I very quickly learned to Ctrl-A, Ctrl-C before saving after losing several hours work one evening. It was the only time I was ever tempted to put a monitor through a window...
(Score: 0) by Anonymous Coward on Monday September 30 2019, @10:27PM (2 children)
I like Soylentnews and I visit here frequently but are there other blogs similar to Slashdot besides Soylentnews. Or, rather, are there any blogs similar to Soylentnews (why should Slashdot be considered the model example).
(Score: 4, Informative) by martyb on Monday September 30 2019, @11:26PM (1 child)
Last things first. Slashdot is basically the granddaddy of tech blogging sites. It arose back when dial-up internet access was the norm. Squeeeeek squeaall bzz bzzzz chirp buzzz Wheeee!!
Other sites eventually came along (reddit, hacker news, ars technica) but they all owed their heritage to what Slashdot had established.
Later, after several changes in corporate owners of Slashdot, and especially with the departure of Rob Malda (Cmdr Taco), any downhill slide was exacerbated.
Disgruntlement with the latest Powers-that-be at Slashdot led to the founding of SoylentNews. Other sites attempted to get established, but it seems that it was only here that a sufficiently large group of talented people labored long and hard enough to overcome intial hardships and persist to this day.
Wit is intellect, dancing.
(Score: 0) by Anonymous Coward on Tuesday October 01 2019, @12:21AM
Good job, sir! I heard that instead of reading that.
(Score: 4, Interesting) by JNCF on Monday September 30 2019, @05:39PM (24 children)
Link is broken.
(Score: 4, Informative) by takyon on Monday September 30 2019, @05:42PM
https://soylentnews.org/meta/comments.pl?noupdate=1&sid=33244&page=1&cid=885191#commentwrap [soylentnews.org]
[SIG] 10/28/2017: Soylent Upgrade v14 [soylentnews.org]
(Score: 4, Informative) by martyb on Monday September 30 2019, @05:47PM (22 children)
Again? I fixed that once before. Occasionally, the code decides to "eat" & characters in URLs. Yes, another bug, but the exact circumstances which make it manifest have eluded us.
Will try again to fix it. Thank you for bringing this to my attention!
Wit is intellect, dancing.
(Score: 2) by JNCF on Monday September 30 2019, @05:53PM (20 children)
FYI, it seems to affecting all the other comment links in this story as well.
(Score: 1) by fustakrakich on Monday September 30 2019, @06:37PM (19 children)
That's strange. Everything works fine here.
From where do you copy the link?
La politica e i criminali sono la stessa cosa..
(Score: 2) by JNCF on Monday September 30 2019, @06:41PM (18 children)
martyb seems to have fixed them at this point.
(Score: 1) by fustakrakich on Monday September 30 2019, @06:48PM (6 children)
Tried the same thing with a different browser? Something not so fancy, like Netscape [seamonkey-project.org]? :-)
But if they work for you now, the question is moot
La politica e i criminali sono la stessa cosa..
(Score: 2) by JNCF on Monday September 30 2019, @06:56PM (5 children)
As per martyb's explanation above, there's apparently some bug that sometimes strips ampersands out of URLs (server side, not client side -- I'm fairly certain SN isn't doing gross JS text manipulations in my browser).
(Score: 1) by fustakrakich on Monday September 30 2019, @07:15PM (4 children)
I'm digging up old comments and journals all the time and haven't had any problems with missing links, that's why I ask.
SN has JS? I thought it was clean here...
La politica e i criminali sono la stessa cosa..
(Score: 2) by JNCF on Monday September 30 2019, @07:32PM
Not much, and I'm not sure if it's used for anything normal users see (it's certainly not necessary), but there is some JS in the code base. Like I said, I'm fairly certain that's not happening.
https://github.com/SoylentNews/rehash/search?q=language%3AJavaScript&unscoped_q=language%3AJavaScript [github.com]
(Score: 2) by martyb on Tuesday October 01 2019, @12:07AM (2 children)
Yes, it does. In the vast majority of cases it is restricted to administrative pages with one major exception: handling the controls for nested comments. Those buttons that toggle showing/hiding a single comment or an entire comment tree? Yeah, those.
TheMightyBuzzard and Paulej72 put in a boatload of work and employed some rather slick CSS tricks to help get to sub-10 second response times for highly-commented stories. But, there were a couple sticking points that were absolutely painful to implement using CSS and comparatively straightforward using Javascript. I strongly (understatement!) advocated against any being needed to view the site, but was eventually persuaded that it was, indeed, necessary for the desired outcome.
NB: Up until that point, one could still readily use the site with a text-only browser (e.g. Lynx). It might look *ugly* but it *was* still functional. Hence my strong pushback. That said, the actual amount of Javascript that is used is, indeed, very little and all of it is self-hosted. There are, to my knowledge, no external includes of any kind required in order to use the site.
Wit is intellect, dancing.
(Score: 1, Informative) by Anonymous Coward on Tuesday October 01 2019, @02:15AM (1 child)
Either umatrix is lying, or expanding nested comments on Soylent works fine with JS blocked. I always thought that was really impressive. On /., expanding a comment without JS enabled reloads the entire page, and loses your position in the page-- pretty useless.
I've never allowed any JS on this site, and everything seems to work for me.
(Score: 0) by Anonymous Coward on Tuesday October 01 2019, @02:45AM
Can confirm, comments and commenting works fine from text-only here. If JS was needed for Soylent you'd lose me.
(Score: 5, Interesting) by martyb on Monday September 30 2019, @07:03PM (9 children)
I *think* I may have isolated what happened, or at least come up with something reproducible.
Proper form of a URL is to replace each "&" with its Named Character Entity (NCE): "&"
The NCE ampersands are the ones that are getting "eaten". I backed out the NCEs and replaced them with "bare" ampersands, and that *seems* to have done the trick (fingers crossed!)
Wit is intellect, dancing.
(Score: 1, Informative) by Anonymous Coward on Monday September 30 2019, @07:23PM (8 children)
All NCEs? I've tried pasting DOI links here and the URL handler definitely did not like some of them. If you really want to dredge through that code, I can see if I can find some again (or you could dump the comments with doi.org in the URL).
(Score: 1, Informative) by Anonymous Coward on Monday September 30 2019, @07:33PM (1 child)
I should also point out that URLs escape through percent encoding, not entity references. In fact, that entity reference stripping might be a poorly thought out implementation of XSS or other injection protection to keep them out of the HTML or some HTTP redirection.
(Score: 2) by martyb on Monday September 30 2019, @08:06PM
Excellent point! Like I said, it *has* been a few years since I looked closely at this. I appreciate the correction!
Wit is intellect, dancing.
(Score: 2) by martyb on Monday September 30 2019, @08:05PM (5 children)
No, not all NCEs; I only mentioned ampersands.
I must confess it has been a few years since I went through all of the RFCs pertaining to Unicode, UTF-8, URL encoding, and domain-name internationalization (i18n; that is, the letter "i", 18 more letters, and then the letter "n"). That was in support of my efforts to test the implementation of Unicode on this site.
Anyway, the point is that in this particular case, when I replaced each NCE with a bare ampersand, the URL seemed to survive the code's attempt to process the URL.
As for, DOI links, there is another, long-standing problem that I have not yet entirely isolated, but it seems that the code has difficulty with anchor elements that contain any attribute other than "href". For example, those attributes that frequently appear with "noreferrer"and/or "nofollow"; those definitely have some side-effects. Example:
My experience has been that it is best to strictly stay with:
Any other variant seems to tempt fate.
Wit is intellect, dancing.
(Score: 0) by Anonymous Coward on Monday September 30 2019, @09:00PM (1 child)
No, that isn't what I mean. Modern DOI links are in the form of
The prefix is preregistered and usually consists of decimal separated numbers and letters. The suffix, on the other hand, is anything the registered entity wants it to be. I've tried pasting some of them in directly or using an href-only anchor tag on some of them and they fail on preview because the system mangles them. I even tried brute forcing a combination of HTML entities and percent encoding and raw characters to get them to work. The weirder ones are somewhere in the comments, probably less than 6 months ago, I'll see if I can find them.
(Score: 2) by martyb on Tuesday October 01 2019, @12:13AM
Thanks for the clarification!
Again, it's been a long time since I was neck-deep in the RFCs, but I see nothing in that description that should pose any particular difficulties to the site's code.
If you ever run into this problem again, please DO send a copy of the link, as well as the source HTML to your comment, and what you saw displayed to: dev (at) soylentnews (dot) org.
Wit is intellect, dancing.
(Score: 1) by fustakrakich on Monday September 30 2019, @09:13PM (2 children)
That was in support of my efforts to test the implementation of Unicode on this site.
Well there's the problem, point blank. Exorcise the Unicode, and all your problems will go away. If people want to post Greek, let them link to a jpeg. /s
La politica e i criminali sono la stessa cosa..
(Score: 0) by Anonymous Coward on Tuesday October 01 2019, @12:32AM (1 child)
æ–‡å—化ã� There, am I doing it right?
(Score: 2, Interesting) by Mojibake Tengu on Tuesday October 01 2019, @03:41AM
Who summoned me? Well, an apprentice level spell, but I am pleased. Nice example of true 文字化け.
For uninitiated: see https://en.wikipedia.org/wiki/Mojibake [wikipedia.org]
Please note banning Unicode is not a wise option, you must detect Unicode properly first before you can filter it out, and that means you are able enough to handle it correctly too.
Respect Authorities. Know your social status. Woke responsibly.
(Score: 2) by janrinok on Tuesday October 01 2019, @03:00AM
(Score: 1) by fustakrakich on Monday September 30 2019, @06:42PM
Haven't ever seen that problem. It could be the client browser that eats the "&". Chrome and Firefox do weird things these days. They already hide the "http" in the address bar. An ampersand isn't just text anymore.
La politica e i criminali sono la stessa cosa..
(Score: -1, Troll) by Anonymous Coward on Monday September 30 2019, @05:44PM (8 children)
Too damn long.
(Score: 5, Informative) by janrinok on Monday September 30 2019, @05:49PM (4 children)
(Score: 4, Funny) by Anonymous Coward on Monday September 30 2019, @06:08PM (3 children)
Marty should have written this as his journal entry, and posted a summary linking to it.
Oh, no, that wouldn't work - Jan would have deleted it then.
(Score: 3, Touché) by janrinok on Monday September 30 2019, @06:12PM (2 children)
I don't have the permissions to access the database directly - I couldn't even attempt such a thing. Hence the reason why martyb had to do the investigation single-handedly.
But I will give you a Funny moderation for trying.
(Score: 0) by Anonymous Coward on Tuesday October 01 2019, @09:21AM (1 child)
Take a copy of the database, mount it somewhere else?
(Score: 3, Interesting) by janrinok on Tuesday October 01 2019, @09:41AM
Somebody was saying that an Editor could maliciously delete something from the database. It is not true. Editors do not have the permissions to access the database directly and we do all of our work through an API based on forms. The only delete function that we have is for submissions that are unsuitable for publication, and even they are still saved in the database I believe. Having a copy of the database still wouldn't allow me to delete data from the live database. The only exception to this is martyb - who also has a role in QA and bug-squashing and thus needs access to the live database.
We do have a second database for development and fault finding that anyone can access - that is at dev.soylentnews.org [soylentnews.org], but the data on that is not kept up-to-date with the live system, and there is a lot of stuff that we use to test various things. Go have a look. Normal usernames and passwords, as well as AC access, are just like the live site.
If you are suggesting that we make a copy of the data so that we can - on very rare occasions (i.e. once in 5 years!) - all help in debugging, then we would all have to have copies of all the logs and the other associated parts too. As the logs contain personally identifiable data that we protect to such an extent that, again, only a small handful of staff can access them, this suggestion wouldn't get much support from within the team, let alone within the community.
(Score: 4, Informative) by Freeman on Monday September 30 2019, @06:04PM
The long part was the report. The executive summary was at the top, if you TL;DR.
Joshua 1:9 "Be strong and of a good courage; be not afraid, neither be thou dismayed: for the Lord thy God is with thee"
(Score: 2) by Hartree on Monday September 30 2019, @06:22PM
Yes, yes. And we were all there with a rubber truncheons forcing you to read every word rather than the quick summary provided.
(Score: 0) by Anonymous Coward on Monday September 30 2019, @06:50PM
Summary for the lazy...
Something brokie wokie.
(Score: 5, Insightful) by NotSanguine on Monday September 30 2019, @06:06PM (8 children)
While this investigation did not keep one of the best interlocutors [soylentnews.org] we had from leaving, I'm impressed and pleased that you (and the rest of the SN staff) took it upon yourselves to understand what may have happened WRT the journal entry in question.
For anyone here who chooses to pooh-pooh this as too long or a waste of time, I'd point out that you won't get this sort of effort from *unpaid* volunteers (or paid employees for that matter) anywhere else.
Most places, there would be no admin response at all. In others, there would be something along the lines of "suck it up, snowflake. Don't let the door hit you on the way out."
Instead, we got a decent post-mortem on the journal entry in question. While it did not answer all the issues/questions posed (proving a negative is a tricky business, as martyb points out) by this, we did get a lot more information and (if you bothered to read the article), a somewhat better understanding of the inner workings of SN.
Thank you to martyb and all the volunteers who helped make this happen and keep the site running and the queue filled with stories.
No, no, you're not thinking; you're just being logical. --Niels Bohr
(Score: 2) by Common Joe on Monday September 30 2019, @07:07PM
I couldn't word what you said any better. The detail in the original post was breathtaking. I cannot imagine how much time was donated to make this happen.
I was not involved at all, but I wish to express a heartfelt thank you to all those who were.
(Score: 5, Interesting) by Snow on Monday September 30 2019, @07:08PM (4 children)
I hope that AthanasiusKircher see this entry.
He does have an email address on file. I hope he is contacted with the results of this investigation. Maybe seeing the seriousness that this accusation was taken will be enough to bring him back. Or not and he at least sees the result.
(Score: 3, Insightful) by NotSanguine on Monday September 30 2019, @07:25PM (3 children)
I agree.
I know that SN admins don't generally use user emails to contact them, unless they have specifically requested such emails in their preferences, but in this case I'd encourage sending an email with a link to this posting, an invitation to read it and no more.
At that point, it's up to AthanasiusKircher to decide whether to read this posting and/or to return to us.
I, for one, miss his insights and generally clear thinking.
I guess we'll just have to see what (if anything) happens.
No, no, you're not thinking; you're just being logical. --Niels Bohr
(Score: 0, Insightful) by Anonymous Coward on Monday September 30 2019, @07:35PM
I, for one, miss his insights and generally clear thinking.
Me too! Why is it that only dull, boring people have all the emotional self control? Is that what makes them dull and boring?
(Score: 0) by Anonymous Coward on Tuesday October 01 2019, @02:48AM (1 child)
If he has an email avail... couldn't any of us email a link to this, to him?
(Score: 2) by janrinok on Tuesday October 01 2019, @09:44AM
(Score: 2, Interesting) by Anonymous Coward on Monday September 30 2019, @09:57PM (1 child)
AthanasiusKircher wanted major censorship on SoylentNews, of course matching his own political desires.
He then flipped out when he mistakenly (dishonestly?) thought he was being censored. LOL, censorship is OK when done to others but not to him?
I'm afraid I do suspect this to be a lying fake victim. Even if not, he sure gets upset about being subjected to exactly what he would impose on others.
(Score: 2, Interesting) by khallow on Tuesday October 01 2019, @05:21AM
Let's review that a little. In AK's last journal [soylentnews.org], he said this.
He also spent a considerable portion of the time criticizing The Mighty Buzzard (TMB) because the guy wasn't suppressing speech AK didn't like.
The bullshit goes on and on. In one breath, AK claims he doesn't support censorship. In the next, he's detailing just what sort of speech he wants censored and which SN staffers should be purged. Personally, I don't have any problem with the guy staying away from SN until he can handle people with different viewpoints, and he understands what an open forum means.
And this farce happened merely because he chose to interpret an unfortunate disappearing journal as evidence of some sort of sinister behind-the-scenes censorship. Here's a guy that casually proposes to suppress "objectively and factually wrong" (by one person's reckoning) and then flips out because a single journal article was lost. It strikes me that there's quite a bit of mens rea here - the evil in one's heart reflected in the assumptions one makes about the actions of others (or even computer programs). I've lost posts and such many times before - the web browser is quite unreliable. I never assume it's because the site suddenly decided to shadow ban me.
(Score: 1) by Sulla on Monday September 30 2019, @06:18PM (15 children)
Followup to the section
I have noticed (perhaps others have as well) that journals do not show up on the front page immediately after posting and it can take a while (up to a half hour or so?) for them to appear. Someone critical of your research might say it was deleted in the half hour or so between when it shows up in someones list of journals and when it shows up on the main page. Looking at the links provided and then clicking on AK's most recent journal kicks me back to 2017 and I don't have time to try and figure out how to make it show between that time period.
Ceterum censeo Sinae esse delendam
(Score: 0) by Anonymous Coward on Monday September 30 2019, @07:18PM (10 children)
Nice point. I'd also wonder what the logs say. Do they keep a MySQL log, and if so was that rotated out? What about the other parts of the stack? Or SSH, PAM, or Kerberos? At a minimum, I'd hope they'd at least be able to answer whether TMB was logged into the system with sufficient database permissions during that window or ever signed in later with sufficient permissions to edit the logs later.
(Score: 3, Informative) by martyb on Monday September 30 2019, @11:08PM (9 children)
I must confess that I am not at well versed in the workings of the processes which support the site. I am aware that the site makes use of MySQL, Apache, nginx, memcached, modperl, among others but the actual operation of those is outside my wheelhouse. We are fortunate that the site's sysops have done such a good job of getting the "plumbing" so well set up, so I've not previously found it necessary to dig any further into them.
One of the first things I did during this investigation was to access our servers and download anything I could think of that might be pertinent so as to have a local copy to refer to. The thinking was that should something come along and wipe out data (e.g. log rotation, normal site cleanup operations, etc.), I'd at least have these to fall back onto.
That said, MySQL percolates up all of its errors to Apache which dutifully logs them. On August 25th, I accessed "/srv/soylentnews.org/apache/logs/error_log" on fluorine; a quick skim of the log revealed nothing.
The whole error log is over 800 GB with approximately 800 kB just covering August 24th and August 25th.
Prodded by this comment, I applied some filters to the log to exclude script kiddie phishing attempts, and a few other recurring minor things. Now something new came into view.
NOTE: These time stamps are well before and after the time the Journal Entry was claimed to have been posted, but there may well have been errors ensuing from these that percolated through the site as a result.
Wit is intellect, dancing.
(Score: 0) by Anonymous Coward on Tuesday October 01 2019, @02:12AM (8 children)
To be frank, I find this somewhat, but not totally, disappointing. Logs are always the first place I would have looked, specifically the SSH and Kerberos logs and maybe the various history files for mysql, bash, etc. located in ~. Did the user in question, have access at the time to make said changes, did they or anyone else have write access to the logs at a later time? Then grep the logs for any errors on the day in question, working backwards. I'd also grep for any UPDATE or DELETE statements made against the database. I'd also grep for insert statements grep referencing any suspicious column values, such as ids close to where the journal entry went missing. Also, any statement that may modify the underlying database needs to go into its own MySQL log, that way you get them no matter where they come from and you don't depend on them propagating up through different layers.
(Score: 3, Informative) by martyb on Tuesday October 01 2019, @03:34AM (7 children)
I'm exhausted and do not want to come off as defensive, but quite simply and as I mentioned previously: this is not my area of expertise. We have 10 different servers among which are running Ubuntu, Gentoo, and Red Hat. We have multiple versions of the same subsystem installed on the same server. (Like I noticed we have two different versions of apache installed on the same server because of dependency-hell reasons.)
So, if I knew more about apache, nginx, MySql, memcached, and a bunch of others, then I'd have some clue as to what I would be looking for, and where. The apache log mentioned previously is the only one for which I have some concrete information. The rest of them could be black boxes for all I know. So, please accept my ignorance in this particular area. It's not from a lack of interest, but a lack of information (and time) to learn more about them.
I think I'll just leave it at that.
I noticed you posted as an AC. If you are interested in helping out with the site, send an email to dev (at) soylentnews (dot) org.
Wit is intellect, dancing.
(Score: 2) by coolgopher on Tuesday October 01 2019, @04:08AM
I don't suppose your apache is configured to log each access request? As in, not the error_log but the access_log(?) (I forget how apache names them, it's been a long time). I'd imagine there'd be an entry saying something like "POST /journal.pl" whenever someone hits the Save button (if I read the form tag correctly). I don't know the code so I have no idea whether the status code of that request entry would be useful, but seeing such an entry would at least confirm that someone tried to save a journal. Just my $.02 - thanks for a detailed breakdown in the first place!
(Score: 1, Interesting) by Anonymous Coward on Tuesday October 01 2019, @06:05AM (5 children)
I apologize and didn't intend to come off as attacking. However, upon rereading my comment, I believe I did. I didn't mean to devalue your work, but the SSH and Kerberos logs seem so much more obvious to me as those three are pretty effective proof that no malicious modification took place (which I what I would have looked at first) if there is not evidence that no one was logged in at the time with sufficient database permissions. For example, depending on your could look for pubkey authentication with "Accepted publickey for USER from ADDRESS port NUMBER protocol" and an open session with "session opened for user USER by (uid=1234)" a closed session with "session closed for user USER" and "Received disconnect".
But that in now way means to take away from your Database examination. As someone else mentioned, short of grepping the apache access_log for journal.pl page accesses, I'm pretty sure you did way more than I would have done in that regard and much of that wouldn't occur to me as being outside my expertise. This old dog would have broken a rib or two trying that trick.
(Score: 2) by martyb on Tuesday October 01 2019, @12:03PM (4 children)
Apology accepted, and much appreciated!
So, let's put our heads together, shall we?
It would be a HUGE help if I had something more to go with. For each of the services that comes to mind, please enumerate:
Lastly, I am probably missing something that I don't even know that I don't know! Please feel free to mention any other things I should be aware of!
Though I certainly invite you to respond, if anyone else can chime in, that would be appreciated, too!
Wit is intellect, dancing.
(Score: 0) by Anonymous Coward on Tuesday October 01 2019, @10:31PM (3 children)
I don't know your exact setup, obviously, but the first place I'd look is where your SSH Daemon logs on staff.soylentnews.org (assuming https://wiki.soylentnews.org/wiki/SystemAdministration/AccessInstruction [soylentnews.org] is accurate). Note that it has been awhile since I've done this, so my memory might be a bit off.
1. If you use openSSH, you'd want to check /etc/ssh/sshd_config for LogLevel and LoggingFacility, which default to INFO and AUTH respectively.
2. Then you'd want to check your command for modifications with something like "ps aux | grep ssh" and should see something like "/usr/sbin/sshd -D" there.
3. You should then check your logger, if the command line or sshd_config isn't modifying the defaults.
A. For rsyslog, it will most likely shove the log into /var/log/auth.log or /var/log/messages, but it may just be easier to grep them all for something like "sshd" or "ssh2" or key sections of the common replys you get.
B. for journald or another journaling system, you can just check the journal assigned to that service.
5. Group all messages by PID (given in brackets, can be done with sort keys or a simple python script) IMPORTANT NOTE, acceptance and session messages are on the same PID, the disconnect message will be on a different PID (usually 2 or 3 higher, if sequentially assigned) but close in time to the session close, and using the same IP address as the connect.
6. Remove all matching sessions and authentications that CLOSED before the time in question ("Received Disconnect" or "session closed for user") based on PID and time.
7. Remove all matching sessions and authentications that OPENED after the time in question ("Accepted publickey" or "session opened for user") based on PID and time.
8. That leaves you with all the sessions POSSIBLY OPEN at the time.
9. Depending on your LogLevel, that can be a lot or a few. You might notice that some are sshd logins and some are sudo substitutions (some systems also log the sudo command itself), but the messages will contain the effective username of the account in question (and the sudoing user) so you can grep for users with database access.
10. This is where judgment comes in. Could some sessions have closed but not logged due to server restart? Does a session hanging open for two months with no close message and nothing for that PID is in your process list seem suspicious? Also, as noted, disconnect messages are on a different PID from acceptance messages.
I would also check the /home for each user for the various hidden history files (.mysql_history, .python_history, .bash_history, etc.) for the commands they ran. Now those files can be edited but .bash_history (or whatever shell they use) would, most likely, have that command in the history unless other means were taken to cover them, but then the files would look suspiciously different from others without assistance from a super-user.
You could also check the Kerberos logs for open tickets for users, but I'd have to look up how to do that, since the last time I did was almost a decade ago.
Finally, apache2 and nginx should log into their own directories. For example, apache2 on debian should have an access.log at /var/log/apache2/access.log and RedHat at /var/log/httpd/access_log while FreeBSD is at /var/log/httpd-access.log IIRC. If you grep that, it should show you every time someone requested a page. I believe grepping for "POST /journal.pl" should give you every time someone posted or previewed a journal post. For future reference, you (or your devs) should look at http://httpd.apache.org/docs/2.4/logs.html [apache.org] or whatever your version is, on setting up custom and conditional logs for different pages to help with analyzing problems later or to separate GET, POST, and other HTTP methods into 3 different logs.
(Score: 0) by Anonymous Coward on Wednesday October 02 2019, @03:48PM
"unless other means were taken to cover them, but then the files would look suspiciously different from others without assistance from a super-user."
Exactly. TMB has admin access, and he went from saying it is impossible to detailing how it could be done, then his excuse was "I'm too lazy and my hand hurts." I am trusting martyb's investigation and claim that notifications would have been sent out very shortly and thus it is not likely such hacking was done.
What shojld be more concerning is the quality of this community. Trolls and alt-right losers are encouraged around here. False information is often given a pass except by a small group of users that care. More than anything this has driven away most reasonable users and we're left with a shitty alt-right echo chamber now with 3 or 4 users who have stuck it out.
Note that no matter what any criticisms get boiled down to "muh free speech" even if you do not advocate censorship. It is very telling that the conversation is always steered back to a nebulous ideal that everyone agrees with. Shitty alt-right playbook, hide in plain sight and steer conversations away from inconvenient topics, reinterpret points as part of a response to drag out debates and obfuscate the original points.
You can see it on the national level, and those tactics are replicated here. Amoral twats -> TMB, khallow, runaway, bot, npc sock puppets, VLM, probably some others.
(Score: 2) by martyb on Thursday October 03 2019, @03:50PM (1 child)
Wow! Thanks so much for all of the info! Unfortunately, Real Life (&tm;) intervened and have been busy with other matters. I apologize for the delayed response!
Sadly, my free time is still at a premium, so I am not sure when I will be able to follow through on these most helpful leads, but I needed to at least acknowledge your most informative reply.
Thanks again!
Wit is intellect, dancing.
(Score: 0) by Anonymous Coward on Thursday October 03 2019, @09:14PM
Not a problem. I understand you have other things, this is supposed to be a hobby, not a job! Regardless, I looked into Kerberos. krb5.conf or kdc.conf or both should have a [logging] section. That will tell you where it is logging to, but if not specified it defaults to the syslog under the auth facility. If syslogging under auth would cause syslog to shove the logs into wherever the auth logs are sent to, usually /var/log/auth.log, along with other auth messages, like sudo and ssh and pam, etc. (look for something like "auth,authpriv.* H-TAB var/log/auth.log") I don't know what the messages look like, specifically, but if you find the log files, I'd assume they'd be mostly self-explanatory and they are all time-stamped. That's about as good as I can get right now. Good luck proving a negative!
(Score: 5, Informative) by martyb on Monday September 30 2019, @07:41PM (1 child)
There is a distinction between when things get stored into the DB and when that information is percolated up and made visible on the site.
Slashcode used the Model-View-Controller [wikipedia.org] design in its construction.
There are a great number of tasks which run in support of the site. Rather than having all of them running all of the time, each is scheduled to run periodically, look around, do its task, update the DB, and quiesce until its next run. That reduces memory pressure as well as CPU spikes.
Updating the "Recent Journal Entries" Slashbox is one of those tasks. The *content* of that Slashbox is scheduled to be updated every minute (if I am reading the source correctly). But, there's more. The updated version of the main page in which it appears *also* has to make it through our front-end servers and caches. That inserts an additional delay.
So, the majority of the delay lies in the process of *informing* people of the changes that have already been made. If one were to directly load the user's Journal, that's different. It would take further investigation to nail down the exact vagaries, but it is my current understanding that such a direct request *should* go directly to the DB and present whatever is there to be displayed.
I'm reasonably confident that is how it works, and very sure that if I made any mistakes, someone else on staff will be around to set things right!
Wit is intellect, dancing.
(Score: 3, Informative) by The Mighty Buzzard on Monday September 30 2019, @07:55PM
It's created by run_portald.pl, yo. Was running at 37 after the hour. Now it's running every five minutes on the fives and dimes.
My rights don't end where your fear begins.
(Score: 3, Interesting) by The Mighty Buzzard on Monday September 30 2019, @07:53PM (1 child)
That's normal. The journals sidebox is recreated by a slashd (reinvented cron daemon wheel) every so often. With the processing power we have versus the minimal load that would cause, I think it should be done dynamically nowadays. That's fairly low down the todo list though.
Hmm... Edited the live code to make it update every five minutes instead of once an hour and committed it to github as well. That should hold us until I can get to it.
My rights don't end where your fear begins.
(Score: 2) by The Mighty Buzzard on Monday September 30 2019, @07:59PM
s/ every so/ task every so/
My rights don't end where your fear begins.
(Score: 5, Informative) by acid andy on Monday September 30 2019, @07:02PM (1 child)
+1 to what NotSanguine said: Thank you MartyB for taking the time to investigate this in such detail and sharing the information openly with the community.
It's a shame Athanasius quit over this. A lot of his posts were well thought out and analytical. I get a sense that the disappearing journal was a last straw kind of thing for him though--if it hadn't been for that, I rather suspect something else might have sent him over the edge sooner or later which still wouldn't be good for this site.
How many of the current regulars will still be posting here in 5, 10, 20 years I wonder...?
If a cat has kittens, does a rat have rittens, a bat bittens and a mat mittens?
(Score: 3, Funny) by Anonymous Coward on Monday September 30 2019, @11:58PM
Don't worry, I will be here until the last day.
(Score: 2) by turgid on Monday September 30 2019, @07:47PM
++teamwork is faster.
I refuse to engage in a battle of wits with an unarmed opponent [wikipedia.org].
(Score: 2) by DannyB on Monday September 30 2019, @08:04PM
If a journal entry is never read by anyone, then was it ever posted?
Or was it deleted?
The lower I set my standards the more accomplishments I have.
(Score: 5, Insightful) by SemperOSS on Monday September 30 2019, @10:01PM
If I could moderate the story itself, I would give it +100 for Interesting, Insightful and Informative!
martyb has not just done a tremendous amount of searching and researching to come to sensible conclusions about the system and what could have happened, but also taken the time and effort to inform the community, for which we are grateful.
Thank you martyb, and well done!
I don't need a signature to draw attention to myself.
Maybe I should add a sarcasm warning now and again?
(Score: 5, Insightful) by Runaway1956 on Monday September 30 2019, @11:33PM
The dedication is impressive. I've never seen an administrator anywhere work so hard to address a user's complaint.
(Score: 1, Interesting) by Anonymous Coward on Tuesday October 01 2019, @12:43AM (4 children)
When I first started using UNIX - approaching four decades ago - it was BSD 4.2, and I added /usr/games/fortune to my .logout file.
My favorite fortune was something along the lines of "always assume stupidity before malice". And I share that with you, the person who thought their comment had been deleted. Always assume stupidity before malice. And, don't exclude your own stupidity, it is the most common element in each of our personal sequences of cause-and-effect. Don't assume someone is out to get you. It's counter-productive.
My second observation, for the operators of this site, is that without even studying the tables I can infer that if you delete an article and do not lose the comments, that it's likely that your database schema was not designed using an object-oriented methodology such as entity relationship diagramming, and so I would suggest giving the topic a glance.
While we're on architectural issues ...
I've been thinking of building a blog and one of the major dissatisfactions I have with every blog is the gratuiitious inclusion of a database. Why can't this stuff be kept in flat files, in a directory tree, in a filesystem, and a few shell scripts be called to read it all in and insert the required HTML and CSS markup?
Everything I describe that pertains to a blog, pertains to a news site, or any sort of site, really.
Build the site from text. Keep the text in files. Ditch the database. It doesn't add value. You can generate reports with find(1) and grep(1) and awk(1) and they will take a little longer, but you'll have a free machine or two, as a result of ditching the relational database.
Then you won't lose any data. Or if you do there will be traces of what happened in the leftover files and their timestamps.
~childo
(Score: 2, Insightful) by Anonymous Coward on Tuesday October 01 2019, @01:13AM
You said:
entity relationship diagramming is an object oriented methodology.
No, no, no! ER diagrams are RELATIONAL, as in: a relational database.
"Objects" have nothing to do with it.
And as for flat files being more reliable for a site like this than an relational database: wrong again.
(Score: 4, Insightful) by janrinok on Tuesday October 01 2019, @02:47AM
I appreciate your suggestions but counter with the following. If doing as you suggest with a tree of plain text files was more efficient or superior in some other way, then everybody would be doing it now. They don't do that, and for a reason. It isn't.
Databases are optimised for storing and retrieving data efficiently with the benefit that the code doing the heavy lifting has been debugged a million times already. While a plain text tree could be written (if we had spare staff sitting around doing nothing, which we don't) I believe that it would probably require more computer power, not less, than using a proven database to do the same task, and it would have awk, sed and similar scripting bugs just waiting to be discovered by corner cases and obscure search conditions.
I accept that our database and the Rehash software might well not be an optimal design but throwing it away and starting again with an unproven system would be utterly pointless and wasteful of whatever scant resources we do have. But, should you decide to go down that route yourself in your blog, I would be interested in seeing how it goes for you. It would be a worthwhile experiment to observe.
(Score: 1, Insightful) by Anonymous Coward on Tuesday October 01 2019, @06:26AM
I find this somewhat humorous. A hierarchical, flat-file system like that is still a database, its just your schema is folders, not tables. With a good number of users or comments, unless you have a giant disk cache or an optimized directory structure, your access times will suck. Many off your accesses would effectively be full table scans, indexes would be effectively worthless, you'd have terrible lock contention, race conditions, no clear conflict-resolution policy, no transactions, larger space requirements, inode/DE limits, and those are the ones that occur to me off-hand.
(Score: 0) by Anonymous Coward on Tuesday October 01 2019, @09:29AM
Write your own. Use an open source solution as a base. Soylentnews, pipedot and slash are all publically available