Operating Systems:
One of the early missteps was the choice of CentOS as the operating system for one of our servers: beryllium. All of our other servers ran Ubuntu. That CentOS server, beryllium, became the server for all the other services that were not directly required for site operations. Quite frankly, it's a bit of a mess. For the curious, expand the following for a subset of what is runs there:

*SPOILER* (click to show) *SPOILER* (click to hide)

Charybdis, IRC server, http://irc.soylentnews.org - port 6667, 6697(SSL)
Atheme, IRC services
Iris, IRC web chat, http://chat.soylentnews.org - port 3989, forwarded from 80 by apache
Various IRC bots
ZNC, IRC bouncer for staff, http://irc.soylentnews.org - port 60000
Yourls, URL shortener service on http://sylnt.us - port 80
MySQL, used for Yourls
Postfix
Mailman
Dovecot
Apache2/httpd
OpenSSH
ntpd

Progress:
We are in the process of cleaning things up.

We now have 3 servers running Gentoo: lithium, magnesium, and our new server aluminum. Gentoo lets us custom build our servers so they are only running the services we need. That gives us better security (smaller attack surface) and better performance, too. Oh, and no systemd.

The Nitty Gritty: At this point, I'll turn the microphone over to Deucallion (aka Juggs) on what's happening with IRC on aluminum (lightly edited):

So far we have brought a new ircd (Internet Relay Chat - Daemon) into the network: "call.me.al". The 2 crucial key points are:

1. Moving services (NickServ, ChanServ, GroupServ, HostServ, SaslServ et al.) Those are all provided by one server side process (atheme), anyone not clued up won't really to know they exist as a separate thing and just interact with it to register a nick and then as the channel bots they see with all the daft names.
2. Will be reversing DNS entries for irc1 and irc3.

If I do my part right, there will be minimal to no outage time caused by any of it.

Then there are all the ancillary bits and bots that do logs and stats and story subs and the like but they are not intrinsic to the main IRC infrastructure and just an inconvenience if they go away for an hour or so while ported across.

I announce to everyone here on IRC when I am doing work on something and anticipate a possible outage of some kind as TBH the only people who care if IRC goes down or is degraded in some form are the people using it at that time. As a user it is nice to know in that scenario that it is not your client playing up, nor your network, or your ISP etc. it's just gone for maintenance and sit it out; do not bother investigating. Same reason I announce when I stop messing with stuff so people know there are no works underway.

And for clarification the 3 ircds we currently have now are all classified as hubs, no leafs, they are peers in a network. There is no master-slave relationship in play. We think of irc. as being master because all the other ancillaries sit on it but they can just as well sit on irc2. or "call.me.al". The ircds and services do not give a flying monkey what DNS name resolves to them, it is just convention to name the ircd that resides at irc2.soylentnews.org "irc2.soylentnews.org" or as it is "irc2.sylnt.us" - but it is just that, a name, a label.

This is specifically why I am going with "call.me.al" for aluminum: it breaks that cognitive second guessing about "do I need to match the reverse DNS here or not" questions in my mind at least when I come back to look at it in a year or 2 or 3 or 5. Maybe I am just a simpleton with OCD or some such, but to my mind - a label should be a label, the DNS should be another thing. If they do not need to match, make them different for clarity.

Epilog:
Do keep in mind, this is all being done by volunteers from their (limited) spare time and at no charge. There's still much to do, but we are making progress. Our goal is that over the next couple months or so, to have all of our servers refreshed and moved over to Gentoo. There will be hiccups. Hopefully they will be minor, few, and far between. As always, we will keep the community apprised as to our progress.

So cross your fingers, and join me in thanking these fine folk for all their efforts: TheMightyBuzzard, Deucallion, audioguy, and NCommander!

Previously:
(2020-05-09) Site Potpourri for Mother's Day [Updated]

Original Submission

Silly Season: It came a bit earlier than usual, but we are well into the "Silly Season". It's that time of the year when places of higher education close for the summer (in the northern hemisphere) and people's minds turn to summer vacation. Research reports are few and far between. To fill the gap, many publications turn to lighter fare for lack of anything better to publish. Compounded with the COVID-19 pandemic, what research that still continues tends to be slowed down by safety precautions.

What that means is we have less of a selection to choose from in trying to bring stories to the community for discussion. See something tech-related on the web that you think the community might be interested in? Please submit it to SoylentNews! It does not have to be the next "Great American Novel", either! Of course the more "publication ready" you make it, the easier it is for an editor to decide to run with it. On the other hand, if the topic is interesting, chances are an editor will see it and decide to run with it. If you have any questions, it's helpful to consult the Story Submission Guidelines. Also, a quick scan of stories that have recently been posted to the site should provide guidance as to story organization and layout. Lastly, we appreciate side comments within the story submission; for example: "Doesn't contain the links from the story." Sensational, spittle-spewing spouting soon sees silence. Try to stick with the basics of answering "who", "why", "what", "where", "when", and "how" and you'll be on your way!

Subscriptions: Thanks to a generous first-time subscription of $120.00, we just passed $2,100.00 towards our goal of $3,500.00 for the first half of the year (2020-01-01 through 2020-06-30). Thank you to all who have subscribed and helped pay for things like servers, taxes, and an accountant to prepare the taxes.

As you may recall, we made an announcement on April 19 concerning site subscriptions. Yes, coming down with the SAR-CoV-2 virus is bad. But so is having so many people out of work and and trying to make ends meet. We wanted to support people spending their money locally to support their community and the economy. Therefore, anyone who had a subscription that would otherwise have ended earlier was granted a free extension through 2020-05-30.

That said, we do still have bills to pay. If you are of a mind to do so and can afford it, we still are accepting subscriptions! Do bear in mind that Javascript needs to be enabled. We do not process the transactions directly, but instead invoke the API (Application Programming Interface) form provided by PayPal or Stripe. They, in turn, handle processing the payment and then deposit the payment (less processing fees) to our account.

Reminder: the indicated amount (e.g. $20.00 for one year) is a minimum for that duration. So, you can absolutely select a one year subscription and change the amount to, say, $100.00 from the $20.00 that was suggested. (For the record, the largest subscription to date was an extremely generous $1,000.00!) On the other hand, we have two Soylentils who subscribe for $4.00 — every month — like clockwork. It warms my heart every time I see their subscriptions arrive! It's one of the things I love about this community: everybody contributes in their own way and somehow it all comes together. And it has held together since February of 2014! Thanks everybody!

Servers, Part 1. Behind the scenes, TheMightyBuzzard spent the weekend setting up a new server, aluminum. We are gradually moving to a Gentoo Linux base for our servers. Rather than pre-compiled binaries that get downloaded and run locally, Gentoo provides source code for download that one compiles and builds locally. At the moment we have three Gentoo-based servers (lithium, magnesium, and aluminum), one server on CentOS (beryllium), and the rest are on Ubuntu. By moving to Gentoo Linux, we get a streamlined server with a smaller attack surface as only the things we need are built into the kernel. That lone CentOS server? It has been with us from the start and has been no end of a hassle. Several services "live" on it and these need to be migrated before we can retire it. The first stage of that process is underway as Deucalion has been working on bringing up IRC on aluminum. In turn, other services will be brought over. Then we can (finally!) retire beryllium for good! Next on the list are sodium and boron (aiming to have completed by June.) Along with that, there have been new (security and otherwise) releases of other services that site depends on. We intend to get those upgraded as we move to an entirely Gentoo platform. Please join me in wishing them well on the migrations and upgrades!

Servers, Part 2: We had a hiccup with Linode (our server provider) on Friday. Through it all, our servers stayed up and running! Unfortunately, the problem was with one or more network switches at Linode. (Cf: Bert & I on YouTube 😀) The front end (which processes requests for web pages) as well as IRC (and possibly other things of which I am unaware) were inaccessible for the better part of an hour. Given how frequently SoylentNews used to crash (several times each *day*), it is a testament to the hard work put in at the outset that this is such a rarity for us today. Our servers currently have uptimes in the range of 6-9 months... and it would be longer except for some behind-the-scenes work to take advantage of free storage upgrades made available to us by Linode. Remember all work on the site is performed by volunteers who give of their limited free time to keep things humming here.

Summary: Our Folding@Home team is helping to find a cure for COVID-19. Please send in story submissions. We are still accepting subscriptions. Our servers were NOT "Pining for the fjords". Server upgrades are in progress.

Original Submission

Symptom: It all started with a tag (i.e. HTML element) error in this comment in a user's journal where the user coded a <strike> tag, but neglected to provide a matching </strike> tag.

Example: Let's look back to the original comment which manifested this bug. Here's the latter part of it, after being corrected:

Looks who's projecting. Consider your phrase "steal jobs and send them overseas for cheap/free labor" (no such thing as free labor). That helps billions of poor people improve their lives. Yet here you are, selfishly obsessing over your developed world pricing power (with some very unempathic label spewing) rather than display the alleged empathy or morality that you claim to be concerned about.

Your empathy is nonexistent and your morals are bankrupt - definitely not the sort of person I should be taking advice from!

The problem is that there was no closing <strike> after the word "cheap", so it looked like this:

Looks who's projecting. Consider your phrase "steal jobs and send them overseas for cheap/free labor" (no such thing as free labor). That helps billions of poor people improve their lives. Yet here you are, selfishly obsessing over your developed world pricing power (with some very unempathic label spewing) rather than display the alleged empathy or morality that you claim to be concerned about.

Your empathy is nonexistent and your morals are bankrupt - definitely not the sort of person I should be taking advice from!

If that was all that happened it would be ugly, but tolerable. Unfortunately, every single character following it on the page was struck through, too. Not Good™.

Immediate Fix: To my knowledge there was only one way to rectify the immediate issue: manually go into the DB and insert the missing tag. This I was able to do quite quickly, but I still saw a problem.

More to Come: Anyone who saw this comment discussion, either at this moment, or who happened upon it later, would see an opportunity to intentionally leave a hanging tag and thus disfigure the site. Trolls gotta troll. So, I made the fix and noted same in this comment reply.

So, an instance of the problem was fixed, but now what? There's a "proper" way to do it, and there is another way to get the same effect that can be quickly implemented. I chose the latter.

Perl Code: Normally, such HTML errors in a user's comment or journal entry (or an editor's edit of a story!) are caught and handled by a routine in our Perl code: balanceTags(). The code looks though all the tags, with whatever nesting is present, detects where tags do not have a required closing tag,and silently inserts it into the text that makes it into the DB. It's rather hairy code because it also needs to handle: extra closing tags, mis-matched closing tags (e.g.: <b> bold <bold and italic> </b> </i>), mistyped or otherwise non-existent tags, restricting what tags are supported, and custom-created site tags! Whew!

Further, to fix it in the Perl code means going through the process of: checking the code out from GitHub, understanding the code, making the change, compiling the change, testing the change, (after rolling it out to our dev server), and then -- if all looks good -- rolling the change out to our production servers. And, of course, nobody was around at the moment who could support such activities even if it were an easy coding change (and it is not!)

Expediency: I realized there was another approach which would mitigate the problem -- not requiring Perl coding changes -- but could still prevent its recurrence: changing the value of a "Site Variable" (aka "site var").

Rehash Primer: Now I need to step back for a moment and explain a couple things. The code for SoylentNews.org is a fork of ancient Slashcode that was put up on GitHub. Slashcode was implemented using a Model View Controller design. There is a clear demarcation between what is done where and at what level.

Templates: As part of its implementation, the SlashCode implemented "Templates" which generate the HTML pages that get sent to the browser and act as an interface between the code and the user. As far as I know, every page you see on the site comes by way of a template. Each template may, in turn, make use of other templates. Templates can make calls to underlying Perl code. This is where the site implements the heavy lifting of talking to the database (DB), creating e-mails, and other closer-to-the-metal activities. The template language (from personal inspection; I have yet to find an official document as to its syntax and semantics) presents what appears to be a simple, macro-capable language. The templates are stored in the DB and loaded into memory when the site is started. An advantage of this is that changes to templates can be made "on the fly" using a template editor (which is, itself, a template!) There is one caveat: for the changes to take effect, processes on the front-end servers need to be "bounced", i.e. restarted, so the changes are loaded into memory from the (updated) DB.

Site Variables: There are some parameters whose values affect the site's operations: Name of the site, domain name of the site, the name of the Anonymous User account, ... it goes on and on and on. There are no less than 750 site variables! And, as many things that grew beyond their initial construction, there is no simple way to look for what site vars might be appropriate to any given situation. One is just expected to know what they are and what they do and how they do it. Simple enough approach when they first started, I guess. A search capability would be very nice to have, but it will take some coding to make that happen, so it has become just another of the several changes that would be nice to make to the site.

So, back to the matter at hand, I knew about the "approvedtags" site var which lists all tags which are permitted to be used on the site. Sure enough, "STRIKE" was in there! And, I saw that "DEL" was in there, too. Does "DEL" have the same problem? I tried a quick test comment on our development server and it revealed that balanceTags() properly handled a hanging <DEL> without a matching </DEL>. Yay! I removed "STRIKE" from the "approvedtags" list, saved the change, bounced the front-end servers, and breathed a sigh of relief.

All was good, until someone asked in a footnote to a comment why do we still list STRIKE as being a permitted tag for comments? What? I double-checked and verified that "STRIKE" was no longer listed in "approvedtags". What is going on? So, I commenced searching and finally discovered another site var: "approvedtags_visible" which contains the list of tags that is presented to the user as being available. And, sure enough, "STRIKE" was in that list. Grrr! I removed "STRIKE" from "approvedtags_visible", saved the changes, and saw no further issues mentioned there. Finally!

Or so I thought. Did you see what was missed? The site vars were now correct and up-to-date. The changes were saved to the DB. But... those changes existed only in the DB. Still needed to 'bounce' the front end servers for the changes to take effect. So, that entailed a quick SSH to our servers, running the bounce scripts, and verifying that "STRIKE" was truly and properly removed from the tags presented to the user as being available for use, and that anyone trying to use <STRIKE>, anyway, would discover it did not work.

Conclusions: So, here are some lessons learned:

1. When you want to use a <STRIKE> tag, use <DEL>, instead.
2. There is no assurance that reporting a problem in the comments will be noticed.
3. Please report site issues with an e-mail sent to admin (at) soylentnews (dot) org.
4. In addition to sending an e-mail, mention it in the "#dev" channel of our IRC server.
5. This particular issue should now be well and truly fixed. Please report any problems you may discover with it.
6. When communicating changes made to the site, a mention in the comments does not suffice.

7. --martyb

[Janrinok says: TL:DR Martyb fixed it, OK, OK, I have read it....]

Original Submission

So, I hacked up something that I hope addressed the initial concern: "kind of long to scroll down through". I'll be the first to admit the implementation is crude. We can go for pretty later. (The perfect is the enemy of good enough, right?) I think the ideal would be to have a separate nexus for virus-related stories. That way we would not feel compelled to gather a bunch of story submissions into a single story. We could process each submission independently and release each on its own. Unfortunately, there is much more to it than just adding an entry to the site DB.

It has been implemented on our development server: https://dev.soylentnews.org/ and I hereby solicit feedback from the community on how well it works. It was implemented with one addition to an in-memory copy of a single site template (dispStory;misc;default).

For the curious, see Original and Updated Versions of Template: "dispStory;misc;default" ("Skip to Comment(s)" button), but do be aware that rehash replaced tabs with spaces, so what you see is NOT an exact copy of the sources.

Now what? Feedback! This is your site. I am well aware there are Soylentils who have a much better grasp of HTML and CSS than I do, and am hereby soliciting supportive feedback.

Test scenarios:

1. Does the "Skip to comment(s)" Button not appear on the main page?
2. Does the "Skip to comment(s)" Button appear on the story page?
3. Does it work?
4. Is the appearance consistent across all of the available themes?
   1. Site Default
   2. BadA55
   3. Chillax
   4. Grayscale
   5. Black IcIcle
   6. Night Mode
   7. NV
   8. OMG PWNIES
   9. SoylentNews
   10. Vomit
   11. VT100
   12. VT220
5. Is the layout consistent other homepage settings?

   [] Simple Design

   Simplifies the design of Dev.SN to strip away some of the excesses of the UI.

   [] Low Bandwidth

   Reduces the size of pages for people with slower network connections

   [] No Icons

   Disable topic icon images on stories.

6. Which of the preceding homepage settings would be better served with just a simple anchor?

   <a href="@acomments">Skip to Comment(s)</a>

7. Other, what did I miss?

Original Submission