from the not-actually-an-NCommander-post dept.
Okay, I know it's been a long time since we did one of these but life does intrude on volunteer dev time. Hopefully this one will be worth the wait. Bear with me if I seem a bit off today, I'm writing this with a really fun head cold.
First, what didn't make it into this update but is directly upcoming. Bitpay is still down on account of them changing the API without notifying existing customers or versioning the new API and leaving the old one still up and functional. It's the first thing I'm going to work on after we get this update rolled out but it will basically require a complete rewrite. Don't expect it any earlier than two months from now because we like to test the complete hell out of any code that deals with your money.
Also, adding a Jobs nexus didn't quite make the cut because we're not entirely sure how/if we want to work it. One thing we are certain about, it would not be for headhunters or HR drones to spam us silly but for registered members who have a specific vacancy they need to fill and would like to throw it open to the community.
The API still has some broken bits but it's been low priority compared to what I've been busy with. I'm thinking I'll jump on it after Bitpay unless paulej72 cracks the whip and makes me fix bugs/implement features instead.
There were several other things that I had lined up for post-Bitpay but I can't remember them just now what with my head feeling like it's stuffed full of dirty gym socks.
Now let's throw the list of what did make it out there and go over it in more detail afterwards.
- Tweaked the themes a bit where they were off.
- Changed or fixed some adminy/editory stuff that most of you will never see or care about.
- Fixed a mess of minor bugs not worth noting individually.
- Improved Rehash installation. It should almost be possible to just follow directions and have a site working in an hour or two now.
- Added a very restrictive Content Security Policy.
- Added a link to the Hall of Fame. It was always there, just not linked to.
- Return to where you just moderated after moderating. (yay!)
- Return to where you just were after commenting. (yay some more!)
- Added a field for department on submissions. Editors get final say but if you have a good one, go for it.
- Added a Community Reviews nexus.
- Added a Politics nexus.
- Added <spoiler> tags for the Reviews nexus in case you want to talk about a novel without ruining it for everyone else. They function everywhere though.
- Changed really freaking long comments to have a scrollbar now instead of being click-to-show.
- Massively sped up comment rendering on heavily commented stories.
- Dimming of comments you've already read. (You can turn this off with the controls on the "Comments" tab of your preferences page if it annoys you.)
- Added a "*NEW*" badge to new comments in case you don't like dimming but still want to easily see new posts. (Disable it the same place as above.)
- Removed Nested, Threaded, and Improved threaded comment rendering modes (Necessary due to the changes required for the massive speed-up)
- Added Threaded-TOS and Threaded-TNG comment rendering modes. (TOS is the default)
Morning Update: Really digging the constructive criticism. Some quality thoughts in there. Keep them coming and we'll see how fast we can get a few done. --TMB
Before the specifics, I know some of you are going to see the new Threaded modes and be like "that's pretty awesome" and some of you are going to call us dev types very bad names. Well, this ain't the other site. We're not saying "You Shall Use This Because It's New And Shiny". We're saying something had to be done about page load times approaching a full minute on heavily trafficked stories and the way we pulled and rendered comments made up nearly all of that time.
So, the first thing we did was we stopped pulling every single comment and then removing the ones we didn't want to display. Mostly that means that the comment counts in the dropdown menus for Threshold and Breakthrough are on a per-page basis now.
Third, we paginated every mode. I know it was nice being able to see every comment on one page but that meant pulling and rendering every comment and that simply doesn't work if a story has over a hundred comments.
The removal of sorting by score we can't roll back though. Its loss was a necessity due to the way we pull and sort only the comments that the user actually requests. Previously, we were pulling every single comment for a story and then removing the ones we didn't want. That was both bloody stupid and slow as hell, so it had to go. Unfortunately it means we have to do things slightly differently. It may make a triumphant return eventually but it would require some moderately tricky coding with the particular way our code is laid out.
Oh and if you have objections to the new Threaded modes, by all means bitch about specifics in comments here and we'll see what we can do to address them. After having spent so much time recently bashing on exactly these bits of code, we're quite familiar with them and changes/additions shouldn't take too terribly long to whip out.
Now to the specifics.
Flat: Flat is still flat but now with a collapse/expand button that functions like the ones from Improved Threaded.
Threaded-TOS: If you can find significant differences between Improved Threaded and Threaded-TOS, let us know because it's probably a bug. The idea was to make it as much like Improved Threaded as technically possible with just CSS but paginated like Nested so we don't have to render more than 100 comments at a go. We defaulted everyone on Nested/Threaded/Improved threaded to Threaded-TOS to minimize the aggravation of unexpected change. Oh, and Breakthrough now takes precedence over Threshold, so high scoring comments will always be visible even if they're responding to blatant trolling.
Threaded-TNG: All comment trees start fully branched out but with the individual comments either expanded or collapsed. "Comment Below Threshold" functionality is gone. Breakthrough gets compared to a comment's score to decide if it gets expanded or collapsed. Play with it a couple minutes; it's not terribly hard to grok. Why do we need this mode if TOS covers most all of the best bits of the three old modes? Because I like it. You don't have to use it. Shut up.
Why not leave the old comment rendering modes in as well as the new ones? Because by rewriting them we got a rendering speed increase around a factor of two+, to go with the factor of two+ increase we got by pulling only the necessary comments instead of every last comment a story has with every page load. This has been becoming necessary as we increasingly go way above the 100 comment mark on busy stories. It's not cool for you lot to have to wait forty-five seconds to load a page of comments and it's even less cool to peg a cpu core for forty-five seconds to deliver it to you. If you ever again find a story that takes 10+s to load, something's going wrong and we'd appreciate a heads up. We think there's still some room in the code for improvement but this was the lowest-hanging fruit.
Now on to the rest of the details.
The Content Security Policy should cover what's required for operation of this site (plus allowing for Stripe payments) and nothing else. If your browser honors CSPs, it should not be possible to get smacked with XSS or inline script injection on this site any more; even if we write code buggy enough to allow it, which we have once or twice.
On dimmed comments... This only functions for logged in users currently as it would take some serious work to get it functioning for individual ACs, even using cookies. What it does is when you load a page of comments, it picks the highest comment ID from that story and marks that comment as read by you. Switching between pages of comments or changing your Threshold/sort order should not update which comments you have read, even if new ones have come in since your last read comment ID was set. Hitting the "Mark All as Read" button or hitting your browser's Refresh button on the main story page should take the stored comment ID and set the opacity to 60% on all the comments with a comment ID equal to or less than that. It's not entirely accurate but it's pretty damned close and it doesn't bloat the db much at all. Oh and read histories get wiped after two weeks of not being updated for a particular user/story combination to save on db space as well.
The new comment badge functions exactly opposite of dimmed comments. It puts "* NEW *" in the title bar of comments you haven't read yet. It's there strictly so you can have the same functionality but dislike the aesthetics of comment dimming. You can technically use both if you really want new comments to stand out but that would just be weird.
Returning to where you last moderated works like this. If you moderate one comment, you'll get sent back to that comment. If you moderate several in one go, you should get sent to the one farthest down the page. Moderating does not update the comment ID of what you've read for dimming purposes.
Returning to where you just made a comment? That's pretty self-explanatory. It also should not update the comment ID of what you've read for dimming purposes.
The Politics nexus. This does not mean we're looking to have even more political stories. The balance of tech/science/etc... to political stories is not going to change nor will the quality of accepted political submissions. It's primarily a way to let people who are sick and bloody tired of seeing politics here set a preference and never see political stories again. It's also handy if you wish to see what political stories we've run recently as clicking on the nexus link on the left of the page will show you only those stories.
The Reviews nexus has been brought up three separate times that I can remember by different groups of people, so we decided to go ahead with it. It's going to be a book/film/software/hardware/etc... review and discussion place. By my understanding, though I'm not really involved, it's getting its own space because some folks wanted to start what amounts to a site book club. Tech books will of course be welcome but it's open to all genres of printed and bound words. Ditto non-book reviews. Just don't go sending in a review of something we normally wouldn't publish news about on the site. Not enough people are going to be interested in your review of the barber shop down the street from your house, so it won't get published.
Spoiler tags, <spoiler>text you don't want casually seen</spoiler>, work both in stories and comments and are just a bit of css trickery that hide the text between them until the person viewing them hovers over the *SPOILER* text. There's a slight delay, so don't think it's not working because it's not immediate. That's intentional so you don't accidentally trigger showing the contained text by briefly crossing it.
By popular demand, <del> tags were also added.
That's all worth mentioning in this site update. Look for another one hopefully in May or late April. If you find any bugs, please slap them up as issues on our github repo or email them to firstname.lastname@example.org.
Martyb here once again with today's update on our site's update. I apologize that this is not as well written as my prior updates as exhaustion and outside issues are demanding of my time. I ask you to please bear with me.
Quick Recap: As you may recall, our servers were getting melted trying to serve up highly-commented stories. Further, this made for an unacceptably long delay between the time one would request a story and when it would finally get returned for display. Our devs have implemented alternative display modes, "Threaded-TOS" and "Threaded-TNG" which use a much more efficient means of processing comments and getting them to you. These changes went "live" on Saturday, February 25.
Like anything new, we expected there would be issues. We very much appreciate your patience as we tried to work through these as they were reported. And did you ever let us know!
Paulej72 (aka PJ) and TheMightyBuzzard (TMB) have been laboring mightily to keep up with the issues that have been reported as well as a few they found independently. Similarly, as their fixes have gone live, our community has had to deal with a changing landscape of "what happens when I do this?" To add to this, comments being such a major part of the site's purpose, there are "knobs" in several places where users can customize which comments are presented to them and how they appear. The permutations are many and wide-raging. As bug fixes have been made, the impact of changing these has had different effects over time.
I've been astounded at how much the community has been supportive of our efforts, how well problems have been described and isolated, and how quickly the devs have been able to fix bugs as they have been noted. Even more impressive was the discussion in our last update story on possible alternative means of implementing the <spoiler> tag. I'm proud to be part of this community — you rock!
Stories: While all this activity has been happening, stories have still been posted to our site for your reading and commenting pleasure. We are working with a reduced editorial staff at the moment. Us long-timers have been posting as we can, but I would like to personally thanks our new editors fnord666 and charon for their heroic efforts getting stories posted, and takyon for his continued efforts at providing well-written stories. I have noticed submissions from new folks as well, and the heartens me immensely! (Note: I hesitate to call out people in particular for fear I will overlook someone; any omission is purely my fault and I would appreciate being called out on it if I have failed to list your contribution.)
Plans: This development blitz has, however, come at a cost. For those who were with this site at its inception, there was a "day of rest" imposed on the developers who had worked basically non-stop trying to get our site up and somewhat stable. I have suggested a similar break to our dev staff. Recall we are all volunteers doing this in our spare time. PJ has plans coming up and will be unavailable on Friday and Saturday. From what I've seen, TMB is well nigh a crispy critter at this point and most certainly needs a break. And, quite frankly, I've put a lot of personal stuff on hold while working on this update and could use a break, as well. In short, we are tired.
So, PJ is around for a bit (in his free time while at work) for today and TMB is getting a well-deserved breather. NCommander is nearing burnout has been tied up with an outside project that demands his full attention and has been unable to help much. I'll poke in from time to time, but I really need some time off, too.
What I ask from the community is that we do something similar. Step back for a moment. Look at the forest and not just the trees. Play around with the different display Modes. Try setting a different "Breakthrough" and/or "Threshold". Things should be much more stable today, so that will make it easier to gain a "mental model" of what does what.
The other thing I would ask is for the community to pull together and try to address issues together. Someone posts an issue about struggling with having to click on all the little chevrons? Inquire about their user preference settings, and suggest a different value for Threshold/Breakthrough. My sense is that some are more adept at using the new features and they can help others to get a better understanding of how things work. With those issues addressed, we can more clearly identify and isolate underlying problems and focus our energies more productively.
tl;dr We're not done yet, we truly appreciate your patience and forbearance during this transition, we need a break, and you guys rock in helping others in the community understand and use the new stuff. As always, keep our toes to the fire — we are here for you — let us catch our breath and we'll be better able to move forward.
So the Dev Team has been hard at work fixing up issues with with the 17_02 release. We compiled all of your comments from the 17_02 Meta stories into a large bug and feature request list. We have been working on getting these issues fixed as soon as we can.
You may have noticed some changes over the last week that went out to fix some issues, and we just released some more fixes today.
Here is a list of the major fixes since the last story:
- Comment titles now act to trigger actions as well as the buttons.
- New icons for the buttons that are bigger and spaced out a bit more.
- Removed extra spacing for some of the modes when buttons are hidden. (Still needs some more work, but much better than before.)
- Changed how we set states for hidden threads so that the individual comments are not set hidden as well. This will lessen the number of clicks needed to open a comment in this case.
- Flat comment mode will now show a comment's children when look at comment (cid is set).
- Temp fix for Content Security Policy eating CSS when not on https://soylentnews.org.
- Domain Tags now show in comments.
- Fixed broken messages when looking at them from messages.pl.
- Fixed for @user: shortcut links that would eat a character after the ":".
And here are the latest updates:
- Editor fixes: added new lines to story editor and fixed topic tree popup issue.
- Fix for black on black text boxes in Grayscale theme
- Added SVG icons for buttons and CSS fixes to enhance look.
- Mod Points now back to UTC 00:10.
- Add # of children to collapsed thread title.
- Fix comment details to bring in the correct data which broke email and journal links.
- Add time to collapsed comment title.
- Fix select all button in messages.pl.
- Redirect returns to correct article.pl page after moderating.
- Updated CSP to fix issues. We still need to search for some inline JS that may need to be purged or rewritten to work correctly.
So if you see any new bugs that you think are related to these changes, or just want to let us know about an ongoing issue, please feel free to comment below.
Here are the currently known bugs that we are working on:
Continuation of: Site Update 2/27
So, the recent site update got a lot of news, and comments. Predictably, there was a lot of comments split on the fence both ways. I've been out sick and haven't been actively involved in SN in a few days, but I did review the updated changes on dev before they went out. I'm still not up to responding to you guys personally, and TMB/Paul have had things covered, so I'm just going to write a blanket story. So, let's open this and say THIS ISN'T THE FINAL SET OF HOW THINGS WILL BE. I'm leaving my comments above the fold to make it clear what's going on. I'd put that in a blink tag on if that was still in the HTML standard.
The changes to commenting were primarily driven on technical grounds. To do D1.5, the site had to load a mass load of comments and do server side processing to thread them. To give you an example, on a cold page load, before we apply caching a few points in the site would take over a minute to load, render and thread. The only thing that prevented the site from becoming unusable in 503s is that the frontend has a lot of caching. Even with that, we can't cache every single bit of the site at once. In a "cold cache" scenario such as after a varnish or DB update, the site would be borderline unusable until those caches could be loaded. So let me make this clear that this change wasn't a change for changes sake. There was (and is) a need to revamp the commenting.
We noted that this change was coming in other meta stories, and even had a landing article on dev for people coming to check it out. No one did. How we use commenting on dev and how we use it on production are two different things; you can't realistically test these things in real world conditions without updating production.
As TMB stated, we couldn't get the same behavior without making the site cry in the corner, and this was fairly extensively tested on dev before it went live. For older users to the site, you may remember this is not the first time we've changed comments, and rather predictably, the roll out of Improved Commenting actually was fairly buggy. This is a more drastic update.
Right now, we're going to keep improving and changing things to address as many things as possible. To that extent, there will be a daily article for at least this week if not longer to allow for feedback as we work to make things better. If, at the end of all the tweaking, we can't satisfy the vast majority of folks, a revert remains as an available option. We've built this entire site on listening to the community, and taking their feedback into account. That isn't going to change now. I'm hoping we've earned enough trust from you guys collectively to be allowed to at least experiment for a bit.
I'm going to leave the rest of the article for the dev crew to use. Due to personal real life issues, I'm likely not going to be around much, so if you don't see me, that's why. I have full faith in the staff in helping manage and keep things going.
Hi! I'm martyb (aka Bytram) your friendly neighborhood QA/test guy chiming in with my 2¢ on the upgrade/rollout.
Firstly, I apologize that you are seeing ANY issues with the site upgrade. I took this update very seriously and was, unfortunately, only able to perform about half of the testing that I wanted to see done before we went live. That said, there are some issues that were reported that I had not foreseen, so this has been a learning experience for me, too.
Secondly, I'd like to point out what you are NOT seeing -- the many MANY changes that TMB and PJ made as a result of feedback arising from testing. That said, comments are THE thing that makes this site. It's not the timeliness or fine writing of the stories — as I see it, this site is all about providing a venue for discussion.
Look past the fold for the rest of my comments.
Hi there. Martyb again with an update of our progress on issues arising from the site update. (The new comment grouping and display code was necessitated by huge server loads as well as long delays on constructing and returning highly-commented articles.)
First off, please accept my sincere thanks to all of you who made the time to comment in the prior stories and/or engaged us on the #dev channel on IRC. Really! Thank you for your passion and willingness to provide steps to reproduce and ideas for overcoming the issues that have been found.
ACs: If you access the site as an Anonymous Coward, be aware that we have NOT forgotten you. We are still trying to ascertain what features work best for the most people and are holding off changing (and rechanging and...) settings until we have a better idea of what to change those settings to be. So, please speak up on anything that you continue to find problematic and help guide us to making a choice that works the best for the most.
Scrolling Within a Comment: From what I saw in the reports from Monday, one of the key issues had to do with the scrolling within comments. We heard you. Oh, did we ever! Scrolling within comments was quickly removed and replaced by setting a limit on how long a comment could be submitted. This was especially problematic on mobiles and tablets.
Display Modes: Another of the often discussed issues had to do with "Display Mode." This can be set in your preferences (for logged-in users) and ad hoc when you load a story.
Display Mode - Defaults: If, prior to the release you had chosen "Flat", then you were transitioned to "Flat" (Doh!) If you had anything else as your selection for "Display Mode", you were transitioned to "Threaded-TOS". That mode was intended, as best as we were able to do using only CSS, to replicate the behavior previously supported by the old "Threaded" mode. You CAN change this. Many have reported that changing "Threaded-TOS" to "Threaded-TNG" and setting a lower value for "Breakthrough" (in this mode, "Threshold" is ignored) seemed to do the trick.
Display Mode - ad hoc setting: For the ad hoc case, just load the story as you normally would. Below the actual story text and before the comments is a set of controls. If you are having issues with the current default of "Threaded-TOS" click on that text and change it to "Threaded-TNG". if you find you have way too many icons to click in order to read comments, choose a smaller value for Breakthrough (-1 displays all; in this mode Threshold value is ignored).
Spoiler: Another popular topic of discussion was the way the new <spoiler> tag was implemented. We've heard you, but have not as yet decided on a course of action on how to update its functionality... Stay tuned!
*NEW* and/or Dimming: A surprising (to me at least) number of folks had issues with how we flagged old/new comments. For logged-in users, again go to the "Comments" tab of your "preferences" page, scroll down a little, and there are checkboxes that you can toggle:
Highlight New Comments [ ] Highlight new comments with *NEW* tag
Dim Read Comments [ ] Dim already read comments
Please give those a try and see if that works for you. Our first implementation of "Dimming" was a bit too strong for most folk's liking - this has been reduced so as to be less jarring. As for the "*NEW*" text, there were several positive comments that on mobile devices especially, one could quickly search for the text and rapidly navigate comments to find out what was new. There was a suggestion that uppercase-only looks like YELLING. Yes, it does. On the other hand, whatever text is selected for display has to be a reasonably unlikely string to appear in the normal course of reading comments. (False positives, anyone?)
There were some suggestions on changing the color of the comment title to flag it as new. This sounds pretty simple, but the devil is in the details. We have some in our community who are color-blind and others who have very limited vision, if any at all. For them, any color changes could be well nigh invisible. But it gets worse. On the "Homepage" tab of the "Preferences" page, there are currently 11 different themes that one can choose as your default. Setting a new comment to have a lighter (or darker) title bar would not work across all of those disparate themes.
Chevrons: And as for those chevrons that control the display of a single comment and of a comment tree, yes we heard you. Work is underway to see if we can replace those images with single/double plus/minus characters.
Penultimately, I would like to add a call-out to Paulej72 who took point yesterday (giving TheMightyBuzzard a well-deserved respite) and worked tirelessly into the night to address the issues that were raised.
Lastly, again many thanks to you, our community, who have guided us through this transition. Your feedback matters. We listen and for those who have been following along, I hope you can see the changes and the progress. We continue to strive to earn your trust and support. Thank you!
Dev Note: Currently there is an issue with Flat mode and viewing single comments such as https://soylentnews.org/comments.pl?sid=18223&cid=472653. It just came to our attention and we will be working on it to fix it. This issue will cause you to get a server error. Workarounds are to either switch modes to anything other than Flat or avoid going to single comment views.
Martyb here once again with an update on our progress with the site upgrade.
Our development team (paulej72, TheMightyBuzzard, and NCommander) have been hard at it trying to isolate and quash the bugs that have been reported. Having looked at some of the site source code (Perl) I can attest there are places where the comparison of Perl code to line noise is an apt description. Also, some of the code we inherited was written by, um, creative people who did not write the most readable code. Further the code documents what it does, but is just a wee bit short on the why. Translation: we have an amazing dev team here who have slogged many many hours trying to isolate and correct the issues that have arisen. If you've ever been bleary-eyed after a several-day coding sprint, you have an idea of things. I hereby express my personal thanks to the brain-numbing hard work these guys have put in for this site. And now on to where things stand.
We had an issue with getting a single comment to display correctly in "Flat" mode which appears to have been caused by issues with specifying the correct page it appeared in. Also, there was a rewrite of this code so things should be better, but watch out for regressions.
There are known issues with accessing the site via TOR most likely because we added a very restrictive Content Security Policy.
The new comment viewing modes "Threaded-TOS" and "Threaded-TNG" have been tweaked.
There is a strong voice to replicate the old "Threaded" behavior and it appears that may be feasible, now that we better understand how the community used it in the past. No promises, but it is being looked into.
We are close to making some changes for the defaults for Anonymous Cowards (non logged-in users), so if you have a preference, please speak up and make your voice known.
Oh, we have had reports of seemingly random 503 (Site Unavailable) errors. If you should experience one, please reply to this story with a description of what you were doing and a copy/paste of the entire error message. That will greatly help in our identifying, isolating, and hopefully fixing whatever gremlin is in the gears.
We have not forgotten about replacing chevrons with single/double plus/minus, but had some fires to put out that postponed action on these.
I expect I've left out a thing (or three) — please reply with a comment to (gently) remind us if you see a problem persisting, or if you find something new. it is most helpful to provide your user nickname, the date/time (and timezone), steps taken to cause the problem, and (ideally) suggestions on how you expected it to behave. Reports so far have for the most part been amazingly detailed and helpful — thanks!
Penultimately (I like that word!), I must express my sincere appreciation to the community who has been amazingly supportive and helpful in this transition. One benefit of the upgrade is you should see quicker page-load times on highly-commented stories. Our servers are experiencing a much lighter load to serve up those pages, too. Speaking of servers, I noticed that several of you have renewed your subscription to the site which is the primary way we can afford to keeps the lights on. Please accept my sincere and heart-felt thanks! The "Site News" slashbox has been updated to reflect our current situation.
Lastly, I must express my sincere gratitude to the community. I continue to be amazed at the breadth of knowledge that is freely shared here. Nary a day goes by that I don't learn something new. And many days when I am just blown away. Some long-held ideas have been challenged, and in some cases changed, thanks to what I've read here. Thank you!
Dev Note: Deployed a fix tonight for broken comment links that was due to yesterday's deploy. Alos deployed a partial fix for Flat comments and single comments. TMB will be working on getting it fixed up fully but I thought we needed what we had out now. -- paulej72