So, as I write this, day one has officially come to an end. I'm still somewhat in shock over it. Last night when I was editing the database to change over hostnames and such, I was thinking, man, it would be great if we got 100 regular users by tomorrow. Turns out I was wrong. By a factor of ten. Holy cow, people. I'm still in a state of disbelief, partially due to the epic turnout, but also because our very modest server hardware hasn't soiled itself from the influx (the numbers are, well, "impressive" is a way to put it). Anyway, I wanted to do a bit of a writeup of where we stand now, what works, and what doesn't. Check it out (and some raw numbers) after the break! Warning, it is a bit lengthy.
As some of you may have noticed, we've had a couple of hiccups today, but all and all, it was a huge smashing success for day one, and you, the community, made it possible. Here's our version of the good, the bad, and the ugly.
The Good!
Articles, Commenting, and User Profiles: As the basic life blood of any website, these features need to work 100%, be usable for both staff and for the community. As many are aware, we had an issue with story submission; the problem here was that Slashcode has a "Human Confirmation" module which basically went nuts after we launched. That's been fixed, and submissions should work from any user, and from ACs. As for user registration, this is working, but we've had issues with emails either getting placed into the spam bin, or dropping into the nether regions. We're still looking into this, and hope to have it addressed soon. As for user features (referred to as the Zoo in code), things like Friends/Foes/etc. are all working as expected.
Moderation: Of all the bits of Slashcode that needed tweaking, this is the only one that required a rewrite. As it's difficult to test any code that depends on a huge group of users, we went live knowing that this was going to need ongoing massaging. I won't go into the nitty-gritty here, but suffice to say we're keeping a very close eye on it. Basically, modpoints are issued if you're logged in and actively browsing on the site, and expire four hours after being issued. The UI doesn't accurately reflect this, as this change was made a couple hours ago, and the templates don't provide enough granularity to show hours vs. days. For those who are curious, I did a write up the original slash system (and why it had to go) here, and a second one going into details behind the current implementation, and the thought process that went into it here. Comments welcome.
Site Responsiveness: Appears to be good all around, despite the very modest hardware we're running on. Slash does plenty of caching with memcache, and varnish is helping loads. I don't expect any short term issues with us getting overloaded based on our current capacity and the resources we're utilizing. If there's one thing slash excels at, it is at being an incredibly scalable chunk of perl.
Development Portal: Up and running on our local FusionForge installation. Like the main site, this one also having some email issues, but we're manually approving accounts as they come in. The current latest code can be found there, as well as our bug trackers. The SSL issue is known, and we've got a proper certificate waiting to be installed. The git repository for the site is there (as well as mirrored on github) if you want to dive in. Feel free to drop in at ##altslashdot on irc.freenode.net if you have questions.
Statistics: Slash's internal stats engine is mostly working, but is slightly skewed due to Varnish. We can get realistic numbers by combining slash's numbers with those from the host itself. Needless to say, I was impressed.
The So-So!
Journals: On a technical level, these are working as best as we could expect. The problem is there's a lot of missing functionality, either in stuff that you expect to be there, and isn't, or stuff the other site has that we don't. The biggest glaring omission is the missing "Submit for Submission" button. I'm not sure if it's just not existent, or the HTML for it fell off the templates. It would not be the first feature we had to reconnect to the rest of slash. (Next/Prev article was another one). To be honest, overhauling this entire section is on my wish list, and I have some thoughts on this that I will eventually write down, discuss, and perhaps even implement.
Topics: Oh boy ... for a simple feature, this proved incredibly irritating to get to work, resulting in us hosing the site horribly for several hours. In the end, it required implementing the topics one by one in the database by hand then trying to get backslash to cooperate. This feature is somewhat over-engineered due to the fact that every article requires at least one "Topic" and a "Nexus", which is a section of the site which can exist as a subdomain. For the most part, these are implemented, but are all sharing the same icon (the art wasn't quite available by go-live). Expect to see icons sometime soon. The topics.pl page however is glitching out, for reasons I can't completely understand. For the most part, it's functional.
UTF-8: Believe it or not, we do support it (and it was pretty simple to get working, but it's not 100% bug free. This seems to be mostly related to the internal HTML validator slash uses on incoming comments, but the results are mixed based on your operating system, your browser, and the phases of the moon. If you want to see the UTF-8 in actual for yourself, check out here and here.
The Bad!
Search: Mostly hosed. The backend requires Sphinx, and possibly patching MySQL. It seems to be broken on the other site, as well, when I tested it, so at least we're not alone in this department. It does work for searching based on authors/comments by user, thus it was left in place. Just don't expect it to work for general queries right now.
The Skin: We've had ongoing CSS and theming issues since slash was installed. Considering what we had when we started, it's somewhat impressive that it's usable at all. PrestonL and many, many others on IRC contributed CSS fixes, and a proper fix for the user's page went live late today. I need to go through the logs and get an exact list of contributors so I can all give them the proper kudos and put them in the AUTHORS file.
Static HTML Pages: Slash comes with a fair number of static pages, like the About and Help pages. Most of these pages are there, but are either horribly out of date (for example, anything related to slashcode), horribly wrong (moderation), or just nonexistent. Basically, if it is documentation related to the site, the odds are that it's either 404ing or just wrong. Fixing this is mostly a matter of writing them and putting them in the correct places.
The Ugly! Welcome to the top (or perhaps bottom) issues currently plaguing the site today, those that have no easy resolution. Help is especially needed for all the below.
D2: As many people no doubt noticed, we launched with the old D1 system. While perfectly functional, it has a *very* 1997 feel to it, and there are users that do prefer the more dynamic D2 system. This wasn't some conspiracy to launch an entirely Web 1.0 website (which happens to mostly work with NCSA Mosaic), but due to the fact that most of it is missing. We have the actual code and backend stuff, but none of the javascript that makes it work. Furthermore, even if we did, it would be the rather ugly version that had the "floating box of doom" (since that code is from that era). The switch in the UI creeped back in after go-live, but functionally it might as well not be there. I'd like to give the D1 system a SLIGHT facelift (similar to the old Greasemonkey script for Slashdot), yet make sure this site always works correctly. Ideas and volunteers welcome.
Firehose: It's essentially the same story as D2, except I'm not even sure that all the backend code is there for it. If we want it, it's basically a burn and rewrite.
Tags: Unlike the other two, we've got most of bits for them. They're just not exposed anywhere on the skin. It's probably possible to bring them back if the community feels that they add value, but it will require someone who knows HTML, javascript, and Template::Toolkit to do it.
Meta-moderation: This is a case where the implementation itself is questionable. We've got all the code for this, but it's broken. When the other site rolled out firehose, they broke this. It appears all the logic is in place, it just needs minor brain surgery. The problem is, the old moderation system was built on a fundamentally wrong premise that got lucky, and I'm honestly not sure many people ever metamodded in anything approaching significant value. This entire feature needs a rethink in light of the moderation rework.
Theming Engine: Slash's theme engine is an example of evolution, not of intelligent design. Basically, if we want multiple themes, it requires multiple Apache instances, running different vhosts for different Nexuses. If I lost you, the take away is its pretty damn ugly. Its not unsalvagable, most of the themed bits live in the database, and it might be possible to make themes user-selectable (it appears some work was done to implement this in slash before being abandoned).
Backslash: Most of you are probably "WTF is backslash." It's slash's version of an admin panel, and it was clearly never designed for mere mortals. We've smacked most of the bugs out of it, but it remains somewhat awkward to use, and for suadmins, it is possible to actually break the site with a misclick (we've already done it and put slashd in an infinite loop). For the most part, the story editors is "relatively" usable, and I finally smote the long standing 500 error. For end users, the biggest thing on this to know is that the two-man system isn't enforced. Articles will be flagged yellow or red (depending on the author) on the master list, but nothing actually prevents those from going live. Furthermore, there's very little in terms of sanity checking, and it's far too easy to accidentally publish something. I just ask the community bears with us as we fight our backend tools.
Mod_perl: And we get to the 600-pound gorilla in the corner. Slash is tied to mod_perl 1.3 and mod_perl is tied to Apache 1.3. If you know Apache, you can guess the problem. If not, it's basically summed up as this: 1.3 is EOL, and has been for some time. There are no "known" CVEs, which helps, but I think everyone wants us to get to supported server software ASAP. This is compounded by the fact that mod_perl development appears to have ceased, and no version exists thats compatible with Apache 2.4. While not an IMMEDIATE problem, it's something we need to figure out for longer term plans for. There's been talk on going to FastCGI or a similar solution, but no definite long term plans have been made. At least one serious effort was historically made to try and update the codebase to 2.0, and MOST of the Apache binder stuff is relatively self-contained. At least we're not alone on this, the few slashcode sites that we're aware of (aside from the other site, which Netcraft reports running Apache 2.2) are still running 1.3.
Final Remarks, and Server Information, and Day 1 Stats
Wooo, still with me? Good. The long and short of it is we're doing pretty well overall, but there's a lot of work that still needs doing. If you want to jump in, grab the code and join us on IRC, all contributions are welcome. After this post goes live, I'm going on a hiatus to try and get my life/work balance back in order. I hope to return in a week or two to help the dev team digging. As always, I'll be available on IRC should something (like the mod script) needs my attention.
A couple^Wsome^Wlot of people have been asking about the server hardware, who is UID #1 (Anonymous Coward), and what the day one counts were. I'm pleased to announce that we are hosted on Linode, specifically, two Linode 2048s running Ubuntu 12.04 LTS. Average server load during day 1 was between 10-20% percent, spiking occasionally when we had to flush caches out for config changes.
In the last 24 hours, we used over 7 GiB(!) of bandwidth, served to 3630 distinct IP addresses. As of midnight tonight EST, the daily stats email dropped out of slashd, so I'm going to repost most of it here. Please note the bandwidth stats from slash are skewed as it is behind Varnish. For the most part, only the actual HTML of a given page is uncached for logged in users. Enjoy.
SoylentNews Stats for 2014-02-17 UIDs IPIDs Pages total: - - 76164 (2123.8 MB) static total: - - 5722 gstatic total: - - 7827 grand total: 916 3630 78372 (2140.9 MB) secure total: - - 0 total hits: 103059 posts: 267 333 comments: 607 1502 17478 (412.1 MB) index: 868 1818 14931 articles: 738 2298 12508 (573.6 MB) search: 224 568 1841 (31.4 MB) journals: 276 394 1572 (25.2 MB) users: 652 812 5390 (105.0 MB) rss: 55 172 2208 (17.1 MB) other: 43 52 2086 (37.0 MB) formkeys: 1732 rows total comments: 837 posted yesterday submissions: 28 submissions sub/comments: 82.1% of the submissions came from comment posters from this day ------------------------ Yesterday | 2 days ago | 3 days ago Avg Hits Per Article: 962.2| 168.4| 0.0 Avg Comments Per Article: 58.0| 16.4| 6.2 Pages From RSS By Section ------------------------------------------------ Section Pages UIDS IPIDS Main Page 524 36 169 For Main Page Pages IPs Bandwidth Users total: 76164 3600 2123.8 MB 916 index: 14931 1818 644.1 MB 868 comments: 17478 1502 412.1 MB 607 articles: 12508 2298 573.6 MB 738 search: 1841 568 31.4 MB 224 rss: 2208 172 17.1 MB 55 other: 2086 52 37.0 MB 916 ----------------------- Top stories viewed by article.pl: 1885 14/02/17/0148235 Dopefish Welcome to the World of Tomorr 865 14/02/17/1745207 mattie_p What "News for Nerds" Sites Sh 725 14/02/17/0745210 Dopefish Environmentalists Concerned Ab 613 14/02/16/2220240 NCommander Announcing UTF-8 Support on So 608 14/02/16/1331209 NCommander Massive Site Progress - Status 560 14/02/17/145204 LaminatorX Australia spied on US law firm 531 14/02/17/0818240 Dopefish MIT Students Subpoenaed Over T 514 14/02/17/1453216 LaminatorX Find those bugs! 510 14/02/16/2318242 Dopefish AIDS Deniers Use Bogus Copyrig 413 14/02/17/0814201 Dopefish Wondering Where Microsoft Bing 393 14/02/16/0731223 mattie_p Global Warming Finally Debunke 385 14/02/17/180204 Cactus Desalination Project in Califo 292 14/02/17/1734251 mattie_p Coal Mine Fire Threatens Power 292 14/02/16/237205 Dopefish CD Projekt RED Considering The 277 14/02/12/0715245 NCommander Welcome to SoylentNews! 268 14/02/16/0327202 Cactus Kickstarter Hacked 257 14/02/17/1630228 mattie_p 3D Printers for Kids 210 14/02/13/0412219 NCommander Reworking Moderation Access 182 14/02/16/2059230 mattie_p Amazon Electric Fish Inspire U 176 14/02/15/189225 mattie_p Tuna Hearts Damaged by Oil Pol 175 14/02/16/217209 Dopefish How Facebook Can Be Addictive 172 14/02/14/201241 mattie_p Google Buys Lease On NASA's Fa 170 14/02/16/2032248 mattie_p China Plans World's Longest Tu 167 14/02/16/1115247 Dopefish PPACA/Obamacare Website Develo 160 14/02/16/2119211 Dopefish Germany Beefing Up Spy Defense ----------------------- Hour Hits Hits/sec 00 1027 0.29 ######## 01 1144 0.32 ######### 02 1282 0.36 ########## 03 1171 0.33 ######### 04 726 0.20 ###### 05 2891 0.80 ######################## 06 4595 1.28 ###################################### 07 3356 0.93 ############################ 08 3377 0.94 ############################ 09 2940 0.82 ######################## 10 2895 0.80 ######################## 11 2991 0.83 ######################### 12 3622 1.01 ############################## 13 3992 1.11 ################################# 14 3896 1.08 ################################ 15 3853 1.07 ################################ 16 4019 1.12 ################################# 17 4110 1.14 ################################## 18 3881 1.08 ################################ 19 4504 1.25 ##################################### 20 4558 1.27 ###################################### 21 4685 1.30 ####################################### 22 4112 1.14 ################################## 23 4745 1.32 ########################################
(Score: 1) by Pav on Tuesday February 18 2014, @05:15PM
...and... what next? I guess that's up to us as nodes of the Soylent superorganism. :-/ Damnit... responsibility and all that. ;)