We're back up and running, with a slightly longer than expected downtime. As part of this maintenance cycle, we've installed new varnish configuration files which *should* hopefully fix the long standing issues with logging in, as well as articles not showing up on the main page.
Furthermore, we've dumped the static page generation used by slash in favor of simply varnishing everything. Now anonymous users are on a 5 minute page cache (so new comments and such will take a bit of time to show up, consistent with the other site), which logged in users can bypass the cache and get live access to articles. A couple of things such as comment count are still dependent on slashd's freshen.pl task, so those don't update in real time (yet).
In other news, we've (finally) got a proper development server up at http://dev.soylentnews.org/, running a copy of the production database where we can stage changes and other various things before deploying here. If you want to see what we're up to before we push it live, check it out. As usual, if we make any large scale changes, we'll announce it BEFORE pushing it here.
Furthermore, we've dumped the static page generation used by slash in favor of simply varnishing everything. Now anonymous users are on a 5 minute page cache (so new comments and such will take a bit of time to show up, consistent with the other site), which logged in users can bypass the cache and get live access to articles. A couple of things such as comment count are still dependent on slashd's freshen.pl task, so those don't update in real time (yet).
In other news, we've (finally) got a proper development server up at http://dev.soylentnews.org/, running a copy of the production database where we can stage changes and other various things before deploying here. If you want to see what we're up to before we push it live, check it out. As usual, if we make any large scale changes, we'll announce it BEFORE pushing it here.
This discussion has been archived.
No new comments can be posted.
Site Maintence Complete: 21-03-2014
|
Log In/Create an Account
| Top
| 33 comments
| Search Discussion
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
(Score: 0) by Anonymous Coward on Friday March 21 2014, @09:58PM
Still having problems logging in.
(Score: 2) by NCommander on Friday March 21 2014, @09:59PM
Can you join us on IRC so I can try and find the root cause of this?
http://chat.soylentnews.org/ [soylentnews.org]
Still always moving
(Score: 3, Funny) by skullz on Friday March 21 2014, @10:01PM
Try not logging in as Anonymous Coward.
It worked for me.
(Score: 2) by istartedi on Friday March 21 2014, @10:15PM
Strangely, I appear to have stayed logged-in through the entire maintenance cycle. Is that possible, or did I just forget re-logging in?
Appended to the end of comments you post. Max: 120 chars.
(Score: 2) by NCommander on Friday March 21 2014, @10:33PM
Cookies won't have timed out, and I didn't flush the season database, so you should be fine :-)
Still always moving
(Score: 1) by Tork on Friday March 21 2014, @09:59PM
🏳️🌈 Proud Ally 🏳️🌈
(Score: 0) by Anonymous Coward on Friday March 21 2014, @10:02PM
how has the site traffic been.
(Score: 2) by skullz on Friday March 21 2014, @10:12PM
Fabulous, as always.
(Score: 1, Funny) by crutchy on Saturday March 22 2014, @12:09AM
especially during maintenance... the servers coped with traffic surprisingly well
(Score: 0) by Anonymous Coward on Friday March 21 2014, @10:18PM
I do hate to say it, 'cause "social media" is so off-putting, but at times when the site is down it would be nice if there were an alternate place for status updates. I even tried the linode address (and the li694-22.org somebody registered) before I figured out you guys were on Chat.
Of course, since you control everything, I suppose you could have a really short DNS TTL and just direct the domain to a static page somewhere?
(Score: 2) by NCommander on Friday March 21 2014, @10:24PM
We had to physically shut nodes down because we needed to change their boot options (from Linode's kernel to stock Ubuntu kernels for now), which is somewhat painful but necessary for some features we'll be rolling out in the very near future for site security/performance. Normally we'd just leave apache up redirecting to a static page or something, but in this instance, we had to pull the plug.
Still always moving
(Score: 0) by Anonymous Coward on Friday March 21 2014, @11:10PM
Right (and thanks for the reply) but I'm saying you could let DNS handle some basic redirection for you too, at a higher level -- especially if downtime's going to be more than an hour or two. (Not sure just how low a TTL your registrar/DNS host will support.)
From a user perspective it's nice to see a note that things are okay, rather than thinking that another set of site upheaval drama is in play. Or maybe I'm just a worrywart.
(Score: 2) by Reziac on Saturday March 22 2014, @02:41AM
Maybe just a static page, link it on the main page and call it something obvious, like, uh, Status. You'd only need to change it if SN is down. Could be on a different domain, we don't care where, so long as we can be reassured. :)
Glad things are still going positively. I like it here.
And there is no Alkibiades to come back and save us from ourselves.
(Score: 1) by neagix on Saturday March 22 2014, @08:12AM
this was not there? I am surprised...yeah it's easy to setup, please guys!
(Score: 2) by Reziac on Saturday March 22 2014, @03:09PM
And I'm sure they could automate a basic announcement easily enough (both a status page and a twitter thing, for those who use the dirty bird), and update it with specifics when they get a chance. They seem to be pretty durn competent at all this.
Once a status page is enacted, we who care can bookmark it and if SN is nowhere to be found, at least we'll know what's the problem. Knowing is a whole lot better than worrying.
I think it says a lot that the stories with the most comments are frequently those about the site itself -- guys, you've got something good here, to make us care this much.
And there is no Alkibiades to come back and save us from ourselves.
(Score: 2) by Open4D on Sunday March 23 2014, @04:23PM
Yes, Twitter would be good for that kind of thing.
In this case, the main Twitter feed did a pretty good job, because this was scheduled downtime that had been published as a story on the front page:3 5738624 [twitter.com]
https://twitter.com/SoylentNews/status/4470451308
In one sense, having a Twitter account is like having an RSS feed hosted independently. And for people who already follow other things on Twitter, and don't follow anything else on RSS, it must be pretty convenient that Soylent has this main Twitter feed.
(Score: 0) by Anonymous Coward on Monday March 24 2014, @05:18PM
Hey, thank you! That feed did NOT come up in my searches, nor did NCommander mention it in his reply above.
Glad I checked back in here. :)
(Score: 4, Funny) by FuckBeta on Friday March 21 2014, @11:09PM
Go on, just for fun!
Quit Slashdot...because Fuck Beta!
(Score: 2) by buswolley on Friday March 21 2014, @11:18PM
Well...this is alphasoylent.
subicular junctures
(Score: 0) by crutchy on Saturday March 22 2014, @12:11AM
full of alphasoyles
(Score: 2) by tynin on Saturday March 22 2014, @12:56AM
Sounds like you are seeing a decent speed boost from varnish. Have any stats on what kind of performance boost you get from it?
(Score: 3, Informative) by NCommander on Saturday March 22 2014, @01:18AM
Varnish is pretty damn impressive. When we get things a little more finalized, I plan to write a howto guide on how we varnish :-)
Still always moving
(Score: 2) by tynin on Saturday March 22 2014, @01:21AM
Awesome, glad to hear it!
(Score: 1) by stroucki on Saturday March 22 2014, @12:57AM
Why not just be honest and call it "beta"?
Fuck beta.
(Score: 2) by neagix on Saturday March 22 2014, @08:14AM
Never! Jump from Alpha to Gamma
(Score: 3, Informative) by umafuckitt on Saturday March 22 2014, @02:10AM
You should update your organisation template and put some links here linking back to your site.
(Score: 1) by aristarchus on Saturday March 22 2014, @02:13AM
I missed the whole thing. I would be disappointed, but on the other hand, that is the way it is supposed to be.
(Score: 0) by Anonymous Coward on Saturday March 22 2014, @04:13AM
From a terminology standpoint, the dev.soylentnews.org instance sounds more like an integration or pre-production environment to me. Development is a VM or other local environment where changes are quickly tried and discarded. The pre-production site is where changes are harmonized and tested more thoroughly before being made live to users. Development [wikipedia.org] is an overloaded term.
(Score: 2) by ls671 on Saturday March 22 2014, @04:55AM
Maybe they don't have dev sites. It is this case, it is common to call pre-prod "dev" ;-)
Everything I write is lies, including this sentence.
(Score: 2) by ls671 on Saturday March 22 2014, @05:01AM
Damn, I meant: ;-)
In these cases, it is common to call pre-prod "dev"
Everything I write is lies, including this sentence.
(Score: 0) by Anonymous Coward on Saturday March 22 2014, @05:27PM
Depends on where you come from. Our shop was mainframe-centric for around 20 years before moving into Unix/Linux and Windows servers, and we traditionally divided things up into dev, qual and prod. But keeping 3 environment in sync takes a lot of hard work. The mainframe guys were incredibly disciplined, the Unix admins and mostly Java developers... not so much. As a result in most places qual doesn't look like prod and dev is even further away. Lately I've been only ordering up dev and prod environments, forcing the programmers to trim up dev so they can preview the changes they want to make in prod (of course that often doesn't happen because people are in too much of a rush and the exceptions outnumber the rules when it comes to change control). On some infrastructure projects I've set up a separate "test" environment to mirror prod exactly so that the sysadmins can have someplace to try out *their* changes (like O/S and other platform updates) without having to worry about where things are in the application development cycle.
(Score: 2) by Open4D on Sunday March 23 2014, @04:33PM
From a marketing perspective, this looks bad. I just looked at the front page and the last 5 or 6 stories were showing zero comments. (In fact they have plenty of comments - well, they have as many comments as I would want to read anyway). I don't remember noticing anything like this before.
(Score: 2) by NCommander on Monday March 24 2014, @07:23AM
slashd hung when we updated the backend authetication. Its done this before, but it tends to self-resolve if left alone long enough. I didn't notice, and it wasn't until several hours later when the comment counts were still at zero that anyone else did. Due to the authetication changeover, no one with access to the boxes could kick it. This was a combination of bad design in slash and bad timing.
We're setting up a crontab to auto restart slashd as a stopgap, and we've got everyone in LDAP now with boot kicking permissions to restart slashd should it do this agian.
Still always moving