posted by
NCommander
on Monday July 21 2014, @07:00PM
from the because-code-bunnies-don't-go-away-until-you-code-it dept.
from the because-code-bunnies-don't-go-away-until-you-code-it dept.
So, on a less serious topic than incorporation, I wanted to open the floor on a possible new feature to Slashcode; the possibility of reading and posting via a standard newsreader. This idea has gnawed on my brain since GoLive, so a few weeks ago, I started coding a proof-of-concept in my spare time to see if the idea is even feasible.
The answer: Yes, it is.
I'm mostly interested in seeing if there's enough people who would like this feature to develop it fully. As of right now, this is a single-shot export of the site, but it would be relatively easy to expand (at least in terms of SN->NetNews. Full bidirectionality is more difficult). For more information (as well as access to the proof-of-concept), read more past the break.
The answer: Yes, it is.
I'm mostly interested in seeing if there's enough people who would like this feature to develop it fully. As of right now, this is a single-shot export of the site, but it would be relatively easy to expand (at least in terms of SN->NetNews. Full bidirectionality is more difficult). For more information (as well as access to the proof-of-concept), read more past the break.
Just to prove this actually exists:
Now, obviously, this isn't a new idea; NNTP (Network News Transfer Protocol) access for the other site was promised WAY back in 2000-ish, but I wanted to see if it could reasonably be done. Because of the quirks (and headache) of running INN (InterNet News), combined with the fact that most discussion forums these days are flat vs. threaded, very few (if any) forums offer access by NNTP. In addition, given our community is very tech-savy, I could see a fair number of users wanting a way to participate in SN without even using their web browser (the NNTP feed looks good with a newsreader on my phone).
These two aspects led me to spend a few weekends reading through RFC 1036, and learning the ins and outs of how NetNews/USENET works, and seeing how it would be possible to interface slashcode to an instance of INN. (I didn't really want to implement an NNTP server that read against our database). I discovered the "easiest" solution was to make slashcode appear as a remote news server accessible via UUCP (Unix-To-Unix Copy), which meant it was simply a matter of writing out a massive spool file, then feeding it into INN with rnews.
After a fair bit of hacking, swearing at perl, and other fun, I was successfully able to export both stories and comments into INN. This opens some interesting possibilities in terms of accessibility to the site. Assuming I continue work on this, it would allow mirroring SoylentNews across the entirety of USENET. From the progress I made thus far, I do think full bi-directionality is indeed possible (with "creative" use of moderated groups on the INN side). There are some headaches, such as some newsreaders (like Thunderbird) handle cancels and supersedes headers poorly (required for times we edit articles), but I do believe the most basic problems are surmountable.
Now, obviously, being a proof of concept, some caveats apply:
Now, obviously, this isn't a new idea; NNTP (Network News Transfer Protocol) access for the other site was promised WAY back in 2000-ish, but I wanted to see if it could reasonably be done. Because of the quirks (and headache) of running INN (InterNet News), combined with the fact that most discussion forums these days are flat vs. threaded, very few (if any) forums offer access by NNTP. In addition, given our community is very tech-savy, I could see a fair number of users wanting a way to participate in SN without even using their web browser (the NNTP feed looks good with a newsreader on my phone).
These two aspects led me to spend a few weekends reading through RFC 1036, and learning the ins and outs of how NetNews/USENET works, and seeing how it would be possible to interface slashcode to an instance of INN. (I didn't really want to implement an NNTP server that read against our database). I discovered the "easiest" solution was to make slashcode appear as a remote news server accessible via UUCP (Unix-To-Unix Copy), which meant it was simply a matter of writing out a massive spool file, then feeding it into INN with rnews.
After a fair bit of hacking, swearing at perl, and other fun, I was successfully able to export both stories and comments into INN. This opens some interesting possibilities in terms of accessibility to the site. Assuming I continue work on this, it would allow mirroring SoylentNews across the entirety of USENET. From the progress I made thus far, I do think full bi-directionality is indeed possible (with "creative" use of moderated groups on the INN side). There are some headaches, such as some newsreaders (like Thunderbird) handle cancels and supersedes headers poorly (required for times we edit articles), but I do believe the most basic problems are surmountable.
Now, obviously, being a proof of concept, some caveats apply:
- The current setup is a single-shot import of the database into INN, and will not be updated regularly
- The main page is read-only for obvious reasons
- No in-line quoting
-
- This is something of an annoying issue as slashcode and traditional USENET quoting styles don't exactly mess. I can modify the export to include one or two levels of quoting on each post, which should help with usability, but this isn't an ideal solution. If I do this, the result will be much closer to email quoting than USENET quoting
- I'm open to discussion on how best to fix this (if at all)
- A couple of posts got mangled through the convert, especially ones that have HTML encodes, or unicode
- I reused the email generator to format both articles and comments, which means that they render "weirdly" when spooled out (they're rendered as spaces instead of ">" characters
All of the above should be considered pre-alpha quality, more like a whiteboard of insanity. If, despite all these caveats, you'd still like to check out the spool for yourself, point your newsreader at nntp-test.soylentnews.org. There are three groups on the server, soylentnews.discuss, just for general discussion for folks who come by, a test group for testing posting, and the spool itself at soylentnews.test.mainpage.
I look forward to hearing your thoughts and comments below.
[UPDATE: Corrected typos and added links]
This discussion has been archived.
No new comments can be posted.
Request For Comments: SoylentNews->Netnews (NNTP) Gateway
|
Log In/Create an Account
| Top
| 39 comments
| Search Discussion
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
(Score: 1) by barrahome on Monday July 21 2014, @07:05PM
Seems nice!
(Score: 2) by Lagg on Monday July 21 2014, @07:16PM
Yes please. I have no comments on the implementation itself since I'm not familiar enough with slashcode and know that protocol translating proxies of any kind are non-trivial to do, but would like to see it nonetheless.
http://lagg.me [lagg.me] 🗿
(Score: 0) by Anonymous Coward on Monday July 21 2014, @07:20PM
I'm fine with just a one-way dump to usenet for archival purposes.
The goal being to capture any knowledge accumulated here so that if this site ever goes "poof" (or individual posts get deleted at the behest of scientology/mafiaa/fbi/etc) there will be other copies out there. No archival system is 100% guaranteed, but regular (real-time?) posts to a well-propagated usenet group gets you maximum bang for the buck.
You can buy "block accounts" at a handful of usenet services out there for dirt cheap (like $10-$25) and they will never expire since their business model is to charge for downloads and posting is free. Use multiple accounts for redundancy (make sure to you aren't just buying from resellers of the same back-end service) and post to a couple of different newsgroups - a dedicated one, perhaps alt.soylent-news, and a couple of more obscure ones just in case.
Crypto-sign them for authenticity and maybe even do some stealthy encrypted posts with a key that's semi-public but not widely published just as a last-resort. There is so much noise on usenet that a couple of hundred encrypted text messages a day won't even be a blip on anyone's radar.
(Score: 2) by Fnord666 on Tuesday July 22 2014, @03:49AM
Would this actually happen though? It's been a long time since I used Usenet but it used to have a fairly restricted hierarchy with only the alt.* groups allowing creation without a formal request, discussion and a vote. Would any other news server actually carry this group?
(Score: 0) by Anonymous Coward on Tuesday July 22 2014, @04:06AM
There are so many crap alt groups that are widely distributed one more won't be a problem.
(Score: 3, Interesting) by VLM on Monday July 21 2014, @07:23PM
I haven't used gnus on emacs since the 90s, this looks interesting. Before than I was a TIN reader. Never got into NN like some people.
I am a little puzzled how you'd stop me from posting as someone else.
Obviously one solution is anything coming in the nntp gateway could be posted as AC, no matter the supplied ID.
Or I could ask for a GPG key for VLM and sign my nntp posts with the special key, so unless someone stole my private key you'd know its me. Which leads to you running a GPG keyserver, seemingly inevitably. Not that thats a bad thing.
I do think its funny that the feature set available on gnus in 1995 far exceeds every web based discussion system I've ever seen so far. You'd expect out of spite or something someone would have replicated something as advanced as gnus on the web.
Integrated with a RSS into NNTP gateway, I could have quite a bit of fun with gnus again.
(Score: 3, Informative) by NCommander on Monday July 21 2014, @07:24PM
As a note, for RSS->NNTP already exists, gmane [gmane.org] offers this for free.
Still always moving
(Score: 2) by VLM on Monday July 21 2014, @07:26PM
I was guessing your hierarchy design would be soylentnews.software soylentnews.devrandom soylentnews.mobile soylentnews.soylent and the like. But whatever.
(Score: 2) by NCommander on Monday July 21 2014, @07:29PM
We could do something like that without too much difficulty via cross-posting. My thought was doing it it based on nexus (i.e., soylentnews.mainpage/soylentnews.meta/soylentnews.diy which would be soylentnews.org/meta.soylentnews.org/diy.soylentnews.org, etc).
Hell, it would be relatively reasonable to do *both*, crossposting is trivial (its the Newsgroups: line, and you can specify as many as you want if its a private server).
Still always moving
(Score: 0) by Anonymous Coward on Monday July 21 2014, @07:33PM
It's a great way to efficiently filter through heaps of content. (Adding "Arduino" to the Hackaday killfile is
FWIW, comp.misc has already been annexed as a Slashdot-replacement.
(Score: 3, Insightful) by weeds on Monday July 21 2014, @07:40PM
I dumped that other site and jumped on the SN bandwagon pretty early. I was involved in some of the very early discussions, planning, and testing. What I'm trying to say is that I'm on-board with SN, _NEVER_ going back. I also admit to being ignorant of news readers (not sure what advantage they have.) So I have to ask...
Is there anyone who can't, won't or doesn't want their Soylent News on a browser: on a desktop, on a laptop, on a notebook, or on a phone? Does this help us get to a wider audience? Does it improve interaction? Does it promote activity in the community? In the end, will it cause more articles and comments to be posted?
Get money out of politics! [mayday.us]
(Score: 3, Interesting) by Nerdfest on Monday July 21 2014, @07:51PM
I'm interested in SN via newsreader. It's just an easier way to get a *really* good interface without a complete site rewrite. Because it's a standard protocol, people can actually choose their interface as well. It works on all device screen sizes without the server having to deal with it.
(Score: -1, Troll) by Anonymous Coward on Monday July 21 2014, @08:25PM
Can I shove my huge, throbbing cock in your tight pucker hole?
(Score: 4, Insightful) by tynin on Monday July 21 2014, @08:32PM
Does this help us get to a wider audience? Does it improve interaction? Does it promote activity in the community? In the end, will it cause more articles and comments to be posted?
Consider it the minimal interface option. Many people do not have a need for so much of the eye candy, and find the functionality of a minimal interface to suit them better. So if it gets made, (and I would love to use it!!!) I hope it isn't... lost behind some unfrequented page.
My suggestion to the dev team:
I think you should have something on the main page that links to each of the protocols that can be used to access this site. For example, there is nothing on the front page telling me I can access the site via .onion, non-DNS specific access, rss, nntp, etc. I think letting new (and old) readers who are also technologist know they have options for accessing the site, it might just drive more traffic to your door.
(Score: 3, Insightful) by frojack on Monday July 21 2014, @08:57PM
This!
But Still don't see me using it due the the arcane nature of news readers being less useful than a simple browser.
If I want simple I just grab the RSS feed, on my phone.
(I still have a news reader installed, but I might look at it maybe once a month).
So few people use NNTP anymore, its becoming the new gopher.
No, you are mistaken. I've always had this sig.
(Score: 1) by Arik on Monday July 21 2014, @10:24PM
That said, I am not at all certain the actual usage would justify the time and effort this would take. It sounds like a really cool idea with a lot of potential, but those do not always pan out in the end.
If laughter is the best medicine, who are the best doctors?
(Score: 0) by Anonymous Coward on Tuesday July 22 2014, @08:40AM
Another advantage of NNTP is peering: If you allow peering, then others can put up their own NNTP servers for Soylent, and thus take off some load from the main server. Probably all the newsgroups should be set to moderated, though, so that any posts have to go through Soylens's servers.
(Score: 3, Interesting) by Aiwendil on Tuesday July 22 2014, @09:09AM
Quite frankly it will increase the attention to information.
You really should give usenet a try - and with that I mean give it a serious try over at least a couple of weeks of daily use (it has a learning curve, and an adjustment curve).
But as I see it the main advantages are that everything will be customizable. For instance I detest interactivity on sites and I belive in having a separation between video and text (completly different styles of how one absorbs such information).
But just to take a few things that are advantages with newsreaders:
* choice of visual representation - some prefer text, some wants stroboscopes and air raid alarms.
* choice of interaction - some want a clickfest, some consider every time one has to reach for a mouse to be a failure (having a single keypress to jump to next thread, or to expand the current thread, or to collapse the current subthread, or to jump back or to the next message/parent/child is _very_ nice, and takes a lot less effort than having to reach for the mouse, scroll the correct button on screen, click the button and then wait)
** Another thing with this is that most newsreaders has a faily complete keyboard control so you often have your hands on the keyboard so there is less of a "break" in the flow of things when jumping between posting and reading.
* sorting of messages - most newsreaders support at least half a dozen different ways of sorting messages, and it isn't that uncommon to add personalized sorting
* highlighting - many newsreaders allow you to set highlights (for instance on topic or on poster [very handy when you just glance the topics of the discussion])
* killfiles/spamfilters - you can normally completly supress (to varying degree) annoying persons (want an AC-filter, perfect, just add its ID to the killfile and it's gone), it also will allow you to get rid of uninteresting things (video, presidential election, $sport-event, assange [and still get the information about wikileaks])
* portability - wouldn't be surprised if there are newsreaders on more platforms that there are browsers
* printability - text prints very nicely.
* screenreaders - since it is a pure text medium it allows for more screenreaders to be deployed.
* braille - works just fine on braille-terminals
* speed - since you are working with a local copy reading has very few delays (causes fewer mental breaks)
* bulk - makes it easier to follow multiple newssources
* search - as long as you have your local copy the only thing that limits your search-options are the tools you have at hand (or can write).
To sum it up: newsreaders are made for the sole purpose of interactive with vast amounts of information with as little interference/annoyance as possible.
(The gain for this site however would be prettier formatting of text, better quoting of messages, and probably more natural interaction for the [even] more information-oriented crowd - so I expect the quality of articles to increase)
(Score: 2) by romanr on Monday July 21 2014, @08:31PM
This tool, although it seems _very_ nice and cool, is not really useful for common user of SN in my opinion. Common user just visits the page, reads the article and comments and moves on. Users don't need special app for that. I'm a huge fan of SN but I wouldn't use such application.
What I do think however, that this App might be very useful for staff and people who often moderate. Such app could have some neat GUI tools for various advanced tasks that are needed.
Another thought about this is, that a mobile app for SN could be useful.
(Score: 2) by cykros on Monday July 21 2014, @08:51PM
I'm not so sure a special application is really required for anyone who uses email from something other than their web browser (and I really hope we haven't slipped into a minority at least on tech sites...). Popping open, say, Thunderbird (or Pine!) and having your emails and Soylent all in the same place is something perhaps more useful to those not already using say, an RSS aggregator, who otherwise have to browse to the site and go through headlines.
But I do agree it shouldn't be pursued to the detriment of, well, most anything. Consider it a "20% time [nytimes.com]" project. I mostly just am really curious to see what'll happen with it if it becomes more wildly successful than most any of us are at this point expecting it to, in part because it is a fairly novel way for an already flourishing community to target the matter of mobile devices and alternative interfaces. The prospect of it becoming even a little bit of a trendsetter seems worth at least poking around at it, or at least that's my $0.02.
(Score: 2) by cykros on Monday July 21 2014, @08:39PM
This has always been a feature I've hoped for for the site, and while I don't think it should necessarily be prioritized over certain others, I'd absolutely be interested in seeing it go forward. Having just checked out the test server, I would say it looks better off the bat than I was expecting for a pre-alpha PoC.
I'm curious to see how the moderation system may end up being integrated, but either way having an alternate means of viewing the site (particularly on mobile devices) does seem enticing enough in its own right.
The other thing that this would open the door to, of course, is integration with an ANSI-capable BBS system, as server packages such as Synchronet handle pretty fluid integration of NNTP feeds directly into their message bases. Anyone up for some Usurper or Barren Realms Elite? At this rate, we'll be ready to party like it's 1992 in no time!
(Score: 2) by GlennC on Monday July 21 2014, @08:54PM
I'd probably not use it, but if it helps someone else then I say go for it. It's not as if I could do anything about it anyway.
Incorporated or not, SoylentNews is still your site as far as I'm concerned. Do what you will with it.
Sorry folks...the world is bigger and more varied than you want it to be. Deal with it.
(Score: 0) by Anonymous Coward on Monday July 21 2014, @08:59PM
I've seen a forum to mailing list gateway that had those issues.
(Score: 3, Insightful) by Jaruzel on Monday July 21 2014, @09:22PM
I can this being a fun thing from a technical POV - but my main concern is that, if it was to be propagated across USENet, how would you handle the reply delays? Some people would only have access to a local ISP run NNTP server, and those things are not known for updating that quickly. I foresee comment threads quickly getting out of sync, if you allow NNTP people to post on threads and then re-import those threads back into the main site.
Personally I'm in a quandary with it - it's a cool 'code hacking' type project, but who really would use it regularly?
-Jar
This is my opinion, there are many others, but this one is mine.
(Score: 2) by Magic Oddball on Tuesday July 22 2014, @11:27AM
Based on the discussion thus far, quite a few people would, including me.
Anyway, I can see a few solutions to the issue just offhand:
1) SN could maintain its own server; the vast majority of newsreaders are set up to support separate servers/accounts because that used to be a common practice among orgs, businesses & universities.
2) If not, SN would hopefully already be directing potential users to the high-quality free Usenet providers like EternalSeptember.org, and in my limited experience, they were good about updating.
3) It might be possible to adapt SN code to optionally sort posts by when the most recent comments were posted, as most newsreaders can. That aspect of newsreaders means a discussion traditionally continues for days or weeks, sometimes being dormant for days and then flaring back up when someone suddenly remembers something they wished to say or a busy regular comments. The ability could make SN much more attractive to people that leave high-quality comments but are short on time.
(Score: 2) by zafiro17 on Monday July 21 2014, @10:40PM
I see a lot of tepid, "yeah, that would be nice" replies. Let me up the ante. Yes, hell yes! This would be like Nerd-nirvana, the Holy Grail of online communication, Shangri-Soylenty-Lah. The best of all possible worlds. Proof there is a god and he loves us and wants us to be happy here in our mother's basements, pecking away at the keyboard in the dim light.
I happen to love Usenet; I've mentioned that on other threads. There's a decent group at comp.misc of ex-Slashdotters, and a dedicated forum would be freaking fantastic. I might just succumb to a massive nerdgasm and pass out from the loss of blood pressure.
For inspiration, check out Paperclip, and NewsPortal by Florian Arnheim. They are both projects that gate web forums to NNTP. Neither would work with Soylent but both have probably overcome some of the challenges and can help point a way through the dark.
I for one think a Soylent-NNTP gateway would be the coolest freaking thing ever created.
So: yes. Yes, please.
Dad always thought laughter was the best medicine, which I guess is why several of us died of tuberculosis - Jack Handey
(Score: 2) by Geotti on Monday July 21 2014, @11:56PM
+1 (even if we are all breaking rule #1 here...)
I'd like to see a bi-directional synchronization, so we people can comment and submit articles through the other protocol, but that would be quite difficult to achieve (authentication-wise) I imagine.
(Score: 1) by Theophrastus on Tuesday July 22 2014, @12:00AM
sure, what the hell. the wider nets the better.
but how is this distinct from "s'qute"? http://squte.com/ [squte.com]
(Score: 1) by tonyPick on Tuesday July 22 2014, @07:42AM
Hell, keep doing the export regularly and I'll use it for reading as it is right now, bugs and all.
One question I do have is: how to handle/report on moderation? Particularly with AC postings
I'm having vague flashbacks to the 90's of long (long) kill-lists to filter out the spam, and while that works for registered users it would be nice to identify Anonymous posts that hit decent moderation and handle those separately from the "-1" junk/spam/troll posts.
Short of special casing out high positive mod AC posts and re-posting these under a different username I can't immediately think of any easy way to do this though (and that'd be possible to gamify anyway).
(Score: 2) by Pav on Tuesday July 22 2014, @11:27AM
*thumbs up*
(Score: 2, Interesting) by lairdb on Tuesday July 22 2014, @01:25AM
It would be a neat trick, perhaps even approaching a hack -- but I'd much rather see the same energy go into refinement and improvement of the RSS feed.
(This is, by the way, the classic weakness of "bazaar" development -- what gets focus is what is intriguing to the skilled and involved, rather than what's most needed, most wanted, most strategic.... (There's a makeable argument that this is also a strength, but it is for-sure a weakness.))
(Score: 0) by Anonymous Coward on Tuesday July 22 2014, @06:32AM
> rather than what's most needed, most wanted, most strategic....
I believe the term you are searching for is "most marketable."
(Score: 1) by lairdb on Thursday July 24 2014, @02:26PM
No, no it isn't. Oddly, I chose the words I used because I meant them in their common meaning.
Nearly all "bazaar" projects are entirely driven by solving the problems that are intriguing to the self-motivated solver. I believe even ESR would agree.
Those aren't necessarily the problems that would be of the most utility to the most users. Or the enhancements that would best position the project for future use, or future development. Etc.
(Score: 2) by maxwell demon on Tuesday July 22 2014, @07:07AM
What exactly is the difference between email quoting and Usenet quoting? As far as I know, both use greater-than signs for quoting, like this:
The Tao of math: The numbers you can count are not the real numbers.
(Score: 2) by zafiro17 on Tuesday July 22 2014, @08:20AM
By the way, it occurs to me that at this point it's probably better to start calling it Soycode. It forked back in February and is never going back, no?
Dad always thought laughter was the best medicine, which I guess is why several of us died of tuberculosis - Jack Handey
(Score: 1, Interesting) by Anonymous Coward on Tuesday July 22 2014, @08:36AM
Maybe the right way to proceed is not to directly make an NNTP interface for Slashcode (Soylcode?), but to make a generic web API to access the SoylentNews database in raw form (that is, get individual articles/comments without any HTML except the one inside the comment itself, and get any sort of useful list information in XML format; probably also provide a posting/moderation interface in the API), and then write a completely separate program that uses this API to convert Soylent content to NNTP. I'd also suggest to make the API available to the public (some features — especially those allowing to post — might be reserved for logged-in user or even subscribers, to prevent misuse; also, there could be restrictions on the number of queries you can do using the API — internal use by Soylent would of course be unlimited).
That two-step process would have many advantages:
* It decouples the NNTP interface from Slashcode internals. So if you ever feel the need to e.g. completely restructure the internal database, you'll not need to rewrite the NNTP gateway as well.
* It would enable others to use the interface for developing other interesting stuff. For example standalone Soylent apps with interfaces optimized for the corresponding interface, or a Firefox extension that tells you if new messages appeared for you even if you're not currently on Soylent News. With an open API, the limit is only the imagination (and programming ability) of Soylent users.
* If some API features are only available for subscribers, that might be an incentive to subscribe (and thus increase the revenue stream of the site). Of course you should still be able to do everything from the web site itself, so the subscribers don't get any additional ability, but only an additional way to access the functionality).
* As a long term goal, you could even consider decoupling the main site presentation code from the Slashcode internals (that is, let Slashcode handle all the mechanics, but let separate code generate the presentation, accessing Slashcode purely through the API).
BTW, I'd suggest that you format NNTP posts as Markdown, since that is highly compatible with the standard Usenet formatting, but supports all the formatting available through the HTML allowed in comments. And if you do so anyway, you could also implement it as option for posting comments in the web interface.
(Score: 1) by DNied on Tuesday July 22 2014, @10:34AM
Without a doubt this will be my favourite way of participating in SoylentNews.
My friend slrn and I will be happy to attend.
(Score: 1) by cubancigar11 on Tuesday July 22 2014, @01:24PM
I suppose it would be better if you created a poll over who is interested in seeing this.
(Score: 2) by Aiwendil on Friday July 25 2014, @11:07AM
Just a thought on the quoting. I see three ways of going about it.
1) Skip it altogether and rely on the clients capabilities
2) Post the latest comment first and the parent (and maybe grandparent) posted below (ie, cause a top-post)
3) inverted 2 (ie, cause a non-formatted bottom-post)
In any case it probably would be useful to look for the blockquote, quote and $> to indicate that quotes already are used (and skip the automatic quoting).
Personally I favour option 1 since most likely the entire "tree" of replies should be available at any time as long as any/all messages must pass through the soyldnntp (and most newsreaders allow for quite easy jumping in the trees)