SoylentNews is people
SoylentNews
posted by
NCommander
on Monday July 21 2014, @07:00PM
from the because-code-bunnies-don't-go-away-until-you-code-it dept.
from the because-code-bunnies-don't-go-away-until-you-code-it dept.
So, on a less serious topic than incorporation, I wanted to open the floor on a possible new feature to Slashcode; the possibility of reading and posting via a standard newsreader. This idea has gnawed on my brain since GoLive, so a few weeks ago, I started coding a proof-of-concept in my spare time to see if the idea is even feasible.
The answer: Yes, it is.
I'm mostly interested in seeing if there's enough people who would like this feature to develop it fully. As of right now, this is a single-shot export of the site, but it would be relatively easy to expand (at least in terms of SN->NetNews. Full bidirectionality is more difficult). For more information (as well as access to the proof-of-concept), read more past the break.Just to prove this actually exists:
Now, obviously, this isn't a new idea; NNTP (Network News Transfer Protocol) access for the other site was promised WAY back in 2000-ish, but I wanted to see if it could reasonably be done. Because of the quirks (and headache) of running INN (InterNet News), combined with the fact that most discussion forums these days are flat vs. threaded, very few (if any) forums offer access by NNTP. In addition, given our community is very tech-savy, I could see a fair number of users wanting a way to participate in SN without even using their web browser (the NNTP feed looks good with a newsreader on my phone).
These two aspects led me to spend a few weekends reading through RFC 1036, and learning the ins and outs of how NetNews/USENET works, and seeing how it would be possible to interface slashcode to an instance of INN. (I didn't really want to implement an NNTP server that read against our database). I discovered the "easiest" solution was to make slashcode appear as a remote news server accessible via UUCP (Unix-To-Unix Copy), which meant it was simply a matter of writing out a massive spool file, then feeding it into INN with rnews.
After a fair bit of hacking, swearing at perl, and other fun, I was successfully able to export both stories and comments into INN. This opens some interesting possibilities in terms of accessibility to the site. Assuming I continue work on this, it would allow mirroring SoylentNews across the entirety of USENET. From the progress I made thus far, I do think full bi-directionality is indeed possible (with "creative" use of moderated groups on the INN side). There are some headaches, such as some newsreaders (like Thunderbird) handle cancels and supersedes headers poorly (required for times we edit articles), but I do believe the most basic problems are surmountable.
Now, obviously, being a proof of concept, some caveats apply:
The answer: Yes, it is.
I'm mostly interested in seeing if there's enough people who would like this feature to develop it fully. As of right now, this is a single-shot export of the site, but it would be relatively easy to expand (at least in terms of SN->NetNews. Full bidirectionality is more difficult). For more information (as well as access to the proof-of-concept), read more past the break.Just to prove this actually exists:
Now, obviously, this isn't a new idea; NNTP (Network News Transfer Protocol) access for the other site was promised WAY back in 2000-ish, but I wanted to see if it could reasonably be done. Because of the quirks (and headache) of running INN (InterNet News), combined with the fact that most discussion forums these days are flat vs. threaded, very few (if any) forums offer access by NNTP. In addition, given our community is very tech-savy, I could see a fair number of users wanting a way to participate in SN without even using their web browser (the NNTP feed looks good with a newsreader on my phone).
These two aspects led me to spend a few weekends reading through RFC 1036, and learning the ins and outs of how NetNews/USENET works, and seeing how it would be possible to interface slashcode to an instance of INN. (I didn't really want to implement an NNTP server that read against our database). I discovered the "easiest" solution was to make slashcode appear as a remote news server accessible via UUCP (Unix-To-Unix Copy), which meant it was simply a matter of writing out a massive spool file, then feeding it into INN with rnews.
After a fair bit of hacking, swearing at perl, and other fun, I was successfully able to export both stories and comments into INN. This opens some interesting possibilities in terms of accessibility to the site. Assuming I continue work on this, it would allow mirroring SoylentNews across the entirety of USENET. From the progress I made thus far, I do think full bi-directionality is indeed possible (with "creative" use of moderated groups on the INN side). There are some headaches, such as some newsreaders (like Thunderbird) handle cancels and supersedes headers poorly (required for times we edit articles), but I do believe the most basic problems are surmountable.
Now, obviously, being a proof of concept, some caveats apply:
- The current setup is a single-shot import of the database into INN, and will not be updated regularly
- The main page is read-only for obvious reasons
- No in-line quoting
-
- This is something of an annoying issue as slashcode and traditional USENET quoting styles don't exactly mess. I can modify the export to include one or two levels of quoting on each post, which should help with usability, but this isn't an ideal solution. If I do this, the result will be much closer to email quoting than USENET quoting
- I'm open to discussion on how best to fix this (if at all)
- A couple of posts got mangled through the convert, especially ones that have HTML encodes, or unicode
- I reused the email generator to format both articles and comments, which means that they render "weirdly" when spooled out (they're rendered as spaces instead of ">" characters
All of the above should be considered pre-alpha quality, more like a whiteboard of insanity. If, despite all these caveats, you'd still like to check out the spool for yourself, point your newsreader at nntp-test.soylentnews.org. There are three groups on the server, soylentnews.discuss, just for general discussion for folks who come by, a test group for testing posting, and the spool itself at soylentnews.test.mainpage.
I look forward to hearing your thoughts and comments below.
[UPDATE: Corrected typos and added links]
This discussion has been archived.
No new comments can be posted.
Request For Comments: SoylentNews->Netnews (NNTP) Gateway
|
Log In/Create an Account
| Top
| 39 comments
| Search Discussion
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Lay on, MacDuff, and curs'd be him who first cries, "Hold, enough!".
-- Shakespeare
(Score: 1, Interesting) by Anonymous Coward on Tuesday July 22 2014, @08:36AM
Maybe the right way to proceed is not to directly make an NNTP interface for Slashcode (Soylcode?), but to make a generic web API to access the SoylentNews database in raw form (that is, get individual articles/comments without any HTML except the one inside the comment itself, and get any sort of useful list information in XML format; probably also provide a posting/moderation interface in the API), and then write a completely separate program that uses this API to convert Soylent content to NNTP. I'd also suggest to make the API available to the public (some features — especially those allowing to post — might be reserved for logged-in user or even subscribers, to prevent misuse; also, there could be restrictions on the number of queries you can do using the API — internal use by Soylent would of course be unlimited).
That two-step process would have many advantages:
* It decouples the NNTP interface from Slashcode internals. So if you ever feel the need to e.g. completely restructure the internal database, you'll not need to rewrite the NNTP gateway as well.
* It would enable others to use the interface for developing other interesting stuff. For example standalone Soylent apps with interfaces optimized for the corresponding interface, or a Firefox extension that tells you if new messages appeared for you even if you're not currently on Soylent News. With an open API, the limit is only the imagination (and programming ability) of Soylent users.
* If some API features are only available for subscribers, that might be an incentive to subscribe (and thus increase the revenue stream of the site). Of course you should still be able to do everything from the web site itself, so the subscribers don't get any additional ability, but only an additional way to access the functionality).
* As a long term goal, you could even consider decoupling the main site presentation code from the Slashcode internals (that is, let Slashcode handle all the mechanics, but let separate code generate the presentation, accessing Slashcode purely through the API).
BTW, I'd suggest that you format NNTP posts as Markdown, since that is highly compatible with the standard Usenet formatting, but supports all the formatting available through the HTML allowed in comments. And if you do so anyway, you could also implement it as option for posting comments in the web interface.