SoylentNews is people
SoylentNews
First the good news. I just received word that janrinok, our Editor-in-Chief, is finally out of the hospital and back in his own home! He is very tired and has severe restrictions on his activities but is otherwise in excellent spirits. He very much appreciated the kind thoughts and wishes expressed by the community in our prior stories. It will still be many weeks or months before he can resume his prior level of activities on SoylentNews, but hopes to pop in once in a while to "second" stories that are in the story queue. Please join me in welcoming him back home!
Next, the good news. In janrinok's absence, the other editors have stepped up to the challenge. I'd like to call out chromas, fnord666, mrpg, and takyon who have all freely given from their spare time to make sure we have a steady stream of stories appearing here. I even saw CoolHand pop in on occasion to second some stories! teamwork++
Then, I have to bring up the good news that our development and systems staff have kept this whole thing running so smoothly. Besides the site, there is e-mail, the wiki, our IRC server, and a goodly number of other processes and procedures that make this all happen. That they are largely invisible attests to how well they have things set up and running!
Lastly, the good news. This is what's known in the press as the "silly season". Summer in the Northern Hemisphere means most educational institutions are on break, so less research is done and reported. other ventures are closed or running on reduced staffing levels. In short, the amount of news to draw from is greatly diminished. Yet, even in that environment, the vast majority of the time finds us with a selection of stories in the submissions queue to draw from.
We recently hit a low spot where I combed the web for a couple quick stories I could submit, but that has been the exception rather than the rule. Generally, we look for stories that have some kind of tech-related angle to them. The community has spoken loud and clear that there are plenty of other sites to read about celebrities, politics, and religion. We make a slight nod to politics in so much as it affects technical areas or has large scale ramifications (e.g. a story about President Trump having a meeting with Russian president Vladimir Putin would fit that description). Even then we generally try to keep it down to one story per day.
That said, if you see a story on the 'net that catches your fancy, please send it in! Feel free to draw upon titles listed on our Storybot page, then pop onto IRC (Internet Relay Chat) and simply issue the command ~arthur $code where $code is taken from the second column on the Storybot page.
Community++
Whether you contribute by submitting a story, buying a subscription, writing in one's journal, moderating or making a comment, we continue to provide a place where people can discuss, share knowledge and perspectives, and maybe learn a thing or two, too!
--martyb
(Score: 2) by c0lo on Thursday July 19 2018, @01:37PM (1 child)
You know firehose is one possible solution, but not the only one.
Now, that's an effing yuuge problem.
Perl - jamais couche avec.
How popular would be the idea of some modules in PHP? (and I think of well-written, maintainable ones)
Actually... those are some good ones. Thanks! I don't particularly enjoy (1) and can't afford (4) but... yeah... Thanks.
Mmmm... OK, thanks again. I'll note this runs a bit astray with the idea of "conserving compute and IO" - I'd be looking to those submission one by one, loading each page including the "layouting markup" and navigational structures, when I'd only need content. If I'd ne using (as responsible as possible) a Web-page scrapper for it, I'll finish in discarding lotsa bytes that would not need to be transmitted in the first place.
With the DB structure and if well-written PHP is acceptable, I'd maybe try to implement an API with text/JSON responses to balance a bit the CPU/IO vs bandwidth.
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 2) by martyb on Thursday July 19 2018, @05:49PM
Agreed that the firehose is but one possibility, but just wanted to point out that if you had that in mind, it is FAR from a trivial undertaking to get that back into working shape.
As for PHP, I doubt that would go over well. It's not my call, of course, and if you think you can make a good case for it, please feel free. Just trying to set the right expectations so you don't get your hopes up unnecessarily.
The biggest problem I've had with Perl is the incredibly rich assortment of two-character combinations of punctuation (e.g. "$_") to refer to different, umm, "assemblages" of data... array, hash, positional parameter, etc. That, and there are so many different ways to do the same thing that one could see three different chunks of wildly different-looking pieces of code that would implement the same thing. In my experience with other languages (e.g. C, FORTRAN, etc.)), there are certain, shall I say, 'canonical' ways to do certain things, and once you've figured it out, and see a certain layout of code (i.e. syntax) it is clear what the intended semantics are. I find that more difficult in Perl. But, if you stick with the relatively basic stuff, it's pretty easy to pick up on. At least I found it so.
As for (1)..(4), you're welcome!
Lastly, that is part of the site code that is naturally exposed to dog+world. You may have noticed how I couched my phrasing in terms sufficient to discern a technique to achieve your apparent goal, without posting a clear POC that someone could cut-n-paste. Sometimes, obscurity is a good thing. I mean, you were looking at the list of submissions and did not see that opportunity. We'd like to keep it that way, without having to do backflips in the code to monitor/filter such kinds of access. If/when it becomes a problem, we'll certainly look to stepping things up, but for now there's no need to make additional work on the people here. As long as you are judicious in your retrievals (like, say, keep it to one query every second or two; pulled out my posterior) it will just fit in with our normal traffic. Now, if you want all of the nearly 28K story submissions, then that's another thing altogether. It may be worthwhile to code up a couple more API methods to retrieve what you are looking for. I dunno, would need a more specific idea of what you are looking to achieve. Btw, if you do go down the scraping route, a text based browser can be mighty helpful. Check out something like Lynx, feed its output through AWK, and you should be most of the way there. =)
Hope this has been helpful!
Wit is intellect, dancing.