Stories
Slash Boxes
Comments

SoylentNews is people

SoylentNews is powered by your submissions, so send in your scoop. Only 12 submissions in the queue.
Meta

Help SoylentNews — "Scraping" Web Pages

posted by takyon on Saturday June 04 2016, @03:30PM   Printer-friendly
from the just-a-thought dept.

martyb writes:

Not infrequently, we receive a story submission to SoylentNews that consists of a single URL. Or, a story that contains but a single sentence or paragraph along with the URL. During weekends and holidays, the story submission queue tends to run dry. We have an IRC channel (#rss-bot) that gathers RSS links from around the web. Hmm.

It would be really handy if there were an automated way to "scrape" the contents of that page. In some cases, a simple redirect of the output of a text-based browser like Lynx would do the trick. Unfortunately, all too many sites subscribe to the idea that a web page needs to pull in Javascript and libraries from a myriad of other sites. Failing to do so displays a blank page.

There must be a way to do it — search engines like Google and Bing must extract the page text in order to index it. It would be best to have a general-purpose solution; having a custom template for each site is time-consuming to create and maintain (think if the site changes its layout). Our site is powered by Perl, so that would be the obvious preference.

So, fellow Soylentils, what tools and/or techniques have you used? What has worked for you?

Maybe I'm approaching this the wrong way? When all you have is a hammer... what am I missing here? Is there another approach?

Original Submission

 
This discussion has been archived. No new comments can be posted.
Help SoylentNews — "Scraping" Web Pages | Log In/Create an Account | Top | 53 comments | Search Discussion
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.

  • (Score: 5, Insightful) by Anonymous Coward on Saturday June 04 2016, @06:39PM

    by Anonymous Coward on Saturday June 04 2016, @06:39PM (#355195)

    Don't do it!

    1) If the article isn't good enough for an editor to vet and read, why is it good enough to waste every soylentil's time on?

    2) Scraping JS means soylentils would need to enable JS. Have you /spoken/ to us? When was the last time one of us /wasn't/ running noscript-or-equivalent??

    Starting Score:    0  points
    Moderation   +5  
       Insightful=4, Interesting=1, Total=5
    Extra 'Insightful' Modifier   0  
    Total Score:   5  

  • (Score: 2) by urza9814 on Tuesday June 07 2016, @11:59PM

    by urza9814 (3954) on Tuesday June 07 2016, @11:59PM (#356662) Journal

    1) If the article isn't good enough for an editor to vet and read, why is it good enough to waste every soylentil's time on?

    Seems that they don't need editors, they need article submissions. That's what this is intended to solve. The articles will still be edited as usual, this will just ensure that the editors have more than just a bare link to start with.

    2) Scraping JS means soylentils would need to enable JS. Have you /spoken/ to us? When was the last time one of us /wasn't/ running noscript-or-equivalent??

    The idea is for the SN servers to run the JavaScript so you don't have to.