Stories
Slash Boxes
Comments

SoylentNews is people

Web Crawler User-Agent: Genuine or Fake?

posted by martyb on Tuesday July 05 2016, @10:51AM   Printer-friendly
from the what-would-you-do? dept.

isj writes:

Disclaimer: I work on a search engine (findx). I try not to put competitors in a bad light.

Question: Should a web crawler always reveal its true name?

Background: While crawling the web I've found some situations where using a fake user-agent might help. First example is a web site that checks the user-agent in the http-request and returns a "your browser is not supported" - even for robots.txt. Another example is a site that had an explicit whitelist in robots.txt. Strangely, 'curl' was whitelisted but 'wget' was not. I hesitate in using a fake user-agent, e.g. googlebot because it isn't clear what the clueless webmasters' intentions are. It appears that some websites are misconfigured or so google-optimized that other/new search engines may have to resort to faking user-agent.

I'm also puzzled by Qwant because they claim to have their own search index but my personal website (which is clearly indexed when I search in qwant) has never been crawled by a user-agent resembling anything that could lead to qwant. Apparently they don't reveal what their user-agent is: https://blog.qwant.com/qwant-fr/. And there has been some discussion about it: https://www.webmasterworld.com/search_engine_spiders/4743502.htm

This is different from search engines that don't have their own index (eg. DuckDuckGo uses results from Yahoo! and yandex. Startpage uses Google, etc.)

So what do you Soylentils say, is faking the user-agent in webcrawls necessary? Acceptable? A necessary evil?

Original Submission

 
This discussion has been archived. No new comments can be posted.
Web Crawler User-Agent: Genuine or Fake? | Log In/Create an Account | Top | 63 comments | Search Discussion
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.

  • (Score: 5, Insightful) by GungnirSniper on Tuesday July 05 2016, @01:01PM

    by GungnirSniper (1671) on Tuesday July 05 2016, @01:01PM (#370031) Journal

    So what does he do when using a blank User Agent gets endless loops, and non-standard UAs get banned or blocked?

    Aren't you missing out on potential readers when redirecting traffic to a honeypot and reinforcing Google's absolute dominance in the search space?

    Starting Score:    1  point
    Moderation   +4  
       Insightful=4, Total=4
    Extra 'Insightful' Modifier   0  
    Karma-Bonus Modifier   +1  
    Total Score:   5  

  • (Score: 0) by Anonymous Coward on Tuesday July 05 2016, @01:14PM

    by Anonymous Coward on Tuesday July 05 2016, @01:14PM (#370038)

    Ignore the misconfigured website. It is not important to your insignificant crawler.
    The other way around, your insignificant spam crawler which fakes UA is not important to my insignificant website either.

  • (Score: 1, Interesting) by Anonymous Coward on Tuesday July 05 2016, @01:32PM

    by Anonymous Coward on Tuesday July 05 2016, @01:32PM (#370048)

    > So what does he do when using a blank User Agent gets endless loops, and non-standard UAs get banned or blocked?

    Crawl the site multiple times with different UAs. Don't do it back-to-back, or at least not from the same IP address.

  • (Score: 2) by datapharmer on Thursday July 07 2016, @02:54PM

    by datapharmer (2702) on Thursday July 07 2016, @02:54PM (#371270)

    There is no penalty for not sharing, only for lying.