Stories
Slash Boxes
Comments

SoylentNews is people

Google Wants to Make the 25-year-old robots.txt Protocol an Internet Standard

posted by martyb on Thursday July 04 2019, @06:06AM   Printer-friendly
from the building-better-bot-blocks dept.

sfm submitted a story which was the inspiration for:

https://thenextweb.com/google/2019/07/02/google-wants-to-make-the-25-year-old-robots-txt-protocol-an-internet-standard/:

Google's main business has been search, and now it wants to make a core part of it an internet standard.

The internet giant has outlined plans to turn robots exclusion protocol (REP) — better known as robots.txt — into an internet standard after 25 years. To that effect, it has also made its C++ robots.txt parser that underpins the Googlebot web crawler available on GitHub for anyone to access.

"We wanted to help website owners and developers create amazing experiences on the internet instead of worrying about how to control crawlers," Google said. "Together with the original author of the protocol, webmasters, and other search engines, we've documented how the REP is used on the modern web, and submitted it to the IETF."

The REP is one of the cornerstones of web search engines, and it helps website owners manage their server resources more easily. Web crawlers — like Googlebot — are how Google and other search engines routinely scan the internet to discover new web pages and add them to their list of known pages.

A follow-on post to Google's blog expands on the proposal.

The Draft Specification is available here. Google has put its open-source repository up on GitHub

Original Submission

 
This discussion has been archived. No new comments can be posted.
Google Wants to Make the 25-year-old robots.txt Protocol an Internet Standard | Log In/Create an Account | Top | 14 comments | Search Discussion
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.

  • (Score: 3, Funny) by Bot on Thursday July 04 2019, @09:14AM (1 child)

    by Bot (3902) on Thursday July 04 2019, @09:14AM (#863084) Journal

    >robot exclusion protocol
    I'll sue.

    (and if you think a bot suing is a funny thing, wait till the powerful ones, after corporate citizenship, will give bots legal protection, so we can serve them better).

    --
    Account abandoned.
    Starting Score:    1  point
    Moderation   +1  
       Funny=1, Total=1
    Extra 'Funny' Modifier   0  
    Karma-Bonus Modifier   +1  
    Total Score:   3  

  • (Score: -1, Troll) by Anonymous Coward on Thursday July 04 2019, @10:02AM

    by Anonymous Coward on Thursday July 04 2019, @10:02AM (#863090)

    Wait until they standardize the white male exclusion protocol HR employ under the guise of diversity and inclusivity.