Robots Exclusion Protocol

Specification

Protocol Definition The protocol language consists of rule(s) and group(s) that the service makes available in a file named "robots.txt" as described in :

Rule:: A line with a key-value pair that defines how a crawler may access URIs. See .
Group:: One or more user-agent lines that are followed by one or more rules. The group is terminated by a user-agent line or end of file. See . The last group may have no rules, which means it implicitly allows everything.

Formal Syntax Below is an Augmented Backus-Naur Form (ABNF) description, as described in .

The User-Agent Line Crawlers set their own name, which is called a product token, to find relevant groups. The product token MUST contain only uppercase and lowercase letters ("a-z" and "A-Z"), underscores ("_"), and hyphens ("-"). The product token SHOULD be a substring of the identification string that the crawler sends to the service (for example, in the case of HTTP, the product token SHOULD be a substring in the user-agent header). The identification string SHOULD describe the purpose of the crawler. Here's an example of a user-agent HTTP request header with a link pointing to a page describing the purpose of the ExampleBot crawler, which appears as a substring in the user-agent HTTP header and as a product token in the robots.txt user-agent line: Example of a user-agent HTTP header and robots.txt user-agent line for the ExampleBot product token. Note that the product token (ExampleBot) is a substring of the user-agent HTTP header

user-agent HTTP header	robots.txt user-agent line
user-agent: Mozilla/5.0 (compatible; ExampleBot/0.1; https://www.example.com/bot.html)	user-agent: ExampleBot

Crawlers MUST use case-insensitive matching to find the group that matches the product token, and then obey the rules of the group. If there is more than one group matching the user-agent, the matching groups' rules MUST be combined into one group and parsed according to . Example of how to merge two robots.txt groups that match the same product token

Two groups that match the same product token exactly	Merged group
user-agent: ExampleBot disallow: /foo disallow: /bar user-agent: ExampleBot disallow: /baz	user-agent: ExampleBot disallow: /foo disallow: /bar disallow: /baz

If no matching group exists, crawlers MUST obey the group with a user-agent line with the "*" value, if present. Example of no matching groups other than the "*" for the ExampleBot product token

Two groups that don't explicitly match ExampleBot	Applicable group for ExampleBot
user-agent: * disallow: /foo disallow: /bar user-agent: BazBot disallow: /baz	user-agent: * disallow: /foo disallow: /bar

If no group matches the product token and there is no group with a user-agent line with the "*" value, or no groups are present at all, no rules apply.

The "Allow" and "Disallow" Lines These lines indicate whether accessing a URI that matches the corresponding path is allowed or disallowed. To evaluate if access to a URI is allowed, a crawler MUST match the paths in "allow" and "disallow" rules against the URI. The matching SHOULD be case sensitive. The matching MUST start with the first octet of the path. The most specific match found MUST be used. The most specific match is the match that has the most octets. Duplicate rules in a group MAY be deduplicated. If an "allow" rule and a "disallow" rule are equivalent, then the "allow" rule SHOULD be used. If no match is found amongst the rules in a group for a matching user-agent or there are no rules in the group, the URI is allowed. The /robots.txt URI is implicitly allowed. Octets in the URI and robots.txt paths outside the range of the ASCII coded character set, and those in the reserved range defined by , MUST be percent-encoded as defined by prior to comparison. If a percent-encoded ASCII octet is encountered in the URI, it MUST be unencoded prior to comparison, unless it is a reserved character in the URI as defined by or the character is outside the unreserved character range. The match evaluates positively if and only if the end of the path from the rule is reached before a difference in octets is encountered. For example: Examples of matching percent-encoded URI components

Path	Encoded Path	Path to Match
/foo/bar?baz=quz	/foo/bar?baz=quz	/foo/bar?baz=quz
/foo/bar?baz=http://foo.bar	/foo/bar?baz=http%3A%2F%2Ffoo.bar	/foo/bar?baz=http%3A%2F%2Ffoo.bar
/foo/bar/U+E38384	/foo/bar/%E3%83%84	/foo/bar/%E3%83%84
/foo/bar/%E3%83%84	/foo/bar/%E3%83%84	/foo/bar/%E3%83%84
/foo/bar/%62%61%7A	/foo/bar/%62%61%7A	/foo/bar/baz

The crawler SHOULD ignore "disallow" and "allow" rules that are not in any group (for example, any rule that precedes the first user-agent line). Implementors MAY bridge encoding mismatches if they detect that the robots.txt file is not UTF8 encoded.

Special Characters Crawlers MUST allow the following special characters: List of special characters in robots.txt files

Character	Description	Example
"#"	Designates an end of line comment.	"allow: / # comment in line""# comment on its own line"
"$"	Designates the end of the match pattern.	"allow: /this/path/exactly$"
"*"	Designates 0 or more instances of any character.	"allow: /this/*/exactly"

If crawlers match special characters verbatim in the URI, crawlers SHOULD use "%" encoding. For example: Example of percent-encoding

Percent-encoded Pattern	URI
/path/file-with-a-%2A.html	https://www.example.com/path/file-with-a-*.html
/path/foo-%24	https://www.example.com/path/foo-$

Other Records Crawlers MAY interpret other records that are not part of the robots.txt protocol -- for example, "Sitemaps" . Crawlers MAY be lenient when interpreting other records. For example, crawlers may accept common typos of the record. Parsing of other records MUST NOT interfere with the parsing of explicitly defined records in .

Access Method The rules MUST be accessible in a file named "/robots.txt" (all lowercase) in the top-level path of the service. The file MUST be UTF-8 encoded (as defined in ) and Internet Media Type "text/plain" (as defined in ). As per , the URI of the robots.txt file is: "scheme:[//authority]/robots.txt" For example, in the context of HTTP or FTP, the URI is:

Access Results

Successful Access If the crawler successfully downloads the robots.txt file, the crawler MUST follow the parseable rules.

Redirects It's possible that a server responds to a robots.txt fetch request with a redirect, such as HTTP 301 and HTTP 302 in the case of HTTP. The crawlers SHOULD follow at least five consecutive redirects, even across authorities (for example, hosts in the case of HTTP), as defined in . If a robots.txt file is reached within five consecutive redirects, the robots.txt file MUST be fetched, parsed, and its rules followed in the context of the initial authority. If there are more than five consecutive redirects, crawlers MAY assume that the robots.txt file is unavailable.

"Unavailable" Status "Unavailable" means the crawler tries to fetch the robots.txt file and the server responds with status codes indicating that the resource in question is unavailable. For example, in the context of HTTP, such status codes are in the 400-499 range. If a server status code indicates that the robots.txt file is unavailable to the crawler, then the crawler MAY access any resources on the server.

"Unreachable" Status If the robots.txt file is unreachable due to server or network errors, this means the robots.txt file is undefined and the crawler MUST assume complete disallow. For example, in the context of HTTP, an unreachable robots.txt file has a response code in the 500-599 range. If the robots.txt file is undefined for a reasonably long period of time (for example, 30 days), crawlers MAY assume that the robots.txt file is unavailable as defined in or continue to use a cached copy.

Parsing Errors Crawlers MUST try to parse each line of the robots.txt file. Crawlers MUST use the parseable rules.

Caching Crawlers MAY cache the fetched robots.txt file's contents. Crawlers MAY use standard cache control as defined in . Crawlers SHOULD NOT use the cached version for more than 24 hours, unless the robots.txt file is unreachable.

Limits Crawlers SHOULD impose a parsing limit to protect their systems; see . The parsing limit MUST be at least 500 kibibytes .