robots.txt
robots.txt is a file in a site's root where the owner tells robots which sections they may crawl.
Updated · Prokso editorial team
The file is located at /robots.txt, for example example.com/robots.txt. It contains rules for robots: which sections are closed, what pause to keep between visits, where the sitemap is.
robots.txt is a request, not a lock. But well-behaved crawlers and search engines follow it, and site owners count on that.
Before collecting data, it is worth reading both robots.txt and the site's terms. Keeping to the stated pace protects the site from overload and you from rate limits.
How to organize the collection of public data through proxies is explained on the proxies for scraping page.
Directives and how they work
| Directive | What it does |
|---|---|
User-agent | names the robot the group of rules applies to; * means all |
Disallow | closes paths that start this way; an empty value closes nothing |
Allow | opens a path inside a closed section |
Sitemap | the full address of the sitemap |
Crawl-delay | a pause between requests; not part of the standard, some robots ignore it |
Since 2022 the format has been described in RFC 9309. Paths support two special characters: * matches any sequence of characters and $ marks the end of the address. When Allow and Disallow conflict, the rule with the longer path wins, so Allow: /shop/public/ works inside Disallow: /shop/.
Which group of rules applies to your robot
A robot looks for a group with its own name in User-agent, comparing names without regard to case. If such a group exists, only it applies, and the rules for * are ignored. If there is none, the robot follows the * group.
Example: a file with User-agent: * and Disallow: /search, followed by User-agent: MyCrawler and Disallow: /, closes the whole site to MyCrawler and lets everyone else visit everything except search. That is why a crawler should have its own name and check whether there is a separate group for it.
What robots.txt does not cover
The file works for one host and protocol: shop.example.com has its own robots.txt, separate from example.com. If the file returns 404, robots usually treat everything as allowed. If it returns 5xx errors, the RFC advises treating the site as fully closed for the time being, and a cached copy of the file should not be used for more than 24 hours.
robots.txt also does not replace a site's terms of use: a section can be open to robots but closed by the rules. The use of Prokso proxies is governed by the acceptable use policy.
FAQ
Do I have to follow robots.txt?
Technically the file blocks nothing, but it is an explicit wish of the site owner. Well-behaved robots follow it.
What does Crawl-delay mean?
A pause in seconds between requests that the owner asks you to keep. Not all search engines respect it, but for your own data collection it is a handy guide.
Residential $1.59, mobile $2.50, datacenter $0.65 per GB
Sign up in a minute, get access right after payment.