Crawler (search robot)

A crawler starts from one page, finds links on it and follows them further. This is how search engines build their index and how SEO programs find broken links and duplicates.

The difference from web scraping is the goal: a crawler goes through a site, while a scraper extracts specific data. In practice, one program often does both.

Well-behaved crawlers read robots.txt, keep a moderate pace and identify themselves in the User-Agent.

Proxy setup for popular programs, for example Screaming Frog and Scrapy, is in the guides section.

How to plan a crawl before you start

Start with the sitemap: its address is often listed in robots.txt on a Sitemap: line. It gives a list of pages without following every link. Then set boundaries: the maximum depth from the start page, the number of pages and the domains the crawler may visit.

Bring addresses to one form before adding them to the queue. Remove tracking parameters such as utm_source, write the host name in lower case, handle the trailing slash the same way every time, and skip filter combinations and endless calendars. Otherwise the crawler visits the same page under hundreds of addresses.

Pace and server responses

For one site, a careful starting point is one request every one to several seconds; if robots.txt has Crawl-delay, take that value. React to status codes: on 429 and 503 wait as long as the Retry-After header says, and if the errors keep coming, stop the crawl.

Conditional requests save both the site's resources and your traffic. Send If-Modified-Since or If-None-Match with the value from the last visit, and an unchanged page returns 304 without a body.

Polite settings in Scrapy

• ROBOTSTXT_OBEY = True: Scrapy reads robots.txt and skips closed sections.
• DOWNLOAD_DELAY and CONCURRENT_REQUESTS_PER_DOMAIN: the pause and the number of parallel requests per site.
• AUTOTHROTTLE_ENABLED = True: the pace adapts to how fast the site responds.
• USER_AGENT: the crawler's name with a contact page, for example MyCrawler/1.0 (+https://example.com/bot).

Together these settings keep the crawler below the site's rate limit without manual pauses in the code.