Skip to content
SEO Rapor
Log inStart freeTREN
Sites and crawling › How crawling works

How crawling works

Polite crawling, robots.txt, the sitemap and scope.

The crawl is designed to be polite to the site: low speed, compliance with robots.txt and one crawl at a time.

#Speed

Unverified sites receive 2 requests per second and verified sites receive 5. If the site's robots.txt file has a Crawl-delay, requests are sent at least that far apart. If the site returns 429 (or a 503 with a Retry-After header), the crawler slows down, waits and tries that page at most three times. On temporary server errors (500, 502, 503, 504; e.g. during a deployment) the page is requested again after waiting 5 and 20 seconds; if it still returns an error, it is reported. This waiting is done for at most 20 pages per crawl. Pages that hit the rate limit are marked “rate limited” in the report; because they could not be checked, they are left out of the score and the comparison.

#robots.txt

The crawl reads /robots.txt first. URLs that are disallowed for our bot are not entered; they are not crawled and are dropped from the queue. If the file is faulty or missing, the related checks also show up in the report.

#Sitemap

Sitemaps are read from the Sitemap lines in robots.txt; if there are none, /sitemap.xml is tried. Sitemap indexes are followed. Compressed (.gz) sitemaps are not read. Sitemap URLs are added to the queue after the links in the pages run out; this way, orphan pages that receive no link from any page are found too.

#Scope

  • The crawl starts at the home page and follows the internal links in the pages; each URL is crawled once.
  • Only URLs on the same domain (host) are crawled. External sites are not crawled.
  • Binary files such as images and PDFs are not crawled as pages.
  • When the page limit of the tier is reached, the crawl stops; if URLs remain in the queue, the report says so and the numbers do not reflect the whole site.
  • The timeout for each request is 20 seconds.
  • Besides the pages, robots.txt, the sitemaps and the size of the images used in the pages (with a HEAD request) are also measured.

#After the crawl

At the end of the crawl a PageSpeed measurement is made (Speed tab), the results are saved and an e-mail notification arrives.