Skip to content
SEO Rapor
EN
Documentation

How crawling works

Polite crawling, robots.txt, the sitemap and scope.

The crawl is designed to be polite to the site: low speed, compliance with robots.txt and one crawl at a time.

#Speed

Unverified sites receive 2 requests per second and verified sites receive 5. If the site's robots.txt file has a Crawl-delay, requests are sent at least that far apart. If the site returns 429 (or a 503 with a Retry-After header), the crawler slows down, waits and tries that page at most three times. On temporary server errors (500, 502, 503, 504; e.g. during a deployment) the page is requested again after waiting 5 and 20 seconds; if it still returns an error, it is reported. This waiting is done for at most 20 pages per crawl. Pages that hit the rate limit are marked “rate limited” in the report; because they could not be checked, they are left out of the score and the comparison.

#Bot protection

If the site returns a challenge or block page from a bot protection service such as Cloudflare instead of the page, that response is not treated as content: it is marked “could not be checked” and not reported as the site's error. If blocks come one after another the crawl slows down, and if the block continues it ends early. If the home page is blocked, the crawl counts as failed, the token is refunded and the report names the protection provider and explains how to allow our bot.

#robots.txt

The crawl reads /robots.txt first. URLs that are disallowed for our bot are not entered; they are not crawled and are dropped from the queue. If the file is faulty or missing, the related checks also show up in the report.

#Sitemap

Sitemaps are read from the Sitemap lines in robots.txt; if there are none, /sitemap.xml is tried. Sitemap indexes are followed. Compressed (.gz) sitemaps are not read. Sitemap URLs are added to the queue after the links in the pages run out; this way, orphan pages that receive no link from any page are found too. If the sitemap has more URLs than the page limit (e.g. the 100-page preview or a small tier), the limit is shared: about half is crawled by following links from the home page and half from the sitemap URLs; if one side runs out, the other uses the rest. URLs taken from the sitemap are spread evenly across the files, most recently modified first.

#Scope

  • The crawl starts at the home page and follows the internal links in the pages; each URL is crawled once.
  • Only URLs on the same domain (host) are crawled. External sites are not crawled.
  • Binary files such as images and PDFs are not crawled as pages.
  • When the page limit of the tier is reached, the crawl stops; if URLs remain in the queue, the report says so and the numbers do not reflect the whole site.
  • The timeout for each request is 20 seconds.
  • Besides the pages, robots.txt, the sitemaps and the size of the images used in the pages (with a HEAD request) are also measured.

#After the crawl

When the pages are done, a verification round runs: URLs that failed (rate limited, no response, 5xx, 403, 404/410) and anyrobots.txt or sitemaps that could not be read are requested once more, slowly, after a short wait; slow pages are timed a second time. A result that recovers counts as temporary and produces no finding; if it does not recover, the finding is written with the result of both attempts. The round is limited to 60 URLs and150 seconds; URLs that could not be retried stay at info level as “could not be verified”.

Then a PageSpeed measurement is made (Speed tab), the results are saved and an e-mail notification arrives. PageSpeed requests come from Google's own measuring tool and do not carry our bot's name.