Adding a site

How to enter a domain and which sites you can add.

The site you want to crawl is first added to the dashboard as a domain. You can add any public site; you do not have to be its owner.

#Add the site

Go to the Sites page in the dashboard, click Add site and type the domain. You do not need to type https:// or www: the address is normalized (lowercase, www. removed, port and path deleted). www.example.com and example.com count as the same site.

#Examples

#What cannot be added

• IP addresses (e.g. 192.168.1.1).
• localhost and local addresses such as .local and .internal.
• Single-label names without a dot; a full domain name is required (example.com).
• Addresses other than http and https.

#After adding

A newly added site can be crawled right away. If you own it, you can verify it to unlock faster and larger crawls. You can delete the site from the Site settings tab; the site and its past crawls are deleted, and tokens already spent are not refunded. A site cannot be deleted while a crawl is in progress.

#Frequently asked questions

Is a subdomain a separate site? Yes. A crawl follows only the addresses on the domain you added, so you need to add blog.example.com as its own site.

Can I add a site I do not own? Yes, and it can be crawled, but the rules for unverified sites apply (a lower speed and page limit).

I typed the domain wrong. If you have not run a crawl yet, delete the site from the Site settings tab and add it again with the right name; with no tokens spent, you lose nothing.