All Collections
How can I crawl a website?
How can I crawl a website?
Matt O'Toole avatar
Written by Matt O'Toole
Updated over a week ago

Did you know that domain crawling is free on Authoritas? The Content 360 module is designed to provide vital insights into technical performance areas such as meta data, engagement metrics such as Bounce Rate and the backlink profile, all at a page-by-page level.

This kind of detailed information can be available for your site(s) as well as any competitors. To create an automated schedule for your domains, first head over to the Settings part of the platform and click on Crawl Schedule.

Alternate text

Here you will be able to view and edit any already active schedules, as well as setup a new crawl. As an example, let’s say I want to run a fresh crawl each month for every domain that I am monitoring:

Alternate text

Under the ‘New Schedule’ option, I can:

  • Configure the sites I want to include

  • Choose the interval of the crawl, either weekly or monthly (*)

  • Determine specific settings such as where any ignored paths or parameters

  • Select how many pages I want to return in the crawl data, up to a maximum of 20,000

(*) Daily interval is available as a special option, just ask your Client Success Manager.

You can watch this YouTube video from Authoritas, to gain an understanding of each setting you see above and how it impacts on the crawl.

Hitting save will add the schedule to my list and the platform will automatically run these crawls on the first of next month and every month after. The data can then be viewed in the Content 360 module.

If you have any questions, feel free to reach out to your account manager, or get in touch at

Did this answer your question?