No, not a famous Linkin Park song. Crawling is the process search engines use to discover your content so that it can be indexed and ultimately found by users.
The process is largely automated and split between site crawls and page crawls.
Site crawls, as the name suggests is when an entire site is being crawled – starting from the home page. It will create a domino effect with the crawler – sometimes referred to as the spider – following the links from the home page and discovering other things.
Page crawls are, more simply, simply a crawl of one page or blog post.
Crawling is one of the many reasons why links are so important – both internally and externally and it is very good friends with indexing.
Indexing is the storing and organizing of content that the crawler found. Basically – it is the process of getting your content into the SERP.
We may earn an affiliate commission through some of the links on our site.