Is More Crawling Better for Rankings?

Google Webmaster Trends Analyst John Mueller recently addressed a few very interesting topics about crawling in a hangout chat session. You may have heard the term “crawl budget”, which refers to how many pages Google crawls on your site each day. The number of pages crawled can depend on a variety of factors like the size of the website and how technically sound it is.

Some webmasters are stressing over their crawl budget, but John thinks it’s a whole lot of worrying over something that doesn’t make much of a difference. Here’s why more crawling doesn’t necessarily lead to better ranking.

The Myth Behind Crawl Reoccurrence and Rankings

For years, many webmasters and SEO specialists have believed the crawl budget is a big factor in ranking. The belief was the more frequently pages get crawled by Google the better chance you have of getting top ranking for those pages. Unfortunately, John says this is a myth that far too many people think is a fact.

John outlined some of things webmasters do in an effort to manually improve their crawl budget. Daily sitemap.xml file submissions, index submissions and studying crawl metrics in the Search Console – according to John none of it has a significant impact on ranking.

Google’s take on the matter is pretty straightforward – more crawling is just more crawling. In fact, as long as your site accepts the crawlers, Googlebot is crawling your website virtually everyday, especially if you post content regularly.

Some website owners are actually doing the exact opposite. They are purposely limiting the number of accessible pages on large websites. The goal is to get Google to focus on the most important pages so the crawl budget isn’t wasted.

Yes, pages need to be crawled and indexed to show up in search results. However, the frequency of the crawling isn’t a factor. Your time and energy is better spent improving the quality and usability of your pages rather than making manual tweaks to get every page crawled. Improving page quality with fresh content, social mentions and backlinks is a surefire way to enhance the value for users as well as the “crawl budget.”

Want to build page quality and make it easier for Google to crawl your web pages? The SearchRPM SEO approach addresses the technical foundation of a website first. We’ll make sure pages can be indexed and function beautifully. Next, high quality content, logical navigation and other on-page elements are optimized before we begin outreach to increase your online presence.

It all starts with a FREE SEO Report - get yours today!

By Michael Ramirez
SearchRPM Founder


Michael Ramirez is the Founder of SearchRPM, an Austin, TX based search marketing company that’s well-versed in Search Engine Optimization best practices. You can follow Michael Ramirez on Twitter @openmic0323 or on Google+ to see what he’s up to next.