Its the moment of truth! Your new website is live, and you need people to visit. So you are probably asking the question, ‘how do I make Google crawl my site’?
We have all seen some film where someone creates a website and ‘puts it live’ and within minutes the laptop ‘dings’ because someone is visiting.
Whilst this is a really fun notion, and whilst we have run many a paid campaign and watched with bated breath as the clicks roll in, it still begs the question:
When it comes to organic owned content, how do I make Google crawl my site and is there anything I can do to speed up the crawling process? This blog post will answer that question.
What exactly is a web crawler?
According to the wonderful Wikipedia, A Web crawler sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing (web spidering).
Search Engine Crawlers will look for URLs on your site and basically put them in the right place, or the place it thinks is right.
Google’s crawler is called the Googlebot. It crawls through websites and adds all the information in a page to an index so that it can find the information later when someone is searching that topic on a search engine.
A search engine only knows how to index a page based on what you tell it. This is a mixture of its content (the words on the page) and the HTML Markup (the code) that is used by the browser to render your page in a way visitors understand it.
In order for Google to index your site (or other search engines), the pages need to conform to web standards. Search engines may choose not to index some pages, for example, if they are not mobile friendly.
Another reason you may be passed over by a search engine crawl is duplicated content. Usually the link juice will be attributed to the canonical url (original url)
How Does Google Crawl Websites?
Crawling is when Google visits your website, with its spider crawler. It’s nothing scary, just a little spider that crawls through all the pages of your website. After crawling takes place, Google Indexes your website.
But what actually is a Google crawl? Simply put, the Googlebot ‘follows a path’ through your website. via a sitemap if you have one, or via its pages and linked pages. This is why you need a really good site structure.
Indexing is the process of adding the pages it crawls to an index. rather like a yellow pages, the index is a vast database of web pages. Google and other search engines can then serve them up when you search online for anything.
First Things First. Where to look for the data.
If you aren’t using Google Search Console (formerly Google Webmaster Tools) to monitor and crawl your web page, then you really should be. There are valuable insights here you cannot afford to miss. This is how you find out what people really type in to find your web pages.
Get yourself a Google search console account and hook your website up to it. This post won’t go into how to do that, because it assumes you will have Google Analytics in place, or you probably won’t be asking questions about Google crawl statistics.
Just know, you will have to verify you are the site owner. This is easy if you have Google Analytics set up on your website. All you need to do is add your domain via the search console and you will be presented with a series of ways to verify. The simplest way is via Google Analytics.
The Google Search Console (A free service) will give you specific data on how often Google crawler visits your website.
It looks like this:
At the time of updating this post. Google Search Console stores the last sixteen months of website data and can give you some amazing insights into your website. How you are being found, what key phrases you appear for, and how often users are clicking on those search results.
You can compare results year on year, or with a similar period. for example you can compare the last three months against the prevous three months.
You can also see a list of URLs or pages that were served for each search term. Even more useful, you can view the top pages that drive traffic to your site, and the search terms that triggered an impression for your landing page. This is a really usefull tool for rooting out keyword opprtunities and seeing just where you could rank, when you don’t already.
Google Search Console in this respect is so much more than an error monitor. It’s a brilliant keyword research tool.
Just How Often Does Google Crawl My Site?
If your perfectionist’s brain is asking, “what is the exact Google web crawler frequency” No one knows. It depends on your domain authority, your backlinks, and any constraints. There are also factors that can waste your crawl budget, such as redirects and pages in your sitemap that no longer exist.
In Google’s Words, their robot spiders “regularly” crawl web content to update its grand index used to create the SERPs (Search Engine Results Page).
These crawls result in changes to the SERPs, which display soon after the index is updated. This can mean you may see changes any time from a few moments to a few days. Again, no one really knows, as its all done by a magic algorithm. Ok, it’s not really magic, but it is quite mystical.
The more visitors you get, typically the more your website will get crawled. Googlebots aren’t going to waste time crawling sites that never get visited.
Can I speed up the process?
Whenever you make a change to a page on your site, for example, if you rewrite it, update the content, or improve on-page SEO, you can resubmit the URL in the search console. Usually, you will get reindexed pretty fast. In my experience, it’s around a minute.
Improvements to a web page can increase its position as soon as it’s re-indexed, so it’s worth bearing this in mind.
Although you have no direct control over when Google crawls your pages, there are a few things you can do to encourage them.
Sitemaps & Error Checking
.An XML sitemap one way to ensure Google Search Console is kept informed of your new content. If you work with a popular content management system such as WordPress, then you can easily generate and submit sitemaps to the search console.
Also, check for mobile usability. You need to know your site is 100% mobile friendly. The image here shows minor issues over time but all have been resolved when spotted.
Page Speed and Connectivity.
If Google can’t crawl your site due to a server error, then it will stop crawling your site. If this happens, your rankings can plummet. To clarify though, we aren’t talking momentary lapses here, but if your site is down for a prolonged period of time, it’s not going to keep sending bots your way.
Benefits of frequent content.
If you add new content to your site frequently, then Google will learn to crawl it more frequently. It’s not rocket science.
It can be hard to come up with new content, but the simplest way to do this is to add a blog to your site and fill it with information related to your industry.
You can blog about anything you like to be fair, but Google gets a fair view of what your site is about if you keep it relevant. For example, we write mainly about Google Ads and SEO insights, How to work with agencies, and anything we have gotten up to in the digital agency space.
Internal and Backlinking
Internal links and incoming links are still a key way to be found on Google. When Google notices that your content is being published and shared by other platforms, it will reward you.
Be careful though and avoid spammy directory listings and black hat techniques. The almighty Google can penalise you as well as reward you, and black marks are hard to come back from.
Search engines love structured data when crawling and indexing. Add Schema markup to your website to boost your positions in the serps when you submit any page for web crawling. You could even show for a featured snippet, which means the coveted position zero.
Do you need help with quality content, or making sure your website is working to benefit your business? Speak to our team today and we can audit your online presence for free.
About The Author: Andi Wilkinson
With over 15 years of experience in web & digital marketing, Andi writes and speaks on a variety of B2B Topics, helping a business go and grow online. Made By Factory is a SEO Agency in Manchester. helping ambitious businesses grow online.