It’s the moment of truth! Your new website is live, and you need people to visit. So you are probably asking the question, ‘How do I make Google crawl my site’?
We have all seen some films where someone creates a website and ‘puts it live’ and the laptop ‘dings’ within minutes because someone is visiting.
Whilst this is an amusing notion, and whilst we have run many a paid campaign and watched with bated breath as the clicks roll in, it still begs the question:
When it comes to organic-owned content, how do I make Google crawl my site, how often does Google crawl a site, and is there anything I can do to speed up the crawling process? This blog post will answer that question.
How often does Google crawl a site?
It’s the moment of truth! Your new website is live, and you need people to visit. So you are probably asking the question, ‘How do I make Google crawl my site’?
We have all seen some film where someone creates a website and ‘puts it live’, and within minutes the laptop ‘dings’ because someone is visiting.
Whilst this is an amusing notion, and whilst we have run many a paid campaign and watched with bated breath as the clicks roll in, it still begs the question:
In a webmaster hangout, a web publisher queried how fast Google removed pages from the index if they added a noindex nofollow to it. The publisher said they had added noindex, but the page remained in Google’s index. Google’s John Mueller explained with an answer that revealed some interesting facts about how website pages are indexed.
John Mueller revealed that Google crawls URLs at different rates. That’s not surprising. What was surprising was that he said that some URLs could be crawled as little as once every six months!
Here’s precisely what John Mueller said
“I think the hard part here is that we don’t crawl URLs with the same frequency all the time. So some URLs we will crawl daily. Some URLs may be weekly—other URLs every couple of months, maybe even every once a half year.
John Mueller
So this is something that we try to find the right balance for so we don’t overload your server.
And if you made significant changes on your website across the board, then probably many of those changes are picked up fairly quickly, but there will be some leftover ones.
So, in particular, if you do things like site queries, then there’s a chance that you’ll see those URLs that get crawled once every half year. They’ll still be there after a couple of months.
And that’s kind of the average time for us to reprocess/recrawl things. So it’s not necessarily a sign that something is technically wholly broken.
But it does mean that if you think that these URLs should not be indexed at all, then maybe you can kind of back that up and say, well, here’s a sitemap file with the last modification date so that Google goes off and tries to double-check these a little bit faster than otherwise.”
Search Engine Journal published an article on this hangout.
So, when it comes to organic owned content, how do I make Google crawl my site, how often does Google crawl a site, and is there anything I can do to speed up the crawling process to get more website traffic? This blog post will answer that question.
Why Do You Need To Index Your Site?
If you want your site to show up in the search results, it needs to be indexed. Although we have some small business clients who aren’t always looking for SEO with their websites (they use them as a calling card), you still need to be indexed to be found at all.
And you don’t want to run ‘request Google to reindex site’ once. You want Google (and other search engines) to keep doing it.
Search engines like Google don’t just update automatically. They use spiders. Not real spiders. That would be too scary. I’m talking about snippets of code that go forth and ‘crawl’ the web.
What exactly are web crawlers?
I guess I have written this much without even explaining crawl meaning in digital marketing.
The friendly spider’s job is to look for new content and update the already indexed version of your website. New stuff can mean a new page, a new blog, or changes to an existing page.
According to the wonderful Wikipedia, A Web crawler, sometimes called a spider or spiderbot and often shortened to a crawler, is an Internet bot that systematically browses the World Wide Web, typically for Web indexing (web spidering).
Search Engine Crawlers will look for URLs on your site and basically put them in the right place or the place it thinks is right.
Google’s crawler is called the Googlebot. It crawls through websites and adds all the information on a page to a Google index to find the information later when someone is searching that topic on a search engine.
How Does Google Crawl Websites?
Crawling is when Google visits your website with its spider crawler. It’s nothing scary, just a spider crawling through your website’s pages. After crawling takes place, Google Indexes your website.
But what actually is a Google crawl? Simply put, the Googlebot ‘follows a path through your website. Via a sitemap, if you have one, or via its pages and linked pages. This is why you need a really good site structure.
Indexing is the process of adding the pages it crawls to an index. Rather than yellow pages, the index is a vast database of web pages. Google and other search engines can serve them up when you search online for anything.
The search engine is only as good as your website’s code and content. If you want Google to index your site, it needs to conform to web standards. Some pages might not get indexed if they have errors, like not being mobile-friendly. Google may choose not to crawl duplicate content. Gone are the days of having multiple pages with a tweak of only one word. Google will decide which page deserves to be indexed and ignore the others.
Where Will Google Index My Content?
It used to be simple. Just stuff the keyword on the page as many times as possible. But come on. This is ridiculous because it offers no quality to the user! Fortunately, search engines have gotten much cleverer than they used to be.
Thankfully, these days stuffing keywords into your content (and meta tags) will not just be ignored by search engines. It will probably get you penalised. You might even have your site blacklisted and booted out of the serps. Which means you won’t rank for anything. But why?
Because Google’s algorithm is now far more concerned with overall user experience and ‘search intent. – Why is this person searching for a particular search term? Usually, it’s one of four “I want to go, I want to know, I want to buy, or I want to learn” micro-moments specified by Google.
And that’s why indexing is so important. Regular indexing also improves your search results overall. If you want to rank higher in search, try it and see! As the web crawler notes changes to your website, they are updated in Google’s searchable index. The pages will only be added to the index if they conform to webmaster standards and aren’t doing anything black hat.
When the crawler sees changes on your website, it processes those changes, analyses the content and code, and slots it nicely into the index. That’s it, really.
When a user types something into Google, the algorithm decides where to rank your page compared to all the other pages related to those words. I think it’s a bit like the music charts. The more popular a page is, the higher it will climb.
What are the most crucial indexing factors?
We update our site quite often, and as a result, it does get crawled every couple of days, sometimes daily. As a rule, you want frequent crawling. It’s like Google is interested in what’s going on. Check-in with the Google search console to find out when your site was last crawled. Here are some factors that make it better for Google:
Speed. – This means a better experience for your users. The better the experience, the higher your site will rank. The faster a place, the easier to index. I use SEMRush to audit websites, and if a page is slow, sometimes it times out with SEMRush. Google doesn’t have all day, so make sure your pages are fast.
The other way you can help is by submitting your sitemaps. But I will go into that in more detail later. Let’s get to the steps of getting your site indexed.
Step 1: Am I indexed already?
Quickly check by typing site: followed by your domain into Google. This will tell you if your pages are on Google and how many are there.
Step 2: Set Up Search Console and Google Analytics.
If you aren’t using Google Search Console (formerly Google Webmaster Tools) to monitor and crawl your web page, you should be. There are valuable insights here you cannot afford to miss. This is how you find out what people really type in to find your web pages.
Get yourself a Google search console account and hook your website up. This post won’t go into how to do that because it assumes you will have Google Analytics in place, or you probably won’t be asking questions about Google crawl statistics.
Just know you will have to verify you are the site owner. This is easy if you have Google Analytics set up on your website. All you need to do is add your domain via the search console, and you will be presented with a series of ways to verify. The simplest way is via Google Analytics.
The Google Search Console (A free service) will give you specific data on how often Google crawler visits your website.
It looks like this:
Google Search Console stores the last sixteen months of website data and can give you some fantastic insights into your website. How you are being found, what key phrases you appear for, and how often users click on those search results.
You can compare results year on year or with a similar period. For example, you can compare the last three months against the previous three months.
You can also see a list of URLs or pages served for each search term. Even more helpful, you can view the top pages that drive traffic to your site and the search terms that triggered an impression for your landing page. This is a handy tool for rooting out keyword opportunities and seeing where you could rank when you don’t already.
Google Search Console, in this respect, is so much more than an error monitor. It’s a brilliant keyword research tool.
If you have a WordPress website, install Google Site Kit. It will take care of all of this for you in a few easy clicks.
Step 3: Create a content marketing strategy & Blog
Most website owners just assume that it’s enough to add content and put some keywords in. it isn’t. You need to build a content strategy around search engine results if you want to be found and indexed.
Make a plan of your content based on keyword research, keyword difficulty and search volumes. It’s a great starting point. When I started my digital business, I was also under the impression that I just needed to add content. As a digital agency, we have a range that covers our projects and even company news.
As time went on, I learned how few people who don’t know anything about your business are interested in your company news. You need to create content based on topics users search for and within your niche. Don’t start writing posts about unrelated topics.
A recent HubSpot study suggested that 40% of marketers say content marketing is integral to their overall marketing strategy.
Try and publish informative, valuable and exciting content. No one cares if your company just turned 10.
Use Infographics like this 2020 content marketing infographic
When you create content, look to make it not just for yourself but guest-post on other reputable sites. Conversely, don’t link back to your site from any spammy sites. Google will just assume you are spammy.
We recommend Neil Patel’s content strategy. Here’s his framework.
- What are your goals? Specify SMART goals and how you’ll measure your progress (i.e., metrics).
- Who is your target audience? Customer profiles or personas are essential to understanding your audience and what they want/need.
- What types of content will you produce? You want to make sure you’re delivering the kind of content that your target audience wants to see.
- Where will it be published? Of course, you’ll be hosting your own content on your website, but you may also want to reach out to other sites or utilise platforms such as YouTube, LinkedIn, and Slideshare.
- How often will you publish your content? It’s far better to produce one well-written, high-quality article a week consistently than to post every day and then publish nothing for a month. Consistency is key.
- What systems will you adopt for publishing your content? Systems are basically just repeatable routines and steps to complete a complex task. They’ll help you save time and write your content more quickly to stay on schedule. Anything that enables you to publish content in less time without sacrificing quality will improve your bottom line.
- Include the blogging/content tools and technology you’ll use and how they fit into your system.
Once you have your strategy in place, start writing. You have to commit to it, and it takes a lot of effort. Especially if you aren’t a big company with a team of content writers. I’m sitting at home on a Saturday afternoon, updating this blog post. But this is one of my most visited pages, so It’s worth the time it takes to drive traffic to our website.
You should blog even if you have an eCommerce website. If you have new products, write about them. Can you write about lifestyle topics related to your products? For example, if you have a website selling health supplements, write about healthy lifestyle topics, and link back to your products.
Step 4: Have a linking Strategy.
Internal links, which means linking to other pages on your website, are the best white hat SEO quick wins I know. Firstly, you will have lots of links in your website navigation. Your website structure is essential, and topics and themes should be grouped together.
Make sure you have friendly URLs. I mean that they shouldn’t be too long, and they should contain words, not gobbledygook.
Use a sidebar in your Blog to link to your most popular pages and posts. You can even create manually related topic links such as Related Posts: Why Your Website Structure Matters.
This type of link-building encourages the Google bot to come back and recrawl. It also keeps users on your site for longer and provides them with (hopefully) more helpful content.
Step 5: Leverage Social media
If you don’t have social media profiles for your website, do it now. One component of 2020 search engine optimisation is paying attention to social media signals. Despite being ‘Nofollow’, they seem to add some weight to your website ranking.
Set up Facebook pages, Twitter profiles, LinkedIn profiles or company pages, Pinterest, YouTube channels, and especially Google My Business pages. If your business has separate brands or projects, it can also be worth setting up social media profiles.
And it goes without saying that if you create these profiles, you need to keep them up to date,
Just How Often Does Google Crawl A Site?
If your perfectionist’s brain asks, “What is the exact Google web crawler frequency” No one knows. It depends on your domain authority, your backlinks, and any constraints. Some factors can waste your crawl budget, such as redirects and pages in your sitemap that no longer exist.
In Google’s Words, their robot spiders “regularly” crawl web content to update its grand index to create the SERPs (Search Engine Results Page).
These crawls result in changes to the SERPs, which display soon after the search engine database is updated. This can mean you may see changes from a few moments to a few days. Again, no one really knows, as it’s all done by a magic algorithm. Ok, it’s not really magical, but it is pretty mystical.
Typically, the more visitors you get, the more your website will get crawled. Googlebots aren’t going to waste time crawling sites that never get visited.
Can I speed up the process?
Whenever you change a page on your site, for example, if you rewrite it, update the content, or improve on-page search engine optimization, you can resubmit the URL in the search console. Usually, you will get re-indexed pretty fast. In my experience, it’s around a minute.
Improvements to a web page can increase its position as soon as it’s re-indexed, so it’s worth bearing this in mind.
Although you have no direct control over when Google crawls your pages, there are a few things you can do to encourage them.
Sitemaps & Error Checking
.An XML sitemap is one way to ensure Google Search Console is kept informed of your new content. If you work with a popular content management system like WordPress, you can quickly generate and submit sitemaps to the search console.
Also, check for mobile usability. You need to know your site is 100% mobile-friendly. The image here shows minor issues over time, but all have been resolved when spotted.
Page Speed and Connectivity.
If Google can’t crawl your site due to a server error, it will stop crawling your site. If this happens, your rankings can plummet. To clarify, though, we aren’t talking about momentary lapses here, but if your site is down for a prolonged period, it’s not going to keep sending bots your way.
Benefits of frequent content.
If you add new content to your site frequently, Google will learn to crawl it more often. It’s not rocket science.
It can be hard to come up with new content, but the simplest way is to add a blog to your site and fill it with information related to your industry.
You can blog about anything you like to be fair, but Google gets a fair view of what your site is about if you keep it relevant. For example, we write mainly about Google Ads and SEO insights, How to work with agencies, and anything we have gotten up to in the digital agency space.
Internal and Backlinking
Internal and incoming external links are still key ways to be found on Google. When Google notices that your content is being published and shared by other platforms, it will reward you.
Be careful, though and avoid spammy directory listings and black hat techniques. The almighty Google can penalise you and reward you, and black marks are hard to come back from.
The Robots.txt file.
This is a simple text file that should live in your root folder. Occasionally it’s handled by the host, and you can’t edit it, but if you have WordPress and a plugin like Yoast or RankMath, you should be able to access it directly there.
This file gives instructions to crawlers about which pages to crawl and which to ignore. Again, the aforementioned WordPress plugins can take care of that for you too. If you wonder why it’s good to deindex pages, that’s another topic, but not every page is worth Google’s attention. We don’t index our categories and tags because they can cannibalise our deliberate content.
If you are unsure how to access this file, or configure it, speak to an expert. We have experienced WordPress developers and can help with this.
Structured Data
Search engines love structured data when crawling and indexing. Add Schema markup to your website to boost your positions in the SERPs when you submit any page for web crawling. You could even show up for a featured snippet, which means zero coveted position.
Do you need help with quality content or making sure your website is working to benefit your business? Speak to our team today, and we can audit your online presence for free.
You might also be interested in the following:
get high quality backlinks from Google
About The Author: Andi Wilkinson
With over 15 years of experience in web & digital marketing, Andi writes and speaks on various B2B Topics, helping a business go and grow online. Made By Factory is an SEO Agency in Manchester. And PPC Agency in Manchester assists ambitious companies to succeed online.
