Last year for one of our Data-Driven Digital webinars we talked to Amanda King, SEO Specialist at Optus about crawling and indexation. If you’re asking the question “What does Google see when it crawls my site?” you came to the right place.
Enterprise websites are often large, sprawling things with dark corners people only find out about years later.
In this webinar we go through why it’s important you understand all the dark corners of your website (and how it impacts Google and other search engines seeing the important bits of your site).
Plus, if you’re so inclined, we’ll go through some easy ways to find potential rabbit holes, and other analysis and review to take to your teams or agencies.
Watch the recorded webinar below:
What Does Google See When it Crawls My Site?
Before we begin, let’s dig into what crawling and indexation actually are before moving onto why it matters, what affects website crawling and indexation, what the common issues are, where your website currently sites and how you can improve your crawling and indexation.
What is crawling & indexation?
Basically, Google follows links on your website (and others) to see the Internet. All the time … but it’s got a limited attention span because crawling the entire Internet is tiring and expensive.
So, Google won’t always crawl (read) your entire website, and even if it does, it won’t always index (remember) your entire website.
Google is like a toddler on a sugar high.
If you make it’s life easy, it’ll keep running around finding things, but if you make it challenging, it hits a wall and may not come back, at least for a while.
Why does it matter?
So what? Google only needs to see my website once, right?
Nope!
Just as your website isn’t the same as it was the day it was published, Google changes how it understands the content and relevance of your website. Constantly. This means the most important pages of your website may be crawled and indexed daily.
If you consistently make it difficult for Google to see your website, eventually they may give up. And that could have massive ramifications for your business.
Google needs to see your website in order to understand it and serve it for searches. Why?
Remember Google’s mission: to organise the world’s information.
What affects website crawling & indexation?
Two major factors in how frequently or infrequently your website is crawled (and ranked!) are:
- Popularity
- Staleness
Over the years that’s been considered in algorithm and serving updates like:
So what else do I need to know?
There is an ideal website structure for Google.
Google crawls, indexes and ranks the mobile version of your site.
Javascript makes things more difficult in a few ways
- It requires Google to crawl your page twice
- It (often) increases the weight of your page…
- Which increase the time it takes Google to read one of your pages
What are some common problems?
Faceted navigation
Do you have an eCommerce website where you can filter by everything under the sun … and it creates its own category page? Like, someone wanted to find a blue gaming laptop under $1,000 and that created its own page?
Keep pages out of the index that your customers don’t need to (or shouldn’t) be seeing.
- Thank you pages
- Confirmation pages
- Internal site search results
Page level command issues
Sometimes we tell Google conflicting or unclear things in page level commands in places like:
- robots.txt
- meta robots
- x-robots
- Canonicalisation
Layout based partial indexing
Sometimes, Google will just skip reading parts of your webpage that aren’t directly related to the main topic. This could be sections like:
- Related Topics
- Similar products
Where does my website sit currently?
Google Search Console should tell you what Google is crawling, and a site: search will give you an idea of what they’re indexing.
If they’re wildly different, something’s up.
How can I improve my crawling & indexation?
- Site architecture improvements OR internal linking.
- Fix any Javascript or responsive design issues.
- Make sure your sitemap is up to date.
