In recent years, Google has shifted its crawling strategy to focus on indexing and ranking high-quality, valuable pages that best serve searchers.
This is a change from Google's previous approach of attempting to crawl everything on the internet to build as comprehensive an index as possible.
The main driver behind this change is the realization that Google's index had become polluted with low-value pages that provide little usefulness to searchers.
With advancements in artificial intelligence (AI) and machine learning, Google has gotten better at determining the true value and utility of a web page.
Google now aims to crawl less but crawl smarter. Rather than trying to digest every bit of content on the internet, Google selectively identifies and indexes pages that score highly on what they term a "value index."
This value index allows Google to focus its crawling budget on pages it believes searchers are most likely to find useful and engaging.
The introduction of the value index represents a fundamental shift in how Google views both crawling and indexing. Google is moving away from quantity and more towards quality and value.
For website owners, this means a new focus on creating high-quality content while reducing or eliminating less useful pages. Tactics like keyword stuffing and thin content are unlikely to pass Google's value filter.
In the following sections, we will explore how website owners can align with Google's new focus through steps like refining redirects, consolidating canonical tags, and performing index clean-up. The goal is to improve a site's value index score and visibility in Google search.
Google's value index is a metric that aims to quantify the usefulness and quality of web pages in Google's search index. It helps Google determine which pages are valuable to searchers and should be prominently returned in search results.
The value index works by analyzing various signals related to individual web pages, including estimated clickthrough rate (CTR), dwell time, bounce rate, and external links.
Pages that perform well on these value metrics get a higher value index. Pages with a low value index may be demoted or not indexed at all.
Optimizing for Google's value index involves minimizing low-value pages and focusing on content that are helpful for users.
This includes reducing redirects, canonical tags, and thin or duplicative content. Sites with bloated indexes containing many low-value pages dilute Google's ability to effectively crawl and index their most useful content.
By cleaning up low-value pages and concentrating on publishing high-quality content likely to rank well and satisfy searchers, you can improve your site's visibility in Google search.
Pages with unique and helpful content tend to earn higher value index scores. As Google indexes fewer but more valuable pages for a site, it can better serve users with relevant and authoritative results.
The relationship between number of indexed pages and value index is inverse. Having more pages indexed doesn't directly translate to higher visibility.
Websites should aim to have fewer pages in Google's index, but those pages should be filled with really helpful content that Google likes. This helps Google use its resources better.
As websites get bigger over time, they can end up with lots and lots of pages. Sometimes, these pages don't really have useful content anymore.
This is a big problem for big websites large eCommerce sites, online publishers, wikis, and other sites with extensive content.
When a website has too many unimportant or repeated pages, it causes problems for search engines like Google.
They have to spend a lot of time and resources looking at these pages, even though they're not helpful to users. This makes it harder for Google to focus on the pages that are actually useful.
If you too many pages on your site, this can cause issues with your website's "crawl budget."
This means that if Google is busy looking at unimportant pages, it might miss out on checking new or updated important content. This could mean that important updates or new information on the website might not show up in Google's search results on time.
Cleaning up bloated indexes is critical for large sites to create a lean, focused set of pages for Google to crawl.
Eliminating unnecessary pages through consolidation, removal, or disallowing via robots.txt improves overall site performance.
With a refined index, Google can allocate more crawl budget to pages that truly matter, keeping them up-to-date in the search results.
Think of it like weeding a garden. By getting rid of the weeds (unnecessary pages), there's more space for the important stuff to grow.
It makes the site architecture messy and harder to navigate for both users and search engines. There are unnecessary hops taking place before reaching the final content.
Too many redirects can slow down site performance since there are extra lookups required before resolving to the final page.
The redirect trail may pass PageRank and anchor text signals in ways you don't intend. This dilutes the signals that should go to the final page.
Search engines may index multiple versions of URLs pointing to the same content. This creates duplicate versions that split page metrics and confuse search engine crawling.
Adding too many unhelpful pages makes it tough for search engines to find your important ones.
Excess redirects can indicate a site redesign wasn't properly executed from an SEO perspective. It suggests the redirects weren't well planned and properly implemented.
For large sites with many redirects, it can be challenging to determine which redirects are still providing value and should be kept.
While comprehensive clean-up of unnecessary redirects is important for site health, there are certain instances where redirects should persist.
If an old URL has picked up links from external sites, it is usually beneficial to keep redirecting that URL. Removing the redirect means lost link equity from those third-party sites. Check to see if any external sites are still linking to the old URL using a backlink analysis tool. The presence of third-party links is a good indicator that the redirect should remain in place.
Look at your Google Analytics data to identify redirects still generating a meaningful level of traffic. If users are still arriving at certain pages via redirected URLs, you don't want to cut off that traffic flow. Decide if a redirect is worth keeping based on its traffic. For example, keep redirects with over 100 users per month.
Removing old redirects is important during site cleanup, but consider external links and traffic flow to identify valuable ones. Redirect optimization is about balance, not just wholesale removal.
Internal redirects can accumulate over time as content gets moved around on a site. This creates duplicate versions of URLs that can confuse search engines. Luckily there are tools that can identify these redundant internal redirects to clean them up.
One of the best tools for finding internal redirects is Screaming Frog SEO Spider. This crawler allows you to enter your website and crawl all the pages. Once it finishes crawling, you can view a list of all internal redirects discovered.
Enter your website URL into Screaming Frog and start the crawl. Make sure to enable the "Fetch redirect chains" option.
When the crawl finishes, go to the Redirects tab. This will display a list of all internal redirects.
Sort this list by the number of referring URLs to find which redirected URLs are referenced most on your site. These should be high priority to fix.
For each redirected URL, make note of what URL it is redirecting to.
Use your site's CMS to find all pages linking to the old redirected URL and update them to point to the new URL instead.
Once all references are updated, you can remove the old redirected URL either by deleting the page or updating the redirect to send users and search engines to the appropriate new URL.
Continue this process for all high volume internal redirects until you have cleaned up the majority on your site. Removing the highest referring redirects will have the biggest impact.
Regularly repeating this redirect cleaning process will keep your site's index focused on the URLs you actually want indexed, avoiding wasted crawl budget on duplicate versions.
External redirects that send users away from your site should be closely analyzed to determine if they are still providing value.
Some redirects to other websites can be helpful, like sending users from an old product page to the updated one on a different site. But often, redirects pile up just because the site changes over time.
Old redirects to external sites can mess up the user experience by sending people away from your site when they don't need to go. Plus, they spread out the value of your links instead of keeping it focused within your own site.
Is the redirect generating any referral traffic?
Check Google Analytics to see if the URL has sent any users to your site recently. If the redirect is not leading visitors into your site, it likely can be removed.
Does the redirect have any external links pointing to it?
Use a backlink analysis tool like Ahrefs to determine if other sites are still linking to the redirected URL. Keep redirects that pass link equity.
Is the redirect still relevant for users?
Determine if the redirect provides a logical, user-friendly path. For example, redirecting an old product URL to the new product location may be useful, but random old redirects likely have little purpose.
How old is the redirect?
Redirects that have been in place for years without providing traffic or links are clear candidates for removal.
You can make your website more cohesive and give users a better experience by checking your old redirects now and then. Get rid of the ones that don't really help with SEO or make the site easier to navigate.
When cleaning up redirects, follow these best practices:
As this guide has shown, Google is focused on crawling and indexing pages that demonstrate value, trying to filter out lower quality pages that detract from search results.
The main strategies covered include identifying and resolving problematic internal redirects, assessing external redirects based on value and need, leveraging analytics for data-driven decisions, and establishing ongoing best practices.
Sites with clean redirect structures allow Google to devote more crawling budget to high-value pages. This helps improve rankings for target keywords and content.
For users, it minimizes confusing redirect chains that create slower load times.
Overall, keeping redirects lean, purposeful and optimized for both bots and visitors should be an integral part of any SEO strategy.
