As it becomes increasingly challenging to rank organically with the addition of AI overviews and the rise of zero-click searches, dealing with page indexation issues is like adding insult to injury. Why are so many webpages being crawled but not getting indexed, and what can we do to fix it?
For anyone who has ever been left perplexed by their Google Search Console (GSC) page index report, we’re here to help. We’ll get to the bottom of the “Crawled – currently not indexed” issue and assist in troubleshooting by improving site and page quality.
What Does “Crawled – Currently not Indexed” Mean?
When looking at the GSC page index report, you’ll notice that there are two similar, but different, reasons why your pages weren’t indexed: “Crawled – currently not indexed” and “Discovered – currently not indexed,” as seen below.
So what’s the deal? The first listing means that Google has crawled your page, but did not add it to its search index, which could be for any number of reasons. The second indicates that Google is aware of your page but hasn’t crawled it yet, either because the site is new or because it is prioritizing other pages.
GSC’s help documentation is evasive about both of these page indexation issues, providing one single, vague reason why you might see a “Discovered – currently not indexed” flag. It states: “The page was found by Google, but not crawled yet. Typically, Google wanted to crawl the URL but this was expected to overload the site; therefore Google rescheduled the crawl. This is why the last crawl date is empty on the report.”
Because of the ambiguity surrounding “Discovered – currently not indexed” and “Crawled – currently not indexed,” there has been a lot of online chatter, including among those at Google, about what these indexation issues mean and what causes them to occur. Google Search Engineer and search engine optimization (SEO) advocate John Mueller recently stated that while there could be many reasons why Google might discover a page but not index it, there are two main factors at play: server capacity and overall website quality.
“When it comes to the quality of the content, we don’t mean like just the text of your articles. It’s really the quality of your overall website. And that includes everything from the layout to the design,” said Mueller.
Quality Factors that Can Contribute to “Crawled – Currently not Indexed:”
What factors contribute to site quality that might cause it to be crawled but not indexed?
- Thin content: It’s no secret that Google deems pages low quality if they are inauthentic, irrelevant, too short, or are stuffed with keywords. And if pages are denoted as low quality, Google is more likely to not index them, according to Martin Splitt, a key member of Google’s Webmaster team who also works to educate SEO professionals. “When Google Search notices a pattern of low quality or thin content on pages, they might be removed from the index and might stay in discovered. Googlebot knows about these pages but is choosing not to proceed with them,” Splitt said.
- Overall site quality: It’s not just individual page quality that matters. The caliber of a site itself can also play a role in indexation. Gary Illyes, an analyst on the Google Search team, stated that the number of crawled but not indexed pages can relate to the overall quality of a site. He said, “If the number of these [crawled but not indexed] URLs is very high, that could hint at general quality issues.”
- Duplicate onsite content: We are all familiar with the pitfalls of duplicate content. While it is not specifically penalized by Google, duplicate content can negatively impact a site’s rankings. Having too many pages with the same content or content that is too similar can signal poor site quality to Google.
- Duplicate offsite content: If you have content that is too similar, or exactly the same, as another site where the pages are already indexed and ranked, it can cause indexation issues. This can sometimes occur with syndicated content, where a partner site has more authority and better organic search signals – that can lead to problems with the indexation of content on the source site. As Illyes notes, sometimes “we crawl the page and then we decide to not index it because there’s already a version of that or an extremely similar version of that content available in our index and it has better signals.”
Eight Fixes for “Crawled – Currently not Indexed:”
So, what’s an SEO to do? Most importantly, give Google a reason to crawl your website by making quality improvements to both individual pages and the overall site. As Illyes says, “Generally if the content of a site is of high quality and it’s helpful and people like it in general, then Googlebot – well, Google – tends to crawl more from that site.”
- Improve thin content pages: Using Google’s E-E-A-T (experience, expertise, authoritativeness, and trustworthiness) framework, optimize your site’s pages to be as useful, informative, and accurate as possible. Make sure pages provide users with what they expect to see (note that this is different from what you think they want to see), and with the level of depth and authoritativeness they anticipate. Take a look at what your competitors are doing, including how much they write about a specific topic and the specific questions they address. Improve your thin content pages by adding helpful content such as FAQs and infographics.
- Optimize your product detail pages: Especially important for e-commerce sites, product detail pages (PDPs) are often overlooked indexation opportunities. By adding content that contributes to E-E-A-T, such as unique descriptions, product reviews, FAQs, and expert advice, companies can better optimize their PDPs for indexation.
- Improve internal linking: Strategically using internal links can help improve relevance signals and boost rankings. Take your “crawled – currently not indexed” pages and look for opportunities to link to them from relevant pages that are currently indexed to flow more authority to those not-indexed pages.
- Consider content consolidation: If you have crawled not-indexed pages that are very similar to other pages on your site that are indexed, consider combining the two to avoid duplicate content and content cannibalization. Make sure to 301 redirect the weaker page to the stronger page after you’ve combined the content.
- Perform content audits: Googlebot is more likely to crawl your site if you have up-to-date and relevant content. This means you can’t have a set-it-and-forget-it mentality when it comes to SEO. You should regularly audit your site to identify which pages can be updated and what new articles can be published to stay current with trends.
- Homepage optimization: Use your homepage as an opportunity to showcase your site’s most popular pages and categories. Be sure to link to your most valuable pages in a prominent spot so customers can easily find what they are looking for. By making it easier for users to find helpful content, you are optimizing your site for E-E-A-T and sending positive quality site signals to Google.
- Navigation menu optimization: Just like with the homepage, you can utilize your navigation menu to help improve your site’s overall quality. Optimize the site nav by featuring the most useful and popular pages in a layout that benefits both user experience and SEO.
- Play the long game: SEO is a slow burn, and page indexation is no different. Google is looking for sites that consistently provide relevant, timely, and helpful content, and that means that SEO professionals need to continually update and improve pages for quality.
After making improvements to your “Crawled – currently not indexed” pages, you can always request re-indexing using Google Search Console’s URL Inspection Tool to give Google a little nudge. But, as with all things related to SEO and Google, changes take time, so prepare to be patient. As Google’s Mueller noted, “It takes a lot of time for us to understand how a website fits in with regards to the rest of the Internet… and that’s something that can easily take, I don’t know, a couple of months, a half a year, sometimes even longer than a half a year.”
Just know that with the quality improvements you’ve made to your pages, you’re fighting the good SEO fight, and the rest – page indexation included – will follow in time.
