Noindex
noindex is an indexing directive that tells supported search engines not to include a page in their search index.
It is commonly implemented through a robots meta tag or an HTTP response header.
What It Does
A noindex directive tells the search engine:
This page can be accessed, but it should not appear in search results.
That makes it different from robots.txt, which controls whether a crawler can request the page in the first place.
How It Works
A common HTML implementation is:
<meta name="robots" content="noindex">
When a crawler accesses the page and reads that directive, it can remove or exclude the URL from its search index.
The crawler needs to be able to access the page to see the instruction.
If the page is also blocked through robots.txt, the crawler may not be able to read the noindex directive.
Why It Matters
Not every public URL should necessarily appear in search.
Noindex can be useful for pages such as:
- Internal search results
- Certain utility pages
- Temporary landing pages
- Duplicate pages that need to remain accessible
- Other public URLs without meaningful search value
It provides much more precise indexing control than trying to prevent crawling.
Common Problems
Common problems include:
- Accidentally leaving noindex on production pages
- Staging settings carrying into launch
- Applying noindex across an entire template
- Combining noindex with robots.txt blocking
- Assuming a page will disappear instantly after adding the directive
- Using noindex where consolidation with a canonical URL would make more sense
How to Improve
Use noindex deliberately and document why a page type needs it.
During launches and migrations, inspect important templates for accidental indexing directives.
When troubleshooting missing pages, noindex should be one of the first things checked.