Crawlability
Crawlability describes whether a search engine crawler can access and retrieve a page or resource on a website.
A page needs to be crawlable before a search engine can reliably inspect its current content and instructions.
What It Means in Practice
Search engines use automated crawlers to request URLs and follow links across the web.
A page may be easy for a person to visit while still creating problems for crawlers because of technical restrictions, broken links, server errors, or rendering behaviour.
Crawlability is therefore not the same as whether a page exists.
It describes whether a crawler can actually reach and process it.
How Crawlers Reach Pages
Crawlers can discover URLs through sources such as:
- Internal links
- XML sitemaps
- Redirects
- Links from other websites
- URLs already known from previous crawls
Once a URL is discovered, crawl controls such as robots.txt and the server's response determine whether the crawler can retrieve it.
Why It Matters
If an important page cannot be crawled, search engines may be unable to evaluate its content, follow its links, see changes, or process page-level indexing instructions.
That can affect indexability and make troubleshooting search visibility much harder.
Common Crawlability Problems
Common problems include:
- Important URLs blocked in robots.txt
- Broken internal links
- Navigation generated in ways crawlers cannot reliably follow
- Redirect loops
- Excessive redirect chains
- Persistent server errors
- Important content only becoming available after problematic scripts or interactions
- Pages with no practical discovery path
How to Improve
Make important URLs available through normal internal links.
Review robots.txt, redirects, server responses, and site navigation when pages are unexpectedly missing from crawls.
An XML sitemap can support discovery, but it should complement a navigable site rather than replace internal linking.