Crawl Budget
Crawl budget describes the amount of crawling a search engine can and wants to perform on a website within a given period.
It is primarily an advanced concern for large or rapidly changing websites.
What It Means in Practice
Search engines cannot continuously crawl every URL on the web.
They need to decide:
- How frequently to revisit a site
- Which URLs to request
- How many requests the server can handle
- Which content appears most useful to refresh
For Google, crawl budget is influenced by both available crawl capacity and demand for crawling the site's URLs.
Who Needs to Worry About It
Most small and medium websites do not need to actively optimize crawl budget.
If new or updated pages are being discovered and crawled normally, there is usually a more useful SEO problem to work on.
Crawl budget becomes more relevant for sites with:
- Very large numbers of URLs
- Rapidly changing inventories
- Large ecommerce catalogues
- Extensive filtering or faceted navigation
- Automatically generated URL variations
- Significant “Discovered – currently not indexed” patterns
Why It Matters
On a very large site, crawler time spent requesting duplicate, low-value, broken, or unnecessary URLs can reduce how efficiently important content is revisited.
That makes URL inventory and site structure operational concerns rather than abstract SEO metrics.
Common Crawl-Budget Problems
Common causes of inefficient crawling include:
- Large numbers of duplicate URLs
- Faceted navigation producing excessive URL combinations
- Long redirect chains
- Slow or unreliable server responses
- Unnecessary URL parameters
- Large volumes of low-value pages
- Poor sitemap management
- Crawling sections that serve no search purpose
How to Improve
First determine whether crawl budget is actually a constraint.
If it is, focus on reducing unnecessary URL inventory, improving server reliability, consolidating duplicates with canonical URLs, avoiding long redirects, maintaining accurate XML sitemaps, and controlling unnecessary crawling appropriately through robots.txt.
For an ordinary small-business website, keeping the site clean and crawlable is usually enough.