Crawl Budget
Crawl budget is the amount of crawling Google is willing and able to spend on a site in a given time, combining crawl rate and crawl demand.
- Applies to
- Large sites, frequently changing sites, technically inefficient sites
- Commonly confused with
- Ranking factor, direct ranking signal
Key points
- Crawl budget combines crawl rate (server capacity) and crawl demand (Google's interest in the site).
- Wasted crawling on duplicates, errors, or redirects delays discovery of important pages.
- Improving crawl efficiency helps Google reach more content that matters for indexing and ranking.
- Faster server response and efficient page rendering can increase the crawl rate Google assigns.
- Keeping XML sitemaps up to date helps direct crawling to the URLs you want indexed.
How it works
Crawl budget is the product of two factors: crawl rate and crawl demand. Crawl rate is the speed Googlebot is willing to fetch pages from a site, constrained by server capacity and response times. Crawl demand is Google's subjective interest in the site's content, influenced by popularity, freshness, uniqueness, and user value.
Google dynamically adjusts both factors. If a server responds slowly or returns many errors, crawl rate drops. If the site produces high-quality, regularly updated content, crawl demand rises. The combination determines how many URLs Googlebot will attempt to fetch within a given period.
Tools like Screaming Frog can help identify wasted crawling by showing patterns of redirect chains, duplicate content, and soft 404s that inflate crawl count without adding value.
Why it matters
When crawl budget is wasted on unnecessary URLs, Google may decide it is not worth spending as much time on the rest of a site. This can delay the discovery and indexing of important pages, especially on large or frequently updated sites.
- Duplicate content forces Google to crawl multiple versions of the same information, reducing the budget available for unique pages.
- Long redirect chains and soft 404s consume crawl slots without leading to useful content.
- Faster server responses and efficient rendering let Googlebot read more pages in the same time, effectively increasing the budget.
Consolidating duplicate content with canonical SEO or redirects focuses crawling on the primary version. Similarly, fixing crawlability issues such as broken links or blocked resources prevents wasted requests.
Where it changes your decision
- When a site has more than 10,000 URLs and Google is not indexing all important pages, optimising crawl budget becomes a priority.
- After a site migration or large content addition, crawl patterns may shift; reviewing Search Console crawl stats and log files helps identify new inefficiencies.
- When server performance is poor, upgrading hosting or using a content delivery network can increase the crawl rate Google assigns.
What it is not
- Crawl budget is not a ranking factor; it is an efficiency and discovery issue that affects how quickly pages are indexed.
- It is not solved by simply blocking URLs in robots.txt — that can hide important content and create new problems.
- It is not a fixed number you can calculate precisely; it changes dynamically based on server and content signals.
- It is not something to fix before addressing duplicate content, soft 404s, and redirect chains, which are often the root cause of wasted crawling.
Questions people ask
What is crawl budget?
Crawl budget is the amount of crawling Google is willing and able to spend on a site in a given time, combining crawl rate (server capacity) and crawl demand (Google's interest in the content). It determines how many URLs Googlebot will attempt to fetch and how often.
How to check crawl budget?
You cannot see a specific number for crawl budget, but you can monitor its effects using Google Search Console's crawl stats report, which shows crawl rate, response time, and errors. Analysing server log files also reveals which URLs Googlebot is actually requesting and how often.
What is a crawl budget?
A crawl budget is the limit Google places on how many pages it will try to fetch from a site within a given time. It is the combination of how much Google wants to crawl (crawl demand) and how much the site can be crawled efficiently (crawl rate).
How to increase Google crawl budget?
You can increase crawl budget by improving server response times and rendering efficiency, which allows Google to fetch more pages in the same time. You can also increase crawl demand by producing popular, unique, and frequently updated content that signals Google to spend more resources on the site.
Sources
- Google Search Central: Crawl budget management Primary source for Google’s definition, causes of wasted crawl budget, and optimization guidance.
- Google Search Central: Manage crawling and indexing of site URLs Useful supporting Google documentation on controlling crawl and index behavior through technical SEO signals.
- Google Search Central: Crawl stats report Authoritative reference for measuring crawl activity in Google Search Console.
Written and maintained by Nora Lindholm. Last updated 27 July 2026.
Spotted an error? Email me — corrections are logged and credited on the entry. Outbound links are unpaid; most are nofollow.