Definition
Crawl budget is the number of URLs a search engine can and wants to crawl on a site in a given time, set by crawl capacity and crawl demand.
Crawl budget explained
Google describes crawl budget as the combination of two things: the crawl capacity limit, meaning how much Googlebot can crawl without overloading your server, and crawl demand, meaning how much Google wants to crawl based on how popular and how often updated your pages are.
Most sites don't need to worry about it. Google's own guide is aimed at very large sites, roughly those with a million or more unique pages changing weekly, or ten thousand or more changing daily, and at sites with many URLs stuck as “Discovered - currently not indexed”. Google stresses those numbers are rough estimates.
Where crawl budget does matter, the fixes are about waste: stop faceted filters, session IDs and endless calendars from generating crawlable duplicate URLs, fix redirect chains and soft 404s, keep sitemaps limited to canonical pages, and make the server respond quickly so Googlebot can fetch more pages in the same time.
Example
Your online store has a few thousand products, but filters for size, color and price create millions of crawlable URL combinations. Googlebot spends its time on those and rarely revisits new products. Limiting crawlable filter combinations frees crawl budget for the pages that sell.
Why it matters
On large sites, wasted crawling means new and updated pages are discovered slowly, which delays rankings and revenue from them.
Related service
Technical SEO
Crawlability, indexing, structured data and rendering fixes so search engines and AI crawlers can read your site.
Technical SEO servicesSources
- Google Search Central: Crawl budget management for large sites (updated 22 Jul 2026)
Published by Vidern, founded and led by Malhar Shah. Updated .
Find out what's holding your site back
Get a free SEO audit with prioritized fixes, delivered within 48 hours.
Related terms
- CrawlingCrawling is the process by which search engine and AI bots discover web pages by following links and sitemaps, then download their content for processing.
- Faceted navigationFaceted navigation lets shoppers filter a category by attributes such as size, color, brand or price, often creating a separate URL for each combination.
- Log file analysisLog file analysis is reviewing server access logs to see which URLs search engine and AI crawlers request, how often, and what responses they get.
- Redirect chainA redirect chain is a series of redirects where URL A redirects to B, which redirects to C, and so on, instead of A going straight to the final URL.
- Soft 404A soft 404 is a page that tells visitors it doesn't exist or has no content but returns a 200 success status, instead of a proper 404 or 410.