The Crawl Stats Report Explained

The Crawl Stats report in Google Search Console shows how Google crawls your site: how many requests it made, how much it downloaded, how fast your server responded and whether it hit availability problems. Open it from Settings > Crawl stats. Google says it is aimed at advanced users; sites under a thousand pages should not need it.

TL;DR

• It is only available for root-level properties: a Domain property or a URL-prefix property at the root, such as https://example.com.
• Start with Host status. It checks robots.txt fetching, DNS resolution and server connectivity.
• Then read the four breakdowns: response code, file type, crawl purpose (Discovery or Refresh) and Googlebot type.
• URLs are the actual URLs Google requested, not canonical URLs. Each hop in a redirect chain counts as its own request.
• The report covers Google's crawlers only. For other bots, including non-Google AI crawlers, use your server logs.

What the Crawl Stats report shows

Google's Crawl Stats report documentation (as of 30 September 2026) describes each part:

Section	What it measures	What to watch for
Total crawl requests	All crawl requests for URLs on your site, successful or not, including page resources hosted on your site. Duplicate requests count separately.	Sudden drops or spikes.
Total download size	Bytes downloaded while crawling. A cached resource used by many pages is requested only the first time.	Jumps after a template or asset change.
Average response time	Average response time for all resources fetched. Each resource counts as a separate response.	A rising trend. Google slows crawling when a site responds slowly.
Host status	Whether Google hit availability problems in robots.txt fetching, DNS or server connectivity.	Anything other than green.
By response	Share of requests by response, such as 200, 301, 404 or 5xx.	Growth in 5xx, timeouts or "robots.txt not available".
By file type	HTML, image, JavaScript, CSS, PDF, JSON and others. Redirects fall under "Other file type".	Crawl spent on files that don't need it.
By purpose	Discovery (a URL never crawled before) or Refresh (a recrawl of a known page).	A bump in Discovery after you publish new content.
By Googlebot type	Smartphone, Desktop, Image, Video, Page resource load, AdsBot, StoreBot and Other agent type.	Spikes from one agent, often AdsBot.

Host status explained

Host status has three states. Google defines them by how recently it saw a significant availability problem:

• Green: no significant crawl availability issues in the past 90 days.
• Yellow: at least one significant issue in the past 90 days, but more than a week ago.
• Red: at least one significant issue in the last week.

Each of the three categories (robots.txt fetching, DNS resolution and server connectivity) has a chart with a dotted red line. Going above it counts as an issue. Google's example: DNS resolution failing for more than 5% of requests on a given day.

robots.txt needs the most care. Google treats a 200 (even an empty or invalid file) and a 403, 404 or 410 as successful. A 429 or 5xx is unsuccessful. If the robots.txt request fails, Google stops crawling for the first 12 hours. From 12 hours to 30 days it uses the last good copy. After 30 days it crawls without restrictions if your home page is available, or stops if it is not.

How to read the Crawl Stats report, step by step

1. In Search Console, open a root-level property and click Settings in the left menu.
2. Under Crawling, find Crawl stats and click Open report.
3. Check the three totals at the top: requests, download size and average response time. Look for sudden changes, not absolute numbers.
4. Click Host status. If it is not green, open each category and look for days above the dotted red line.
5. In By response, click any row, such as Server error (5xx), to see a chart and example URLs.
6. In By purpose, check that Discovery rises after you publish or submit a sitemap.
7. In By Googlebot type, check which agent drives any spike.
8. On a Domain property with subdomains, click a host in the Hosts list to see one subdomain on its own.

Google notes that example URLs are "not comprehensive, but just a representative example". A URL missing from the list may still have been crawled.

Response codes: good, possibly good and bad

Group	Responses	Google's guidance
Good	OK (200), Moved permanently (301/308), Moved temporarily (302/307), Moved (other, a meta refresh), Not modified (304)	Most responses should be 200 or other good responses, unless you are reorganising or moving the site.
Possibly good	Not found (404)	Often correct for removed pages. Fix broken links you control.
Bad	robots.txt not available, Unauthorized (401/407), Server error (5xx), Other client error (4xx), DNS unresponsive, DNS error, Fetch error, Page could not be reached, Page timeout, Redirect error, Other error	Fix these to improve crawling.

If 5xx errors or timeouts line up with pages showing as "Discovered – currently not indexed", the two are likely related. Our guide to Page indexing statuses explains that status.

Things the report does not tell you

• Canonical grouping. URLs are the actual URLs Google requested. Unlike some other Search Console reports, data is not assigned to canonical URLs.
• Other domains. Requests to other domains, including images your pages load from another domain, are not shown. Subdomains are visible from the parent Domain property.
• Every request. Google lists a known issue: some requests might not be counted, so small differences from your server logs are expected.
• Non-Google crawlers. The report covers Google's crawling history. Google's crawler documentation also says Google-Extended has no separate user agent string; it is a robots.txt token, so it will not appear as its own Googlebot type. To see GPTBot, ClaudeBot or PerplexityBot, check your server logs. Our AI crawler study shows how sites handle these bots.

When crawl rate drops or spikes

Google lists several reasons for a sudden drop: a new or broad robots.txt rule, slow responses, a higher server error rate, or content that changes rarely. For spikes, Google points to unblocking or adding large sections, or new Dynamic Search Ads targets, which bring AdsBot. If Google is overloading your server, Google suggests returning 503 or 429 for a short time, and warns against doing so for more than two or three days.

Slow average response time is often a server or code problem. Our page speed optimization service works on server response as well as front-end speed, and our technical SEO service covers crawl efficiency on large sites.

Frequently asked questions

Where is the Crawl Stats report in Search Console?

Click Settings in the left menu, then Open report next to Crawl stats. It only appears for Domain properties and URL-prefix properties at the root of a site.

Do small sites need the Crawl Stats report?

Usually not. Google says that a site with fewer than a thousand pages should not need this report.

What is the difference between Discovery and Refresh?

Discovery is a request for a URL Google has never crawled. Refresh is a recrawl of a known page. After publishing new content or submitting a sitemap, you should see Discovery rise.

Why are crawl totals much higher than my server logs?

Google counts crawls it would have made while your robots.txt was unavailable, even though it did not make them. Check the robots.txt fetching chart under Host status.

Can I ask Google to crawl my site more often?

No. Google says you can't ask it to increase your crawl rate. You can keep sitemaps up to date, avoid blocking pages you want crawled, and keep the server fast and stable.

Sources

• Crawl Stats report (Search Console Help, as of 30 September 2026)
• Google's common crawlers (Google for Developers, last updated 14 July 2026)
• Page indexing report (Search Console Help, as of 30 September 2026)

This post is part of our Google Search Console guide.

Is Google crawling the pages that matter?

Our free SEO audit reviews your Crawl Stats and indexing data and shows where crawl effort is lost. It is delivered within 48 hours. Request your free SEO audit.