TL;DR: In July 2026 we tested the world's top 1,000 websites (Majestic Million) to see whether AI crawlers — the bots that feed ChatGPT, Claude, Perplexity and Gemini — can actually read them. The result: 40.9% of the web's most authoritative sites are unreadable to GPTBot, and nearly one in five sites (18.4%) is completely dark to every AI crawler we tested. Much of this blocking is silent: the site's robots.txt says nothing, but the firewall returns 403 Forbidden.
Why we ran this study
AI assistants are becoming a primary discovery channel. When a buyer asks ChatGPT or Perplexity for a recommendation, those platforms can only cite and describe pages their crawlers can access. A site that blocks AI crawlers — deliberately or by accident — is invisible in AI answers, no matter how well it ranks in classic Google search.
We kept running into this with client audits (most memorably a multi-billion-euro industrial distributor whose entire site returned 403 to every AI bot). So we measured how widespread the problem really is.
Methodology
- Sample: the top 1,000 domains from the Majestic Million, the free index of the web's most-linked domains. 927 responded and form the measurable base; 73 were unreachable or errored and are excluded.
- Test 1 — robots.txt policy: we fetched each site's robots.txt and parsed the rules for seven crawler tokens: GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot and Bingbot. This is each site's official, unambiguous policy.
- Test 2 — live server response: we requested each homepage with each crawler's real user-agent string and recorded the HTTP status. This reveals firewall/CDN blocking that robots.txt never shows.
- Honest caveat: our live requests came from a standard datacenter IP. Firewalls that verify crawlers by IP range (such as Cloudflare's verified-bots program) may treat the genuine crawler differently than our user-agent test — openai.com itself returned 403 to our GPTBot-labelled request, which is IP verification working as designed. The robots.txt figures carry no such caveat: they are the sites' declared policy. The live figures measure how servers treat unverified bot traffic — which is also exactly how they treat the growing ecosystem of AI agents and fetchers that aren't on any verified-bot list.
Finding 1: Four in ten top websites are unreadable to ChatGPT's crawler
Combining explicit robots.txt disallows, firewall blocks and failed responses:
| AI crawler | Feeds | Disallowed in robots.txt | Blocked by firewall | Total unreadable |
|---|---|---|---|---|
| ClaudeBot | Claude (Anthropic) | 18.6% | 19.4% | 42.6% |
| CCBot | Common Crawl → many AI models | 21.1% | 16.9% | 42.2% |
| GPTBot | ChatGPT (OpenAI) | 19.1% | 17.6% | 40.9% |
| PerplexityBot | Perplexity | 15.9% | 17.5% | 37.6% |
| ChatGPT-User | Live ChatGPT page visits | 12.6% | 19.0% | 35.9% |
| Google-Extended | Gemini (policy token) | 16.9% | — | — |
| Bingbot | Bing / Microsoft Copilot | 2.8% | 21.6% | 29.3% |
23.9% of the world's top 1,000 websites explicitly disallow at least one AI crawler in robots.txt — and 17.2% publish no robots.txt at all. (Vidern AI Crawler Access Study, July 2026)
Finding 2: The Bingbot contrast proves it's deliberate
Only 2.8% of top sites disallow Bingbot — but 19.1% disallow GPTBot. That's roughly a 7× difference. If this were general bot paranoia, the numbers would match. They don't: site owners still treat classic search crawlers as sacred, while deliberately shutting out AI crawlers. The web's biggest brands — Amazon, X/Twitter, Facebook, TikTok, Pinterest and Reddit all disallow GPTBot outright — are making a conscious trade: protect content from training, at the price of invisibility in AI answers.
For publishers with licensing leverage, that trade may be rational. For a typical business that wants to be found and recommended, copying big-platform robots.txt files is self-sabotage.
Finding 3: The silent blockers — sites that don't know they're invisible
The most important number in this study isn't in anyone's robots.txt. 17.6% of top websites allow GPTBot on paper but return 403 Forbidden when a GPTBot-labelled request actually arrives. Their firewall or CDN bot protection is blocking AI traffic that their published policy permits.
Sites returning 403 to AI-crawler user-agents despite permissive robots.txt include IBM, Oracle, the Wall Street Journal, Etsy, the UN, the CDC, NIH, Springer and Wiley. (Vidern, July 2026)
Some of these use IP-verified allowlists that admit the genuine crawler while rejecting imposters — a legitimate setup. But in our client work we routinely find the simpler explanation: bot protection was switched on for security reasons years ago, and nobody ever checked what it does to AI crawlers. The marketing team optimises content while the firewall silently discards every AI request. Meanwhile 18.4% of top sites are fully dark — inaccessible to every AI crawler we tested — including Reuters, CNN, Bloomberg, TikTok and Reddit.
What this means for your business
The blocked 40% creates an opening for everyone else. When AI platforms can't read a market leader's site, they answer buyers' questions using whatever they can read — competitor pages, review sites, Reddit threads. In AI search, accessibility is a ranking factor with only two values: visible or invisible.
Three checks every site owner should run this week:
- Check your robots.txt — are GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Extended allowed? An old "block all bots" template may be costing you AI visibility.
- Check your firewall/CDN — Cloudflare Bot Fight Mode, Akamai Bot Manager and similar tools often block AI crawlers by default. Whitelist the ones you want reading your site.
- Test, don't assume — policy and reality diverge (17.6% of top sites prove it). Our free AI Crawler Access Checker runs both tests on your domain in about 15 seconds.
Fixing it usually takes under an hour
If your site turns out to be blocking AI crawlers unintentionally: add explicit User-agent allows for the AI crawlers in robots.txt, whitelist verified AI bots in your WAF/CDN rules, and consider publishing an llms.txt file so AI systems can navigate your site efficiently. Crawler access is one of six factors we score in a full GEO audit — but it's the gating one: nothing else matters while the door is locked.
Cite this study
The data and methodology are described above; figures are from our July 2026 scan of the Majestic Million top 1,000 (927 measurable domains). Journalists and researchers are welcome to cite this study with attribution to Vidern and a link to this page. For the underlying dataset or category breakdowns, contact us.
The Vidern SEO Team consists of certified SEO specialists with 10+ years of combined experience helping SaaS companies, e-commerce brands, and local businesses across India grow their organic search presence.