Original Research · 2026-07-08
How many websites accidentally block ChatGPT and Claude?
We checked robots.txt and llms.txt on a random sample of 1,000 websites against the 10 major AI crawlers. 12.2% block at least one — and GPTBot is blocked roughly 3.9x more often than PerplexityBot. Full data and methodology below.
12.2%
of sites block at least one major AI crawler
11.2%
block GPTBot specifically — the single most-blocked AI crawler
29%
already serve an llms.txt file (see the nuance below)
13.2%
have no robots.txt at all — wide open to every crawler by default
Block rate by crawler
Out of 797 reachable domains in the sample.
Vendors, left to right by block rate: OpenAI (GPTBot, ChatGPT-User, OAI-SearchBot), Google (Google-Extended), Anthropic (ClaudeBot, anthropic-ai), Common Crawl (CCBot), Apple (Applebot-Extended), ByteDance (Bytespider), Perplexity (PerplexityBot).
The llms.txt number needs an asterisk
29% of sites in the sample already serve a valid llms.txt — surprisingly high for a standard proposed in 2024. Digging into the actual files, a large share were generated automatically by a popular WordPress SEO plugin rather than deliberately authored, which means genuine, purpose-built adoption is very likely lower than the headline number suggests. Plugin-generated files still work — they give AI tools a sitemap and site summary — but they weren't written with intent, so treat this stat as "some form of llms.txt exists," not "sites are deliberately optimizing for AI discovery."
If you want to write one on purpose, our free llms.txt generator takes about two minutes.
Methodology
Sample: 1,000 domains drawn from Majestic Million (majestic.com), domains ranked 150,000–900,000 by referring-subnet count, random sample without replacement. 797 were reachable at the time of the check (the remainder timed out, refused connections, or otherwise did not respond and were excluded from percentages).
What we checked: for each domain, we fetched /robots.txt and /llms.txt directly (a single request to each, 8-second timeout, one pass, no page content fetched) and evaluated root-level (/) access for 10 named AI crawlers using the most specific matching User-agent group in the file — the same open-source parser that powers our free AI Crawler Access Checker.
Reproducibility: random sample, seed 20260705, collected 2026-07-07.
Limitations
- Rank-based long-tail sample used as a proxy for smaller, less globally trafficked sites. Not a verified SMB classification — see Limitations below.
- A robots.txt disallow is a request, not an enforcement mechanism — some crawlers respect it, some don't. This study measures stated policy, not actual crawl behavior.
- We checked root-level (
/) access only. A site can allow the homepage while blocking specific sections. - Single point-in-time snapshot — robots.txt files change. See "Updates" below for re-runs.
Download the aggregated data
JSON, block rates by crawler, methodology notes. Free to cite with attribution.
Is your own site blocking AI crawlers?
Check your robots.txt and llms.txt in 10 seconds with the free tool that powered this study.