Which websites block AI crawlers? robots.txt rules on 10,000 top sites (October 2026)

Key findings

All shares are of the 6,528 sites whose robots.txt rules we could read. “Fully block” means a Disallow: / rule for that crawler with no exceptions. Download the data (CSV): one row per domain, 10,000 rows.

• 1,126 sites (17.2%) fully block at least one AI crawler. 964 of them (14.8% of all) do it with a rule that names the bot; the rest block it through a catch-all rule for every crawler.
• Among the 20 highest-ranked sites that fully block an AI crawler, instagram.com, twitter.com, x.com, amazon.com, pinterest.com, tiktok.com, msn.com, baidu.com and reddit.com fully block GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended and CCBot, and so do news sites further down such as nytimes.com and cnn.com. github.com fully blocks only Bytespider. The examples table lists all 20.
• Training crawlers are blocked more than twice as often as the search crawlers of the same company: GPTBot 11.7% vs OAI-SearchBot 5.1%, ClaudeBot 10.9% vs Claude-SearchBot 4.7%, meta-externalagent 9.3% vs Meta-WebIndexer 4.3%, Amazonbot 9.0% vs Amzn-SearchBot 3.7%.
• Of the 766 sites that fully block GPTBot, 443 (58%) do not fully block OAI-SearchBot — the crawler OpenAI uses for ChatGPT search. Only 89 of those 443 name OAI-SearchBot in their file; the rest leave it to their general rules.
• 484 of the 585 sites that block Google-Extended still let Googlebot crawl. Google says Google-Extended has no effect on Google Search.
• 155 sites (2.4%) block all 16 AI-specific user agents we checked. 101 of them also block Googlebot — 100 of those with a catch-all rule that closes the site to every crawler.
• The top 1,000 block more: Perplexity-User 11.2% vs 5.0% for ranks 1,001–10,000, Claude-User 9.8% vs 4.7%. For GPTBot (14.6% vs 11.5%) the gap could be chance.
• 1,416 of 5,835 robots.txt files (24%) name at least one AI crawler. Googlebot is fully blocked by only 1.7% of sites.

Search, training and user crawlers: what blocking each one changes

AI companies run separate crawlers for separate jobs, and the name you block in robots.txt decides which job you opt out of. We read each operator's crawler documentation on 3 October 2026; the table further down sums it up.

OpenAI: its crawler overview says OAI-SearchBot surfaces websites in ChatGPT's search features, and sites that opt out “will not be shown in ChatGPT search answers, though can still appear as navigational links”. GPTBot crawls content that may be used to train OpenAI's foundation models. OpenAI says each setting is independent and that search changes take about 24 hours. ChatGPT-User visits a page when a user's question calls for it, and for those visits “robots.txt rules may not apply”.

Anthropic: its help page lists ClaudeBot for training, Claude-SearchBot for search quality and Claude-User for pages fetched when someone asks Claude a question. Blocking ClaudeBot signals that future content should be left out of training; blocking Claude-SearchBot or Claude-User, Anthropic says, may reduce your visibility in user search results.

Google: Google-Extended is a token, not a crawler. It controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI, and it does not affect inclusion or ranking in Google Search. AI Overviews and AI Mode are part of Search: Google's AI features page names Googlebot rules as the control there.

Apple, Perplexity, Meta and Amazon draw the same line between search and training (Apple, Perplexity, Meta, Amazon). Common Crawl's CCBot builds an open archive of the web that anyone can download. ByteDance, the company that owns TikTok, publishes no crawler documentation we could find, so what Bytespider collects data for is not documented by its operator. Cloudflare wrote on 3 July 2024 that Bytespider is reportedly used to gather training data for ByteDance's large language models, including those behind its chatbot Doubao, and that it sent more requests to Cloudflare-protected sites than any other AI crawler at the time — a third-party report, not the operator's own statement.

In practice: blocking a training crawler does not, on the operators' own account, take a site out of AI answers that search the web. Blocking a search crawler does, for that engine. Whether ChatGPT or Gemini then mention your brand is a different measurement — see AI visibility. The full checklist for crawler access is in our guide to LLM SEO.

Which AI crawlers top sites block

Three groups stand out in the table below. Crawlers that collect data for training or for open archives — CCBot, Bytespider, GPTBot, ClaudeBot — are fully blocked by 11–13% of sites. Training tokens and crawlers from Google, Apple, Meta and Amazon follow at about 9%. Search and user-triggered crawlers sit between 3.7% and 7.6%, and the general search crawlers Googlebot and Applebot below 3%.

Most blocks are deliberate: 622 of the 766 full blocks of GPTBot (81%) come from a rule that names GPTBot. The other 144 sites block it only through a catch-all User-agent: * group with Disallow: /, which shuts out every crawler the file does not name.

Partial rules are common but rarely about AI. Between 60% and 69% of sites give each AI crawler some disallowed paths, almost always because the crawler falls under the general rules that keep bots out of admin, search or cart pages. Rules written for a specific AI crawler are rarer: 286 sites give GPTBot its own group that blocks some paths but not the whole site.

Training blocked, search left open

The pattern “block the training bot, allow the search bot” is common, but it is not yet the majority view among the largest sites. Across all readable sites, 55–60% of those that block a company's training crawler leave its search crawler open. In the top 1,000, it is 46.5% for OpenAI and 42.9% for Anthropic — the biggest sites more often block both.

Leaving a search crawler open is mostly passive. Of the 443 sites that block GPTBot but not OAI-SearchBot, 89 name OAI-SearchBot in their file; for the other 354 it simply falls under the general rules, which in 316 cases block a few paths. The table below gives the counts for the four operators that publish a separate search crawler.

Top 1,000 vs ranks 1,001–10,000

All 16 AI-specific user agents are blocked more often by the 1,000 highest-ranked domains than by the next 9,000: ClaudeBot by 15.4% against 10.4%, Bytespider by 18.0% against 11.9%, Google-Extended by 12.9% against 8.6%. The general search crawlers barely differ: Googlebot 1.9% against 1.6%, Applebot 2.5% against 2.7%.

Of the 1,000 top domains, 590 had readable rules (520 robots.txt files, 60 with no file and 10 serving an HTML page instead); the other 410 could not be read, so the comparison rests on 590 sites. Gaps of around three points, as for GPTBot (3.1 points), OAI-SearchBot (3.3), ChatGPT-User (3.1) and Amazonbot (2.4), are under three standard errors and could be chance. The largest gaps — Perplexity-User (6.2 points), Bytespider (6.1), Meta-WebIndexer (5.3), Claude-User (5.1) and ClaudeBot (5.0) — are beyond that.

Content signals: a statement, not a block

167 robots.txt files (2.9%) carry a Content-Signal line, part of the Content Signals Policy Cloudflare launched on 24 September 2025. It states how content may be used after it is fetched: search, ai-input and ai-train. 166 of the 167 files say search=yes, 109 say ai-train=no and 23 say ai-input=no. These lines express a preference and do not stop a crawler from fetching pages, so they are not counted as blocks on this page.

How we measured

• List: the Tranco list 94GG2, generated on 2 October 2026 from 30 days of data (3 September to 2 October 2026) combining the Chrome UX Report, Farsight, Majestic, Cloudflare Radar and Cisco Umbrella. It ranks registrable domains; we took ranks 1 to 10,000.
• Fetch: on 3 October 2026, 14:56–15:27 UTC, we requested https://<domain>/robots.txt for each domain, or the same file on the www. host when the bare domain had no DNS record. Nothing else was requested. Redirects were followed up to three times, but only when they pointed to another /robots.txt; a redirect to a login page or homepage counts as not read.
• Identity: our fetcher identified itself as “Mozilla/5.0 (compatible; LlamatResearch/1.0; +https://llamat.com/methodology)” and never posed as an AI crawler. All requests came from one network in the Czech Republic, at most 32 at a time, with a 10-second timeout. Timeouts and dropped connections were retried once at 8 at a time.
• Parsing follows the matching rules of RFC 9309: a crawler obeys the groups that name it (matched without regard to case), or the * group if none does; within a group the longest matching rule wins, and Allow wins a tie. Apple documents that Applebot follows Googlebot's rules when a file does not name Applebot, so we applied that too.
• Full block: the crawler's group has a Disallow rule covering every path (/ or /*) and no Allow exception. Partial: some paths are disallowed, but not all. Allowed: nothing is disallowed, no group applies, or the site has no robots.txt.
• Readable: a robots.txt file (5,835 sites), a 404 or 410 meaning no file (579), or an HTML page served instead of a robots.txt (114). The two no-file cases count as allowing everything. Unlike RFC 9309, which treats any 4xx answer as “allow all” and a 5xx answer as “disallow all”, we left out sites that refused our fetcher (403 and other 4xx), returned a 5xx or redirected elsewhere, because those answers say nothing about what real crawlers get; and we followed at most 3 redirects, where RFC 9309 asks crawlers to follow at least 5. All shares on this page use these 6,528 sites as the base.
• How we collect and count AI answers for the free report is described separately in our methodology.