Nov. 17, 2026 · The full story, sourced. AboutStandardsSend a tip
IT & Software · How-to

How to Block AI Crawlers With robots.txt: GPTBot, ClaudeBot, Google-Extended and More

Each AI company documents separate robots.txt agents for training, search and user-triggered fetches. Blocking one does not block the others, and robots.txt is voluntary.

Drafted with AI assistance. Reported and edited by ChorusWire Staff. How we use AI

What we know

  • OpenAI says GPTBot is for training content, OAI-SearchBot is for ChatGPT search, and ChatGPT-User handles user-initiated fetches where robots.txt rules may not apply, per OpenAI's bots page.
  • Anthropic documents three robots, ClaudeBot (training), Claude-SearchBot (search) and Claude-User (user requests), and says blocking IP addresses may not reliably opt a site out, per Anthropic's support page.
  • Google says Google-Extended governs use of crawled content for Gemini training and grounding and does not affect Google Search inclusion, per Google's common crawlers page.
  • Perplexity says PerplexityBot is not used to crawl content for AI foundation models and that Perplexity-User generally ignores robots.txt, per Perplexity's crawler page.
  • RFC 9309 says robots.txt rules are not a form of access authorization and not a substitute for content security measures.

To communicate that you do not want certain crawlers collecting your site’s content for AI training, use the robots.txt controls documented by each provider. These controls differ by company: some block crawling, while others restrict how already-crawled content may be used. They do not guarantee exclusion from every AI training dataset. The names involved include GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google), Applebot-Extended (Apple) and CCBot (Common Crawl, an open repository of web crawl data). Blocking those does not hide you from AI search or stop fetches a user triggers by hand, because those use different agents. And robots.txt is a request, not a lock: it works only on crawlers that choose to honor it, per the robots.txt standard, RFC 9309. All details below are as of Nov. 10, 2026, from each company’s own documentation.

Training, search and user fetches are different agents

The companies split their robots on purpose. OpenAI says each setting is independent, so a site can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot to signal that its content should not train OpenAI’s foundation models. Anthropic makes the same three-way split: ClaudeBot for training, Claude-SearchBot for search quality, and Claude-User for fetches made when a person asks Claude a question.

Company Agent Purpose, per the company Robots.txt note, per the company
OpenAI GPTBot Content that may be used for training Disallow signals no training use
OpenAI OAI-SearchBot Surfaces sites in ChatGPT search Opted-out sites will not be shown in search answers
OpenAI ChatGPT-User Fetches triggered by a user Rules may not apply
Anthropic ClaudeBot Collects content that could contribute to training Restricting it excludes future material from training
Anthropic Claude-SearchBot, Claude-User Search quality; user-asked fetches Disabling may reduce visibility
Google Google-Extended Gemini training and grounding No effect on Google Search inclusion
Perplexity PerplexityBot, Perplexity-User Search results; user-asked fetches User fetcher generally ignores rules
Apple Applebot-Extended Opt-out of generative model training Applebot may still crawl for search
Common Crawl CCBot Open repository of web crawl data Disallow stops its crawl

What each one does and does not cover

  • OpenAI. Per OpenAI’s bots page, ChatGPT-User is not used to crawl automatically and is not used to decide what appears in Search. Because a user starts those requests, robots.txt rules may not apply. OpenAI lists a fourth agent, OAI-AdsBot, which checks pages submitted as ChatGPT ads. Search opt-outs can take about 24 hours to take effect, OpenAI says.
  • Anthropic. Anthropic says its bots honor robots.txt and that blocking IP addresses may not work reliably, since it stops the bot from reading your robots.txt. It also says it began recognizing a machine-readable “no AI training” preference signal, based on an IETF draft, on Oct. 30, 2026, and that a rule naming ClaudeBot takes priority over the general signal.
  • Google. Google says Google-Extended controls whether crawled content may train future Gemini models and ground answers in Gemini Apps and Vertex AI, and that it has no separate user agent string. The crawling itself uses existing Google agents. Google also says its common crawlers always obey robots.txt for automatic crawls, while user-triggered fetchers generally ignore it.
  • Perplexity. Per Perplexity’s crawler page, PerplexityBot is for search and is not used to crawl content for AI foundation models. Perplexity-User supports user questions and generally ignores robots.txt because a user requested the fetch.
  • Apple. Apple says Applebot data may help train its foundation models, and disallowing Applebot-Extended opts out of that. Doing so does not stop Applebot from crawling pages for Spotlight, Siri and Safari. To opt out of use in AI-generated answers, Apple points to the nosnippet meta tag.
  • Common Crawl. Common Crawl is a non-profit that maintains an open repository of web crawl data. It says to add a Disallow rule for CCBot to stop its crawling, and warns that some crawlers falsely identify themselves as CCBot.

A short example

This robots.txt example blocks GPTBot and ClaudeBot, requests that CCBot not crawl the site, and uses Google-Extended to opt out of specified Gemini-related uses. It leaves the separately documented search crawlers unrestricted. Place it in the top-level directory of each domain.

User-agent: GPTBot

Disallow: /

User-agent: ClaudeBot

Disallow: /

User-agent: Google-Extended

Disallow: /

User-agent: CCBot

Disallow: /

Add Applebot-Extended the same way if you want Apple’s training opt-out; it is a control token for that opt-out, not a separate crawling process. Anthropic says to repeat the rule for every subdomain you want to opt out. To limit speed instead of blocking, Anthropic supports a non-standard Crawl-delay line; Apple says Applebot does not follow crawl-delay.

What blocking does not do

Robots.txt is voluntary. The standard says crawlers are requested to honor the rules, that they are not a form of access authorization, and that the protocol is not a substitute for real content security. It also warns that listing paths in the file makes them publicly discoverable. For private material, use authentication.

The companies describe opt-outs as forward-looking signals about use in training, not as proof of removal: Anthropic, for example, says restricting ClaudeBot signals that future materials should be excluded. We found no company page promising deletion of anything already collected.

Names and rules change. Google says it intends to support the AIPREF training-preference signal in early 2027, so check each company page before relying on this list.

What we could not verify

We read the pages above but did not test whether any bot obeys your file in practice, and we did not verify how many sites block each agent. Anthropic’s page describes its new preference signal but we did not confirm the exact syntax it reads.

Quick answers

Does blocking GPTBot remove my site from ChatGPT search?

No. OpenAI says GPTBot relates to training, while OAI-SearchBot controls whether a site is shown in ChatGPT search answers. Each setting is independent.

Does blocking Google-Extended affect my Google Search ranking?

Google says Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal in Search.

Do AI bots have to obey robots.txt?

No. The robots.txt standard says crawlers are requested to honor the rules, and the rules are not access authorization. The companies above say their crawlers honor it, but some user-triggered fetchers generally do not.

Is robots.txt enough to protect private pages?

No. The standard says it is not a substitute for content security measures, and listing paths in the file makes them publicly discoverable. Use authentication for private content.

See ChorusWire first in Top Stories. Add us as a preferred source on Google.

Add as preferred source
See our corrections policy. Spot an error? news@choruswire.com
ChorusWire Staff The ChorusWire newsroom desk. Stories are drafted with AI assistance, checked against primary sources and reviewed by a human editor before publishing. See our editorial standards and AI policy. More by ChorusWire