zaDojoAI crawlers and robots.txt: GPTBot, ClaudeBot and more
Which AI crawlers visit your site, what each one is for, and how to write a robots.txt that keeps you in AI answers while opting out of model training if you want to.
AI companies send several different crawlers, and they do different jobs. Some collect pages to train models. Some build the search index an assistant reads when it answers a question. Some fetch a page only because a user asked about it.
If you block them all, you can drop out of AI answers. If you allow them all, your content may be used for training. Most sites want something in between, and robots.txt is where you set it.
The three kinds of AI crawler
- Training crawlers collect pages that may be used to train future models. Blocking them does not remove you from today's answers.
- Search crawlers build the index an assistant searches before it writes an answer. Block these and the assistant cannot cite you.
- User-triggered fetchers load a page because someone pasted your URL or asked about it in a chat. Vendors treat these differently from automated crawling, so check each vendor's documentation for how they handle robots.txt.
The user agents that matter
| Company | User agent | Job |
|---|---|---|
| OpenAI | GPTBot | Training |
| OpenAI | OAI-SearchBot | Search index for ChatGPT search |
| OpenAI | ChatGPT-User | Fetches a page when a user asks |
| Anthropic | ClaudeBot | Training |
| Anthropic | Claude-SearchBot | Search index for Claude |
| Anthropic | Claude-User | Fetches a page when a user asks |
| Perplexity | PerplexityBot | Search index for Perplexity |
| Perplexity | Perplexity-User | Fetches a page when a user asks |
| Google-Extended | A robots.txt token for Gemini model use, not a separate crawler | |
| Apple | Applebot-Extended | A robots.txt token for Apple's AI model use |
| Common Crawl | CCBot | Open web archive that many models are trained on |
Two of these trip people up:
- Google-Extended does not touch Google Search. Blocking it does not remove you from search results or from AI Overviews, which are built from what Googlebot crawls. It only controls whether Google can use your content for Gemini models.
- Blocking Googlebot to stay out of AI Overviews removes you from Google entirely. There is no robots.txt rule that keeps you in search but out of AI Overviews.
Three robots.txt setups
1. Be in AI answers, allow everything
Do nothing. With no rules for these agents, they are allowed. This is the right default for most businesses that want to be recommended by assistants.
2. Be in AI answers, opt out of training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) stay allowed, so assistants can still find and cite your pages. This is the setup most publishers that care about both visibility and licensing end up with.
3. Keep a section private
User-agent: OAI-SearchBot
Disallow: /members/
User-agent: Claude-SearchBot
Disallow: /members/
User-agent: PerplexityBot
Disallow: /members/Block the paths you do not want read, and leave the rest open. Remember that robots.txt is a public request, not a lock. Anything truly private belongs behind a login.
Mistakes we see often
- A blanket block from a CDN or security plugin. Some firewall and bot-protection settings block AI crawlers by default. If assistants never mention you, check your CDN's bot settings as well as robots.txt.
- Blocking by IP or challenge pages. A crawler that hits a CAPTCHA or a JavaScript challenge reads nothing, even if robots.txt allows it.
- Content that only appears after JavaScript runs. Many AI crawlers do not run JavaScript. If your key text is rendered client side, they see an empty page. Check with your browser's "view source": if the text is not there, a crawler may not see it either.
- Copying a list from 2023. Vendors have split their crawlers into training and search agents since then. An old list that blocks "ChatGPT" may block search when you only meant to block training.
How to check what is happening
- Open
yourdomain.com/robots.txtand read it top to bottom. Look for aUser-agent: *block withDisallow: /, which blocks everyone. - Look at your server or CDN logs for the user agents in the table above. If a search crawler never shows up, something upstream may be stopping it.
- Ask the assistants. Ask ChatGPT, Claude or Perplexity the questions your buyers ask and see whether your site is cited.
That last step is the one that tells you whether any of this works. zaDojo's free AI visibility checker runs one ChatGPT answer for a question you choose and shows whether your brand appears, and AI brand visibility tracking repeats your questions across engines over time.
Related
- llms.txt: what it is and whether to use it, the other file people add for AI tools.
- How to get cited by ChatGPT.
- What GEO is, the wider practice this fits into.
Questions people ask
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot is OpenAI's training crawler. ChatGPT search reads pages through OAI-SearchBot, so leave that one allowed if you want to be cited.
Do AI crawlers respect robots.txt?
The major vendors say their automated crawlers follow it. User-triggered fetchers can be treated differently, so read each vendor's crawler documentation. For content that must stay private, use authentication rather than robots.txt.
Will blocking AI crawlers hurt my Google rankings?
Not on its own. Google Search ranks pages from what Googlebot crawls. Blocking GPTBot, ClaudeBot or Google-Extended does not change that.