Reference, updated 2026-10-22
AI crawler reference 2026: 20 bots, their robots.txt tokens and what blocking does
Twenty AI user agents visit websites for three different jobs: collecting data to train models, powering AI search, and fetching a page because a user asked. The job decides what blocking costs you. The vendors document separate robots.txt tokens for training and for search, so blocking a training crawler does not take you out of that vendor's AI search. Blocking a search crawler can.
How to read this list
Each token is what you write after User-agent: in robots.txt, matched case-insensitively. We last checked the list against vendor documentation on 2026-10-19, linked at the end. OpenAI also documents OAI-AdsBot, which only visits pages submitted as ads on ChatGPT. It is not one of our 20.
Model training: 8 crawlers
Blocking these tells the vendor not to train on your pages.
| Token | Vendor | What it does | If you block it |
|---|---|---|---|
| GPTBot | OpenAI | Crawls content that may be used to train OpenAI's generative AI foundation models. | Signals that your content should not be used for training. ChatGPT search uses a separate token, OAI-SearchBot. |
| ClaudeBot | Anthropic | Collects web content that may contribute to Anthropic's model training. | Signals that your future content should be excluded from training datasets. Claude's search and user fetches use separate tokens. |
| Google-Extended | A standalone token, not a separate visitor. Controls whether content Google already crawls is used for Gemini training and for grounding in Gemini Apps and Vertex AI. | Google says it does not affect inclusion in Google Search or ranking. It does cover Gemini grounding. | |
| Applebot-Extended | Apple | A token that controls whether content Applebot crawls is used to train Apple's generative AI foundation models. | Does not stop Applebot, which powers Apple search features. |
| CCBot | Common Crawl | Builds the open Common Crawl dataset, which many AI labs use as a training source. | Stops CCBot from crawling your site. It is a dataset crawler, not a search product. |
| Meta-ExternalAgent | Meta | Crawls the web to train foundation AI models or to improve products by indexing content directly. | Covers both uses as Meta describes them. Meta-ExternalFetcher is a separate token. |
| Amazonbot | Amazon | Improves Amazon products and services and may be used to train Amazon AI models. | Amazon says each of its user agent settings is independent. Alexa search uses Amzn-SearchBot. |
| Bytespider | ByteDance | Crawler commonly attributed to AI and LLM data collection. We found no first-party ByteDance documentation. | Third-party reports say it may not honor robots.txt, so a rule may not stop it. |
AI search: 7 crawlers
These index or retrieve pages so a product can show them, with links, in AI answers. Blocking one can keep you out of that product's answers.
| Token | Vendor | What it does | If you block it |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT's search features. | OpenAI says opted-out sites are not shown in ChatGPT search answers, though they can still appear as navigational links. |
| Claude-SearchBot | Anthropic | Indexes content to improve the relevance and accuracy of Claude search results. | Anthropic says disabling it may reduce your visibility and accuracy in Claude search results. |
| PerplexityBot | Perplexity | Surfaces and links websites in Perplexity search results. Not used to train foundation models. | Perplexity recommends allowing it, and its published IP ranges, if you want to appear in results. |
| Googlebot | The Google Search crawler. | Affects Google Search itself. Google documents Googlebot access plus nosnippet, max-snippet and noindex as the controls for AI Overviews and AI Mode. | |
| Applebot | Apple | Powers Apple search features such as Spotlight, Siri and Safari. Crawled data may also supply AI answers unless nosnippet is used. | Can keep you out of those features. If robots.txt names Googlebot but not Applebot, Applebot follows the Googlebot rules. |
| Amzn-SearchBot | Amazon | Improves search experiences in Amazon products such as Alexa. Not used for generative AI training. | Can keep you out of those search experiences. Amazon says a change can take about 24 hours. |
| DuckAssistBot | DuckDuckGo | Crawls pages in real time for AI-assisted answers that cite sources. Not used to train AI models. | DuckDuckGo says opting out does not affect organic search rankings or whether you appear in its search results. It takes effect after 72 hours. |
Fetch on a user's request: 5 crawlers
These fetch one URL because a person asked an assistant about it. For four of the five, the vendor says robots.txt may not apply. Anthropic documents Claude-User as something you can control with robots.txt.
| Token | Vendor | What it does | If you block it |
|---|---|---|---|
| ChatGPT-User | OpenAI | Visits a page when a ChatGPT user, or a Custom GPT, asks about it. | OpenAI says robots.txt rules may not apply because the user started the request, and that this agent does not decide whether you appear in ChatGPT search. |
| Claude-User | Anthropic | Retrieves a page when a Claude user asks a question that needs it. | Anthropic says disabling it prevents retrieving your content for a user query and may reduce your visibility for user-directed web search. |
| Perplexity-User | Perplexity | Visits a page to answer a Perplexity user and links to it in the response. | Perplexity says it generally ignores robots.txt because a user requested the fetch. |
| Meta-ExternalFetcher | Meta | Fetches individual links at a user's request to support Meta AI assistant functions. | Meta says it may bypass robots.txt because the fetch is user-requested. |
| Amzn-User | Amazon | Fetches live information for a customer question, for example in Alexa. Not used for training. | Amazon says user-initiated fetches may not follow all robots.txt directives. |
The robots.txt to copy
This opts out of the eight training tokens and leaves everyone else, search crawlers included, with normal access. Merge it with your existing rules; the robots.txt guide explains it.
# Opt out of AI model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: Bytespider
Disallow: /
# Everyone else, including AI search crawlers
User-agent: *
Allow: /Three rules that trip people up
- A crawler follows the most specific group that names it. Once you add a group for OAI-SearchBot, your
User-agent: *rules no longer apply to it (RFC 9309). - Applebot falls back to Googlebot. If your robots.txt names Googlebot but not Applebot, Apple says Applebot follows the Googlebot rules. Name Applebot if you want something different.
- Changes are not instant. OpenAI says search results can take about 24 hours to adjust to a robots.txt update, Amazon says about 24 hours and DuckDuckGo says 72 hours.
Blocks that robots.txt does not show
Blocks set in a CDN, firewall or bot-protection service happen before robots.txt is read. A site can allow every crawler in robots.txt and still answer them with a 403. Check the AI bot settings of any such service and allow the search crawlers you want to keep.
OpenAI, Perplexity and Anthropic publish the IP ranges of their crawlers, and Perplexity documents how to allow its bots in Cloudflare and AWS WAF. Anthropic warns that blocking its IP addresses can stop it from reading your robots.txt, so use robots.txt for opt-outs. Common Crawl says other crawlers falsely identify themselves as CCBot and offers reverse DNS to verify the real one.
Vendor documentation
- OpenAI
- Anthropic
- Perplexity
- Google: AI features and common crawlers
- Apple
- Common Crawl
- Meta
- Amazon
- DuckDuckGo
- ByteDance: no first-party page found; Cloudflare's bot list includes Bytespider
Check your site
Our free check applies these 20 tokens to your robots.txt the way the crawlers read it and shows the line that decides each one. In our study of 751 popular sites, 11.9% of Mexican sites, 18.8% of the US sample and 6.6% of Canadian sites blocked at least one AI search crawler. For the other file AI tools may look at, see the llms.txt guide.