Guide, updated 2026-10-21
Block AI training crawlers and stay in AI search
You can tell AI companies not to train models on your pages and still be found and cited when someone asks ChatGPT, Claude, Perplexity or Google. They use different crawlers for each job, so robots.txt can treat them differently.
The robots.txt to copy
Add this to the top of /robots.txt on your domain, or merge it with the rules you already have. The first group opts out of training. The second keeps normal access for everyone else, AI search crawlers included.
# Opt out of AI model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: Bytespider
Disallow: /
# Everyone else, including AI search crawlers
User-agent: *
Allow: /Several User-agent lines in a row form one group that shares the rules below them. That is standard robots.txt (RFC 9309), and every major crawler reads it that way. If your current file has a User-agent: * group with Disallow rules for private paths, keep those lines in that group.
Which crawler does what
Our crawler list, last checked against each vendor's documentation on 2026-10-19, puts the 20 AI crawlers in three groups.
| Group | Crawlers |
|---|---|
| Model training | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| AI search | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Applebot, Amzn-SearchBot, DuckAssistBot |
| Fetch on a user request | ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher, Amzn-User |
Google-Extended and Applebot-Extended never visit your site. They are names you use in robots.txt to say whether pages that Googlebot and Applebot already fetch may be used for model training. Blocking them leaves Google Search and Apple search alone.
The mistake we see most
When we checked 751 popular sites in Mexico, the US and Canada in October 2026, most of the sites that opted out of training had also blocked at least one AI search crawler: 59% in Mexico, 64% in the US sample and 47% in Canada. Usually the cause is one broad rule, such as Disallow: / under User-agent: *, or a copied list that includes OAI-SearchBot and PerplexityBot. Those sites drop out of AI answers along with AI training.
What robots.txt cannot do
robots.txt is a request, and it only covers crawlers that read it. Blocks set in a CDN, firewall or bot-protection service happen before robots.txt is read, so check those settings too. Some vendors also say that fetchers acting on a user's request may not follow robots.txt.
Check your site
Our free check reads your robots.txt the way all 20 crawlers do and shows the line that decides each one.