Free tool, runs in your browser
AI robots.txt generator
Pick which AI crawlers may use your site, then copy the rules into your /robots.txt. The default keeps you in ChatGPT, Claude, Perplexity and Google search while opting out of model training.
Blocking Googlebot or Applebot also removes your site from Google Search or Apple search. To opt out of Gemini or Apple model training only, block Google-Extended or Applebot-Extended instead.
Then check it
robots.txt is only half of it: Cloudflare and other CDNs can block AI crawlers before robots.txt is read. After you publish the file, run the free AI crawler check to see, for 20 crawlers, whether your site lets them in and which rule decides it.
What the categories mean
- Training: collects content that may train AI models (GPTBot, ClaudeBot, Google-Extended, CCBot...). Blocking it does not remove you from AI search.
- Search: indexes pages so AI search can cite them (OAI-SearchBot, Claude-SearchBot, PerplexityBot...). Blocking it keeps you out of those answers.
- User request: fetches a page when a person asks the assistant about it (ChatGPT-User, Claude-User, Perplexity-User).
More detail: OpenAI's three crawlers, block AI training without leaving AI search, and our study of 751 sites.