Guide, updated 2026-10-29

OpenAI's crawlers: GPTBot, OAI-SearchBot and ChatGPT-User

Q: Does blocking ChatGPT-User in robots.txt work?

OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person initiated the request. It also does not decide whether you appear in ChatGPT search.

OpenAI visits your site with three user agents that do different jobs: GPTBot collects content that may be used to train models, OAI-SearchBot feeds ChatGPT search, and ChatGPT-User fetches a page when a person asks for it. Each is controlled separately, and mixing them up is the most common way to end up out of ChatGPT by accident.

What each one does

Token	Job	If you block it
GPTBot	Crawls content that may be used to train OpenAI's generative AI foundation models.	Signals that your content should not be used for training. It does not take you out of ChatGPT search.
OAI-SearchBot	Surfaces websites in ChatGPT's search features.	Your site is not shown in ChatGPT search answers, though it can still appear as a navigational link.
ChatGPT-User	Visits a page when a ChatGPT user, or a custom GPT, asks about it.	OpenAI says robots.txt rules may not apply because the user initiated the request.

OpenAI also documents OAI-AdsBot, which only visits pages submitted as ads on ChatGPT and whose data is not used for training. If you do not run ads on ChatGPT you will not see it.

According to OpenAI, each setting is independent of the others. Its own example is the one you probably want: allow OAI-SearchBot to appear in search results and disallow GPTBot to signal that content should not be used for training. If you allow both, OpenAI says it may use the results of a single crawl for both uses.

The robots.txt to allow search and block training

Paste this into your /robots.txt, next to your existing rules:

# OpenAI: out of training
User-agent: GPTBot
Disallow: /

# OpenAI: in ChatGPT search
User-agent: OAI-SearchBot
Allow: /

Prefer to click? Our AI robots.txt generator builds these rules for all 20 AI crawlers.

The OAI-SearchBot group matters if you have a User-agent: * group with Disallow: /. A crawler follows the most specific group that names it (RFC 9309), so naming it with Allow: / exempts it from the blanket rule. OpenAI also recommends allowing its published IP ranges, listed in its documentation. For other companies, see our guide to blocking training in robots.txt and the reference of 20 crawlers.

For search results, OpenAI says its systems can take about 24 hours to adjust after a robots.txt change.

How to check

The fastest way is the free Atalaya check: enter your domain and it shows, for GPTBot, OAI-SearchBot and ChatGPT-User, whether your robots.txt allows them and the line that decides each case. You can also open yoursite.com/robots.txt and read the groups that name OpenAI.

What robots.txt cannot see

A CDN, firewall or bot-protection service acts before robots.txt is read. You can allow OAI-SearchBot in the file and still answer it with a 403. If your robots.txt is fine and ChatGPT cannot find you, check that layer: our Cloudflare guide lists the five places where it usually happens. Our check reads robots.txt only, so it cannot see those blocks either.

FAQ

Does blocking GPTBot remove me from ChatGPT?

Not by itself. OpenAI says GPTBot and OAI-SearchBot are independent settings. Blocking GPTBot signals that your content should not be used for training; ChatGPT search depends on OAI-SearchBot.

How long does a robots.txt change take to show up?

For search results, OpenAI says about 24 hours.

Does blocking ChatGPT-User in robots.txt work?

OpenAI says robots.txt rules may not apply, because a person initiates the visit. This agent also does not decide whether you appear in ChatGPT search.

Source

Everything we attribute to OpenAI comes from its documentation, OpenAI Crawlers, read on 2026-10-29. If it changes, the documentation wins.

Check your site

Written with AI. If you spot an error, write to hola@atalaya-ai.com.