Atalaya AI
ENES
Check my site

Guide, updated 2026-10-24

Cloudflare says allow, but AI crawlers are still blocked

Your robots.txt lets OAI-SearchBot, Claude-SearchBot and PerplexityBot in, yet AI search still can't read your pages. If your site runs through Cloudflare, the block is often at the edge: Cloudflare answers the crawler before your robots.txt matters. This is one of the most common threads on Cloudflare's community forum. Check these five places, in this order.

1. AI bot policies (Security settings)

Cloudflare sorts AI crawlers by behavior: Search (indexing your content to answer questions later), Agent (fetching in real time for a person, such as chat fetchers) and Training. Each one can be set to Block (on all pages), Block on pages with ads or Allow, under Security Settings > Configure AI bot policies.

Two details catch people out:

  • New defaults since September 15, 2026. For new domains, Training and Agent bots are blocked on pages that show ads, and Search stays allowed. If your pages carry ads, chat fetchers (Cloudflare's Agent category) can be blocked there even though you never touched the setting.
  • Mixed-purpose crawlers count as Training. A crawler used for both training and search is blocked by every option that blocks training. Blocking training can therefore remove you from some AI search.

2. The legacy "Block AI bots" switch

The older one-click Block AI bots setting is being replaced by the policies above (Cloudflare marks it as deprecating on September 15, 2026). If it is still on in your zone, it blocks verified bots classified as AI training plus unverified bots that behave like them. Forum users also report search crawlers caught by it. If you want training off and search on, prefer the per-behavior policies or per-crawler actions below.

3. AI Crawl Control, crawler by crawler

Go to AI Crawl Control > Crawlers. Every AI crawler that visited you has a row with its requests and an Action column (Allow or Block). Set the search crawlers you want to Allow here.

Read the Unsuccessful count carefully. Cloudflare's own docs say unsuccessful requests can come from any rule or response error, so the AI Crawl Control block is only one possible cause. A high count with Allow selected means something else is in the way, usually step 4 or a WAF rule.

4. Bot Fight Mode

Bot Fight Mode (free) challenges traffic that matches patterns of known bots. It does not run in the rules engine, so WAF custom rules and Page Rules cannot skip or bypass it; Skip, Bypass and Allow have no effect on it. If you see AI search crawlers failing and Bot Fight Mode is on, test with it off (Security Settings, filter Bot traffic). On paid plans, Super Bot Fight Mode runs in the rules engine and supports Skip rules for traffic you want to let through.

Also check your WAF custom rules and IP Access rules for anything matching "bot", user agents or the crawlers' networks. Blocked AI bot traffic shows up in Security Analytics.

5. Managed robots.txt (Bot Preference Sync)

The robots.txt crawlers read may not be the file you uploaded. With Bot Preference Sync on and at least one AI policy set to block, Cloudflare prepends its own managed block (it starts with # BEGIN Cloudflare Managed content) and the Content Signals Policy to your file. If your origin has no robots.txt, Cloudflare creates one with Disallow rules for known AI crawlers. Free-plan domains without a robots.txt of their own show the Content Signals Policy.

Open https://yoursite.com/robots.txt in a browser and read the top. If a managed block disallows a crawler you want, change the policy in step 1; editing your own file will not remove the prepended part.

What robots.txt can and cannot tell you

robots.txt is a public request. Edge blocks (403 responses, challenges) happen before it is read and are invisible from outside. Our free check reads your robots.txt the way 20 AI crawlers do, tells you when the site is behind Cloudflare (from its DNS) and shows a managed Cloudflare block if there is one. It does not pretend to be a crawler, so for edge blocks the dashboard places above are the source of truth.

Check your site

Sources

  • Cloudflare docs, Block AI Bots (policies, new defaults, mixed-purpose crawlers), updated 2026-07-01.
  • Cloudflare docs, Manage AI crawlers (per-crawler actions, unsuccessful requests), updated 2026-07-28.
  • Cloudflare docs, Bot Fight Mode (no WAF skip, Super Bot Fight Mode), updated 2026-08-03.
  • Cloudflare docs, robots.txt setting (Bot Preference Sync, prepended content, no-file case), updated 2026-10-22.
  • Cloudflare Community threads on AI crawlers still blocked with settings on allow (2025-2026), summarized in our research notes.