Study, data collected 2026-10-20, updated 2026-10-22
We checked 751 popular sites in Mexico, the US and Canada: who lets AI crawlers in?
In October 2026 we read the robots.txt of 751 popular sites against 20 AI crawlers. 20.2% of the Mexican sites block at least one AI training crawler, against 29.3% in our US sample and 13.2% in Canada. In Mexico and the US sample, most of the sites that opt out of training also block at least one AI search crawler (59% and 64%).
Headline numbers
Every percentage is a share of the sites that gave a verdict: 243 in Mexico, 266 in the US sample and 242 in Canada, out of 300 sampled in each. The 8 training crawlers and the 7 AI search crawlers are the groups in our crawler reference.
| Share of sites that | Mexico | US sample* | Canada |
|---|---|---|---|
| Have a verdict (of 300 sampled) | 243 | 266 | 242 |
| Block at least one training crawler | 20.2% | 29.3% | 13.2% |
| Block all 8 training crawlers | 2.9% | 9.0% | 2.5% |
| Block at least one AI search crawler | 11.9% | 18.8% | 6.6% |
| Block training and also block at least one search crawler (share of the sites that block training) | 59% | 64% | 47% |
Publish a valid llms.txt (has a # Title) | 9.1% | 11.3% | 8.3% |
| Declare a sitemap in robots.txt | 60.9% | 62.8% | 48.8% |
*The US sample is a proxy: top .com domains (global, US-heavy) plus .gov and .edu.
- The US sample is about twice as likely as Canada to block training crawlers (29.3% against 13.2%). Mexico sits between them at 20.2%.
- Full opt-outs are rare: 2.5% to 9.0% block all eight training crawlers.
- The overlap is large. Only 41% of the Mexican sites, 36% of the US sample and 53% of the Canadian sites that opt out of training leave every AI search crawler open.
- About 1 in 10 sites publish a valid llms.txt in all three samples, mostly media, universities and large platforms. Examples include github.com, wordpress.com and adobe.com in the US sample, forbes.com.mx and ibero.mx in Mexico, and queensu.ca and quebec.ca in Canada. See the llms.txt guide for the format.
- Fewer Canadian sites declare a sitemap in robots.txt (48.8% against 60.9% in Mexico and 62.8% in the US sample). Adding one
Sitemap:line fixes it.
Per crawler
Share of sites with a verdict that block each crawler. These numbers count partial blocks.
| Crawler | Group | Mexico | US sample* | Canada |
|---|---|---|---|---|
| GPTBot | Training | 14.4% | 20.7% | 9.5% |
| ClaudeBot | Training | 13.2% | 22.6% | 7.0% |
| Google-Extended | Training | 11.1% | 19.2% | 5.8% |
| Applebot-Extended | Training | 11.1% | 19.9% | 5.4% |
| CCBot | Training | 11.1% | 24.4% | 9.9% |
| Meta-ExternalAgent | Training | 12.3% | 19.2% | 6.6% |
| Amazonbot | Training | 12.8% | 18.4% | 5.0% |
| Bytespider | Training | 10.3% | 24.4% | 6.6% |
| OAI-SearchBot | AI search | 7.8% | 10.2% | 3.7% |
| Claude-SearchBot | AI search | 8.2% | 12.8% | 3.3% |
| PerplexityBot | AI search | 9.9% | 16.9% | 6.2% |
| Googlebot | AI search | 0.8% | 1.5% | 0.0% |
| Applebot | AI search | 1.6% | 2.6% | 1.2% |
| Amzn-SearchBot | AI search | 8.2% | 10.2% | 1.7% |
| DuckAssistBot | AI search | 7.8% | 12.0% | 1.7% |
| ChatGPT-User | User request | 7.8% | 11.7% | 5.4% |
| Claude-User | User request | 9.5% | 12.8% | 2.1% |
| Perplexity-User | User request | 9.5% | 13.9% | 2.1% |
| Meta-ExternalFetcher | User request | 9.1% | 12.8% | 2.5% |
| Amzn-User | User request | 7.4% | 9.4% | 1.2% |
- The most blocked crawler differs by country. Mexico: GPTBot (14.4%), then ClaudeBot (13.2%) and Amazonbot (12.8%). US sample: Bytespider and CCBot tie at 24.4%, then ClaudeBot at 22.6%. Canada: CCBot (9.9%) and GPTBot (9.5%).
- ChatGPT's search crawler is blocked about half as often as GPTBot, or less: OAI-SearchBot against GPTBot is 7.8% against 14.4% in Mexico, 10.2% against 20.7% in the US sample and 3.7% against 9.5% in Canada.
- PerplexityBot is the most blocked AI search crawler in all three samples: 9.9%, 16.9% and 6.2%.
- Googlebot is blocked by 0.8%, 1.5% and 0.0% of sites, and Applebot by 1.6%, 2.6% and 1.2%.
- The five user-request fetchers are blocked at 7.4% to 9.5% in Mexico, 9.4% to 13.9% in the US sample and 1.2% to 5.4% in Canada. Several vendors say robots.txt may not apply to them, so these numbers show what sites ask for, not what is enforced.
Why opting out of training often blocks search too
We did not record which rule produced each block, so we cannot split the overlap by cause. Two things in the data point away from site-wide rules. A Disallow: / under User-agent: * would also block Googlebot unless the site gives Googlebot its own group, and Googlebot is blocked by 0.8% of Mexican sites, 1.5% of the US sample and none of the Canadian sites. And in Mexico the five search crawlers other than Googlebot and Applebot are blocked at 7.8% to 9.9%, close to the 10.3% to 14.4% of the eight training crawlers. That fits block lists that name training and search crawlers together, such as a copied list. We read it as the likely cause, not a measured one.
If you want to opt out of training and stay citable, put only the training tokens in your block list and leave OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed. The robots.txt guide has a file to copy. Then check your CDN and firewall settings, which this study could not see.
Method
- Samples. Majestic Million, list of 2026-10-20. Mexico: the top 300 root domains under
.mx. Canada: the top 300 under.ca. US proxy: the top 200.com, top 50.govand top 50.edu. Majestic ranks by referring subnets, not by traffic. - Requests.
/robots.txt,/llms.txtand/llms-full.txtfor each domain, nothing else. One request at a time per domain, with timeouts, using an honest user agent:AtalayaBot/0.1 (+https://apify.com/atalaya-ai). - Parsing. RFC 9309: user-agent group matching, longest match, Allow wins ties. The 20 tokens fall in 3 groups: 8 model training, 7 AI search and 5 fetchers that act on a user request. A block includes partial blocks of the site root.
- No verdict. Unreachable hosts, robots.txt answering 401, 403, 429 or 5xx, and inconclusive responses: 149 domains (Mexico 57, Canada 58, US sample 34). They are left out of every percentage, which is why 751 of 900 domains are counted.
Limits
- robots.txt is a declaration, not enforcement. Many sites block AI bots in a CDN or firewall (for example with bot-management products), and this study cannot see that. Real blocking is likely higher than these numbers.
- The sites left out for returning 401, 403, 429 or 5xx on robots.txt may include some that block automated requests, which would also push real blocking higher.
- Each country has 242 to 266 sites with a verdict. Treat gaps of a few points as noise.
- The US column is not US-only:
.comincludes global brands. - Popularity by links is not traffic, and this is one snapshot taken on 2026-10-20.
Data and credit
Domain lists come from the Majestic Million by Majestic-12 Ltd, licensed CC BY 3.0. The checks ran with our AI Crawler Access Checker on Apify.
Check your site
Our free check reads your robots.txt the way all 20 crawlers do and shows the line that decides each one.