robots.txt for AI Crawlers: GPTBot, ClaudeBot, PerplexityBot and More

If your robots.txt blocks AI crawlers, your store does not exist in ChatGPT, Perplexity, or Gemini answers — the models simply build their recommendations from competitors' pages instead. This guide covers every major AI crawler user-agent, the allow-versus-block tradeoff, and a complete copy-paste robots.txt for stores that want AI-driven sales.

Why does AI crawler access matter for e-commerce?

Because blocked means invisible. When a shopper asks ChatGPT "what's a good mid-priced hiking pack with replaceable buckles?", the answer is assembled from pages the underlying systems could crawl and index. If your robots.txt (or your CDN's bot protection) turns AI crawlers away, your products cannot appear in that answer — no matter how good your prices, reviews, or classic SEO are. This is the foundational layer of generative engine optimization: before content quality or structured data matters, the crawlers have to be able to reach you at all.

This is different from traditional SEO in one important way: there is no "we'll find you anyway." Reputable AI crawlers honor robots.txt, so an accidental Disallow from years ago — often added during the 2023 wave of AI-blocking defaults — silently removes you from an entire discovery channel.

Should you allow or block AI crawlers?

For a store that wants AI-driven sales, allowing is usually the right call — but the tradeoff is real, so let's state it honestly.

The case for blocking: some AI crawlers (GPTBot, Google-Extended, CCBot, and others) collect content for model training. If you publish original editorial content, photography, or proprietary data, you may object to it being used to train models without compensation. Media companies have litigated exactly this. Blocking training crawlers is a legitimate position.

The case for allowing: e-commerce product pages are not primarily creative works — they are offers you want distributed as widely as possible. Every surface where your product name, price, and differentiators can appear is a sales channel, and AI assistants are becoming one. Blocking AI crawlers to protect a product description is protecting the thing you most want copied.

The practical middle ground: the crawler ecosystem separates training bots from search/answering bots. You can block training-focused agents (GPTBot, Google-Extended, CCBot) while allowing the ones that power live answers and search indexes (OAI-SearchBot, ChatGPT-User, PerplexityBot). Most stores we work with allow everything, because product visibility in future model knowledge is also worth having — but the split configuration exists if you want it.

AI crawler user-agents: the reference table

These are the user-agent tokens you target in robots.txt. The key insight: several companies run multiple crawlers with different jobs, and you control each one independently.

If AI visibility is the goal, the highest-priority allows are OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Googlebot, and Bingbot — these are the agents most directly involved in producing cited, linked answers today.

A copy-paste robots.txt that allows AI crawlers

This configuration is modeled on the live rynex.io/robots.txt, which explicitly allows the full set of AI crawlers above. Adapt the Disallow lines and Sitemap URL to your store:

# robots.txt — explicitly allow AI crawlers
# Keep private paths blocked for everyone via the catch-all group.

# --- OpenAI ---
User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

# --- Anthropic ---
User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

# --- Perplexity ---
User-agent: PerplexityBot
Allow: /

# --- Google ---
User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Allow: /

# --- Microsoft ---
User-agent: Bingbot
Allow: /

# --- Meta ---
User-agent: Meta-ExternalAgent
Allow: /

# --- Apple ---
User-agent: Applebot
Allow: /

# --- Other AI crawlers ---
User-agent: cohere-ai
Allow: /

User-agent: CCBot
Allow: /

User-agent: YouBot
Allow: /

User-agent: Bytespider
Allow: /

User-agent: DuckAssistBot
Allow: /

User-agent: Diffbot
Allow: /

User-agent: Timpibot
Allow: /

User-agent: PetalBot
Allow: /

# --- Everyone else ---
User-agent: *
Allow: /
Disallow: /cart
Disallow: /checkout
Disallow: /account
Disallow: /admin

Sitemap: https://www.yourstore.com/sitemap.xml

Two things to understand about how robots.txt matching works:

1. A crawler obeys the most specific group that names it. Because GPTBot has its own Allow: / group above, it does not inherit the Disallow: /cart lines from the * group. If you want AI crawlers kept out of cart/checkout/account too (you almost certainly do — those pages are useless to them), copy the Disallow lines into each named group, or accept that well-behaved bots have little reason to crawl them anyway. For strictness, duplicate the disallows.
2. The Sitemap line is global and belongs in the file regardless of groups. It is also one of the most commonly forgotten lines — more on that below.

If you'd rather keep training bots out while staying visible in AI search, change the GPTBot, Google-Extended, CCBot, and Bytespider groups to Disallow: / and leave the rest allowed.

How do you verify AI crawlers are actually reaching you?

Check your server logs — robots.txt states policy, logs show reality. Grep your access logs (or your CDN's log export) for the user-agent tokens:

grep -iE "GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|PerplexityBot|Google-Extended|Meta-ExternalAgent|CCBot|Bytespider|Applebot" access.log \
  | awk '{print $1, $12}' | sort | uniq -c | sort -rn | head -25

(Adjust the awk field numbers for your log format.) You're looking for two things: that the crawlers show up at all, and that they're receiving 200 responses rather than 403s. On Cloudflare, the bot analytics dashboard breaks out verified AI bots by name, which gets you the same answer without touching raw logs. Crawler hits are a different signal from human referral visits — for the human side, see tracking AI search traffic in GA4.

Common mistakes

Blocking at the CDN/WAF while robots.txt allows. This is the most common and most damaging failure we see. Cloudflare's "block AI bots" toggle, bot-management rules, and generic WAF challenges will 403 or JS-challenge AI crawlers regardless of what your robots.txt says. The result is a store that believes it is open to AI crawlers and is actually invisible. Your robots.txt is a promise; your edge configuration decides whether you keep it. Audit both.

Forgetting the Sitemap line. AI search crawlers use sitemaps for discovery just like classic ones. A robots.txt without a Sitemap: directive forces crawlers to discover your catalog by link-walking, which means deep products get found late or never.

Stale disallows from 2023. Many sites added blanket AI-crawler blocks when GPTBot first launched and never revisited the decision. If AI visibility now matters to your business, that old paste-in is quietly costing you.

Relying on robots.txt for security. Disallow is a request, not a lock. Anything genuinely private needs authentication, not a robots.txt line (which also happens to advertise the path).

Access is step one. Once crawlers can reach you, what they find matters: a curated llms.txt file gives them canonical brand facts, and Product JSON-LD gives them machine-readable offers. The free GEO Audit checks all three automatically — robots.txt AI-crawler rules, llms.txt presence, and structured data — so you can see in one pass whether your store is actually reachable by the engines you want recommending it.

FAQ

Should my e-commerce store allow GPTBot in robots.txt?

If you want your products recommended in ChatGPT answers, yes. Blocking GPTBot and OAI-SearchBot removes your store from OpenAI's index, which means AI shopping answers get built from competitors' pages instead of yours.

What is the difference between GPTBot, ChatGPT-User, and OAI-SearchBot?

GPTBot collects data for training OpenAI models, ChatGPT-User fetches pages live when a ChatGPT user asks about your site, and OAI-SearchBot builds the index behind ChatGPT search. They are controlled separately in robots.txt.

Does blocking Google-Extended hurt my Google rankings?

No. Google-Extended only controls whether your content is used for Gemini model training. Regular Googlebot crawling, indexing, and rankings — including AI Overviews sourcing — are unaffected by it.

How do I verify AI crawlers are actually reaching my site?

Check your server or CDN logs for the crawler user-agent strings. robots.txt only states your policy; a firewall or bot-protection rule can still block crawlers your robots.txt allows, and logs are the only way to see that.

Do all AI crawlers respect robots.txt?

The major ones from OpenAI, Anthropic, Google, and Apple document that they honor it. Compliance elsewhere is uneven, so treat robots.txt as a policy signal for reputable crawlers, not an enforcement mechanism.