The most expensive line in AI search is a Disallow nobody remembers writing. AI crawlers are separate from Googlebot — block them and you vanish from ChatGPT, Perplexity and Claude while your Google rankings stay perfectly healthy, which is exactly why the problem survives for months undetected.
This is the configuration guide: every major AI user agent, what it does, what blocking it costs you, and how to write a robots.txt that reflects a deliberate decision rather than an accident. Written for UAE brands who cannot work out why their well-ranked site never gets cited.
Why this is the first thing to check
The failure is usually invisible. Google traffic looks healthy, rankings hold, and nothing in your analytics suggests a problem — because the loss is in a channel you were never measuring. Meanwhile every AI assistant your buyers use treats your site as though it does not exist.
The AI user agents that matter
| User agent | Operator | What it does | Blocking it means |
|---|---|---|---|
OAI-SearchBot | OpenAI | Indexes pages for ChatGPT search | Excluded from ChatGPT search |
ChatGPT-User | OpenAI | Fetches a page on user action | ChatGPT cannot read your page live |
GPTBot | OpenAI | Collects training data | Absent from future model knowledge |
PerplexityBot | Perplexity | Indexes pages for Perplexity | Cannot be cited by Perplexity |
Perplexity-User | Perplexity | Fetches on user action | Live lookups fail |
ClaudeBot | Anthropic | Crawls for Claude | Absent from Claude |
Google-Extended | Controls Gemini training use | Excluded from Gemini training — does not affect AI Overviews | |
Applebot-Extended | Apple | Controls Apple AI training use | Excluded from Apple Intelligence training |
Two distinctions do most of the work here. First, search crawlers versus training crawlers: OAI-SearchBot and PerplexityBot decide whether you can be retrieved and cited today; GPTBot, Google-Extended and Applebot-Extended concern model training. Blocking training bots is a defensible business choice. Blocking search bots is self-harm.
Second, Google-Extended does not remove you from AI Overviews. Those are built from Google's ordinary search index. Blocking it protects nothing about your clicks — a point we cover in detail in our AI Overviews guide.
A sane configuration
For most commercial brands, the goal is straightforward: allow everything that leads to citations, and make the training decision consciously.
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: ClaudeBot
Allow: /
# Training — a deliberate business decision
User-agent: GPTBot
Allow: /
# Everything else
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml
robots.txt is rarely the real culprit
In our audits, the file is usually fine and something else is doing the blocking. Check these in order:
Should you block AI crawlers?
There is a real argument for publishers whose product is the content itself. If your revenue depends on people reading your articles on your site, letting models absorb and summarise them is a genuine business threat, and blocking training bots is rational.
For a services business — an agency, clinic, developer, restaurant — the calculus is the opposite. Your content is marketing, not inventory. Being summarised is the point. Blocking AI crawlers to protect content that exists to attract clients is a category error, and it is one we see regularly on UAE sites where the decision was made by an IT team optimising for a different objective entirely.
Once access is confirmed, the next constraints are content structure and corroboration — covered in our AEO service and the ChatGPT guide.
Frequently Asked Questions
Which AI crawlers should I allow?
For a services business, allow the retrieval crawlers: OAI-SearchBot and ChatGPT-User for ChatGPT, PerplexityBot and Perplexity-User for Perplexity, and ClaudeBot for Claude. These determine whether you can be cited at all. The training-related agents — GPTBot, Google-Extended and Applebot-Extended — are a separate, legitimate business decision, since they affect model training rather than live retrieval.
Does blocking Google-Extended remove me from AI Overviews?
No. Google-Extended controls whether your content may be used to train Gemini models. AI Overviews are generated from Google's ordinary search index, so blocking Google-Extended has no effect on whether you appear in them. There is currently no way to appear in Google Search while opting out of AI Overviews specifically.
Why is my site invisible to ChatGPT even though robots.txt allows it?
Usually something upstream is blocking the request. Bot-mitigation services such as Cloudflare Bot Fight Mode challenge non-browser agents wholesale; old WAF rules written to stop scrapers catch modern AI crawlers; aggressive rate limits cause timeouts; and geo-blocking to UAE IPs excludes crawlers operating elsewhere. Content rendered only by JavaScript has the same practical effect. Test by fetching your page with the crawler's user-agent and checking what actually comes back.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects data used to train OpenAI's models. OAI-SearchBot builds the index that ChatGPT search draws on when answering questions that need current information. Blocking GPTBot affects your long-term presence in model knowledge; blocking OAI-SearchBot removes you from ChatGPT search immediately. If you only allow one, allow OAI-SearchBot.
Should I block AI crawlers to protect my content?
It depends on your business model. For publishers whose revenue depends on readers visiting their pages, blocking training crawlers is a rational defence. For a services business — agency, clinic, developer, restaurant — content exists to attract clients, so being summarised and cited is the objective. Blocking AI crawlers to protect marketing content is usually a mistake, and often one made by an IT team optimising for a different goal.
How do I test whether AI crawlers can reach my site?
Fetch your own page using the crawler's user-agent string with curl and inspect the response: a 200 with your real text means access is fine, while a 403, a challenge page or an empty shell means something is blocking you. Then ask ChatGPT and Perplexity, with search enabled, to summarise your business — if they cite only directories and never your own domain, retrieval is failing regardless of what your config says.