Obility Editorial · · 4 min read
Each vendor runs separate bots for separate jobs
A common robots.txt change goes like this. Someone reads that AI companies train on web content, adds a Disallow rule for every AI user agent they can find, and months later wonders why the brand rarely appears as a source in ChatGPT or Perplexity. The rule did what it said. It covered more jobs than anyone intended.
The major answer engine vendors now separate their crawlers by purpose. OpenAI’s crawler documentation lists GPTBot for collecting content that may be used to train its foundation models, OAI-SearchBot for surfacing sites in ChatGPT’s search features, and ChatGPT-User for pages visited when a person asks ChatGPT a question. Anthropic’s help center describes the same split for Claude: ClaudeBot for training, Claude-SearchBot for search indexing and Claude-User for retrieval a user initiates. Perplexity documents PerplexityBot for its search results and Perplexity-User for fetches made while answering a question, and states that PerplexityBot is not used to crawl content for foundation models.
The useful mental model is a grid with one row per vendor and one column per job: training, search indexing, and fetching for a live question. Your robots.txt makes a decision in every cell, whether you meant it to or not.
Blocking training bots leaves search eligibility intact
OpenAI states that each setting is independent. A site can allow OAI-SearchBot to appear in search results while disallowing GPTBot to signal that its content should not be used for training. OpenAI also says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links, and that a robots.txt change can take about 24 hours to reach its search systems. Perplexity gives a similar window of up to 24 hours.
So the decision about training and the decision about citation can be made separately. A company that wants its content kept out of model training, and still wants to be eligible as a cited source, can disallow GPTBot and ClaudeBot while allowing OAI-SearchBot, Claude-SearchBot and PerplexityBot. A blanket rule that disallows every agent with GPT, Claude or Perplexity in its name makes both decisions at once.
Google keeps AI Overviews under Googlebot, not Google-Extended
Google is the case that trips people up. Its crawler documentation describes Google-Extended as a robots.txt product token with no user agent of its own. It governs whether content Google crawls may be used to train future Gemini models and for grounding in the Gemini apps and in Grounding with Google Search on Vertex AI. The same page says Google-Extended does not affect a site’s inclusion in Google Search and is not a ranking signal.
AI Overviews and AI Mode sit inside Search. Google’s guidance on AI features says a page must be indexed and eligible to show in Search with a snippet to appear as a supporting link, with no additional technical requirements. The controls it names are robots.txt rules for Googlebot and the nosnippet, data-nosnippet, max-snippet and noindex directives, all of which also shape how the page appears in regular results. Disallowing Google-Extended therefore does not remove you from AI Overviews, and Google’s documentation does not describe a control that affects AI features alone.
Live fetches follow different rules at each vendor
The third column is the least consistent. OpenAI says that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply, and that ChatGPT-User is not used to decide whether content can appear in search. Perplexity says Perplexity-User generally ignores robots.txt for the same reason. Anthropic takes the opposite position: its help center says disabling Claude-User prevents Claude from retrieving your content in response to a user query and may reduce your visibility for user-directed web search.
That has two practical consequences. Blocking Claude-User opts you out of something real. And a ChatGPT-User or Perplexity-User request to a path you disallowed is consistent with those vendors’ published policies, so it is not by itself evidence of a misbehaving bot.
Your firewall can overrule your robots.txt
A robots.txt file is a request. A bot rule at your CDN or web application firewall is enforcement, and it is often configured by a different team for a different reason. Perplexity’s documentation tells site owners who use a firewall that they may need to explicitly allow its bots, and recommends matching on both the user agent and its published IP ranges. Anthropic warns that blocking its IP addresses may not work as an opt-out, because it stops the crawler from reading your robots.txt at all.
This is where an audit usually finds the surprise: a robots.txt that welcomes search bots, sitting behind a firewall rule that answers them with an error.
Audit the grid from your logs, then watch the answers
Write the grid down with your intended decision in every cell, then compare it with what actually happens. Read robots.txt against each named token, including wildcard groups that catch agents you never listed. Then pull your server or CDN logs for each user agent and note the status codes returned. A search bot that keeps receiving 403 responses is blocked, whatever robots.txt says.
Treat the user-agent string as a claim. OpenAI, Anthropic and Perplexity publish the IP addresses their bots use, and a request that names a bot but comes from elsewhere should not be counted. Obility’s Agent Analytics is built around the same distinction: a crawler request identified by user agent is a useful signal, not verified bot identity, and it is kept apart from citations and from human referral visits.
The boundary worth keeping in mind: allowing search crawlers makes your pages eligible to be retrieved and cited. It does not make them cited. Whether an engine names you still depends on whether your pages answer the question better than the sources it already trusts. After a change, give it the day or so the vendors describe, then compare answers to the same prompts before and after, rather than reading crawler hits as a proxy for mentions.