Obility Editorial · · 4 min read
The platform decides which engines can read your post
A lot of current AEO advice says LinkedIn is one of the most cited sites in AI answers, so the way into those answers is to post more there. The advice skips a step. An answer engine can only cite a page its crawler is allowed to fetch, and LinkedIn, X and Medium each publish their own rules about which crawlers get in. Those rules differ by engine, so the same post can be readable to one assistant and closed to the next.
The AI companies make this easy to check because each one uses separate crawlers for separate jobs. OpenAI’s crawler documentation lists OAI-SearchBot for ChatGPT’s search features, GPTBot for collecting training data, and ChatGPT-User for pages fetched while answering a user. Anthropic’s help center describes the same split for Claude: Claude-SearchBot, ClaudeBot and Claude-User. Perplexity’s documentation lists PerplexityBot for its search results and Perplexity-User for fetches a user asks for. Google’s own documentation says a page must be indexed in Google Search to appear as a link in AI Overviews or AI Mode, and Google Search crawls with Googlebot.
LinkedIn admits two AI search crawlers and shuts out Perplexity
We read LinkedIn’s robots.txt file on 5 November 2026. It runs to more than 4,800 lines. Googlebot and Bingbot may fetch public posts, Pulse articles, company pages and profiles. OAI-SearchBot and Claude-SearchBot each have their own group of rules that allows the same kinds of pages. PerplexityBot is disallowed from the whole site, and so are GPTBot, ClaudeBot and Google-Extended. The last rule in the file, for any crawler it does not name, disallows everything.
Read plainly, a public LinkedIn post can enter the indexes behind ChatGPT search, Claude’s search, Google and Bing. Perplexity’s own search crawler is told to stay out. So when a report says LinkedIn is among the most cited domains in AI answers, ask which engine it measured. The figure for ChatGPT cannot tell you much about Perplexity.
The training crawlers are a separate matter. LinkedIn has asked GPTBot and ClaudeBot to stay away, so a post written today should not be expected to reach those companies’ future models through their crawlers. What a model already knows about your brand, before it searches anything, will have to come from somewhere else.
X is closed to every AI crawler except through Google and Bing
X’s robots.txt gives Googlebot and Bingbot a shared, mostly open set of rules. It then names Google-Extended and several of Meta’s agents and disallows them, and ends with a rule that disallows every other crawler from the whole site. None of the AI search crawlers named above has its own group in that file, so all of them fall under the final rule. A thread on X can surface in Google’s and Bing’s results, but OpenAI’s, Anthropic’s and Perplexity’s own search crawlers are told not to read it.
Medium sits at the other end. Its robots.txt blocks GPTBot, ClaudeBot and a handful of other training and data crawlers by name, and leaves everything else, including OAI-SearchBot, Claude-SearchBot and PerplexityBot, under general rules that allow published stories. A Medium post is open to all three AI search crawlers and closed to two training crawlers.
Put the full argument where every engine can reach it
Keep posting on LinkedIn, since it reaches people directly and reaches some engines. But keep the version you want cited, the definition, the method, the comparison logic, on your own site, where you set the access rules for every engine. Then use the platform post to point people to that argument and treat its AI reach as partial.
Checking a platform takes about ten minutes. Open its robots.txt, look for a group that names the crawler you care about, and fall back to the group for all crawlers if there is none. Within a group, the longest matching rule wins, as the robots exclusion standard published by the IETF specifies. Recheck now and then, because these files change; LinkedIn’s names dozens of agents.
Then compare the rules with what the engines actually cite. Obility’s Answer Engine Insights keeps the sources each answer cites alongside the answer and the engine that gave it, so a platform that shows up in one engine’s citations and never in another’s is visible rather than averaged into one score. If LinkedIn appears in ChatGPT’s citations for your buying questions and never in Perplexity’s, the robots file has already explained why.
A robots file is a ceiling, not a forecast
A robots.txt file says what a platform permits. It does not say what any crawler has indexed. An allowed page may never be fetched, may show a login wall to a logged-out crawler, or may simply lose to a better source. Perplexity says its user fetcher generally ignores robots.txt because a person asked for the page, and OpenAI says robots.txt rules may not apply to ChatGPT-User, so a buyer who pastes a post’s address into a chat may still get it read. That is a different path from being found by someone who never saw the post.
The file also cannot show content agreements between companies, or training data gathered before the current rules were written. And a blocked platform can still shape answers when other sites quote it. Use the rules to know where a post cannot be read today, and use citation tracking to learn where it is.