Obility Editorial · · 4 min read
The API and the app are different products
Your buyers ask their questions in the ChatGPT app, in Gemini, in Perplexity or in Google’s AI Mode. Many tools that report AI visibility do not collect answers there. They send the same prompts to the model through the company’s developer API, because an API call is cheap and returns clean data. The alternative, driving the consumer interface the way a person would, is slower and breaks whenever the interface changes.
The two routes reach the same company and a closely related model, so it is tempting to treat them as interchangeable. The vendors’ own documentation explains why they are not. In the app, the company decides how the model is instructed, whether it searches and what it knows about the user. Through the API, whoever writes the request decides most of that.
The caller decides the search, the instructions and the location
OpenAI’s developer documentation describes web search as a tool that a developer adds to a request. Once it is added, the model chooses whether to search based on the prompt. The developer can also set how much search context the model receives, limit or block up to 100 domains, and pass an approximate user location by country, city, region and time zone. Google’s documentation for grounding with Google Search in the Gemini API works the same way: the tool is enabled per request, and the model then decides whether a search would improve the answer.
Anthropic is the most direct about instructions. It publishes the system prompt that Claude’s web interface and mobile apps use, says the prompt is updated periodically, and states that those updates do not apply to the Claude API.
So an answer collected through an API reflects a set of choices made by the tool builder: which model, whether search was switched on, how much of each result was passed in, and which location was given. None of the apps publishes its own settings for comparison. Some surfaces have no matching API at all. Surfer, describing its own test, had to reuse its AI Mode request to stand in for AI Overviews because no model matching the one behind AI Overviews is sold.
The largest published comparison found wide gaps
Surfer, which sells tracking built on scraped interface data, ran 1,000 prompts through ChatGPT, Perplexity, Gemini, AI Mode and AI Overviews twice: once by scraping the consumer product and once through an API. It collected 13,779 answers in August 2026 and updated its write-up on 4 October 2026.
The brands named for the same prompt overlapped by 15.5 to 23.8 percent between the two routes, rising to between 21.3 and 31.6 percent after spelling variants of a company name were merged. ChatGPT showed the largest differences. Its API named 13.8 brands per answer against 7.9 in the app, wrote about twice as many words, and cited 3.1 sources against 12.1. The direction was not uniform: Perplexity’s API cited more sources than its app did, 19.5 against about 10.3.
For a marketing team, the brand count matters most. If an API answer names fourteen companies and the app names eight, a tool built on the API will credit you with appearances that no buyer saw.
Part of that gap is ordinary noise, by the study’s own account
Read the overlap figures with the study’s caveats attached, because Surfer printed them. It took one sample per prompt, and it notes that the same prompt asked twice does not return the same answer, so some of the disagreement comes from the models varying rather than from the two routes. The study does not report how much two app runs agree with each other, which is the baseline the API figures should be measured against. Our earlier piece on answers changing from run to run covers why that baseline is rarely high.
The averaged counts are on firmer ground than the overlap percentages. A gap of 13.8 against 7.9 brands across a thousand prompts is unlikely to be chance. A 25 percent overlap from single runs probably overstates how far apart the routes are, by an amount nobody has measured.
Scraping has limits of its own. A scraper submits prompts from one account state, one location and one moment, and the earlier piece on AI memory explains how much those change an answer. Interface data is closer to what buyers see because it runs through the product they use. It is still a sample, and a clean session is not your buyer’s session.
Test the tool against the app before you trust its numbers
Any team can check a tool in an afternoon. Take twenty of your buying prompts. Run each five times in the app itself, in clean sessions, then rerun the same set a second time. Record how often your brand appears in each round, then compare the tool’s figure for the same prompts and week. As an illustration, if the app names you in 48 percent of answers one round and 56 percent the next, a tool reporting 52 percent is consistent with it. A tool reporting 85 percent is measuring something else.
Then ask any vendor, us included, six questions: which surface each answer came from, interface or API; which model; whether search was on; which location and language; whether the session was logged in; and how many runs sit behind each figure. A vendor that cannot answer has not decided what it is measuring.
At Obility we confirm the supported engines and collection methods for each program during onboarding, and every observation keeps its language, geography, prompt wording and collection time attached, so a figure can be traced to how it was gathered.
API answers are not worthless. With search switched off, they show what a model says from training alone, which helps when tracing where a wrong claim came from. The problem is a label: an API answer reported as the answer your buyers see.