Obility Editorial · · 4 min read
The follow-up is where the shortlist gets cut
A buyer rarely stops at the first answer. They ask ChatGPT or Perplexity for the best CRM for a small agency, read the four names it gives, then ask which of those connects to HubSpot, or which is cheapest for ten seats. The list they act on is the one after the follow-up, and a tracker that only asks the opening question reports the list they saw first.
That point is sound, and a growing number of AEO guides now make it. The trouble is the evidence they attach. Two figures keep turning up: AI models perform 39 percent worse in multi-turn conversations, and only 20 percent of brands survive five consecutive answers. Neither study measured what those guides use them for.
The 39 percent drop comes from code and arithmetic
The first figure comes from a May 2025 preprint by researchers at Microsoft Research and Salesforce Research, titled “LLMs Get Lost In Multi-Turn Conversation”. They tested 15 models in more than 200,000 simulated conversations across six tasks: writing Python functions, writing SQL queries, producing API calls, solving grade school math word problems, captioning tables and summarising a set of about twenty documents. Each task started as one fully stated instruction, which the researchers split into pieces and revealed one turn at a time. Averaged across the tasks, performance fell by 39 percent.
The paper splits that drop into two parts. Aptitude, a model’s best case score, fell by about 15 percent. Unreliability rose by 112 percent, and the paper defines it as the gap between a model’s best and worst runs of the same instruction, the 90th against the 10th percentile score. At least one AEO guide describes unreliability as incorrect but confidently stated output, which is not what was measured. No task involved recommending a product, and none used web search.
What the paper does offer is a mechanism. The authors found that models make assumptions early, attempt a final answer before they have all the details, and then lean on that first attempt. It is reasonable to suspect that an opening shortlist anchors a conversation the same way. That is a hypothesis to test in your category, not a measured result.
The 20 percent figure counts repeat runs, not turns
The second figure traces to AirOps’ 2026 State of AI Search report, which says only 30 percent of brands stay visible from one answer to the next and 20 percent remain present across five consecutive runs. The report explains the instability by saying models rebuild the answer from scratch each time. That describes asking the same question again in fresh sessions. At least one guide on follow-up questions restates it as brands persisting across five answers in a conversation, which is a different measurement altogether.
A follow-up either reuses the list or searches again
OpenAI’s help center says ChatGPT may search the web automatically when a question would benefit from current information, and that it typically rewrites the question into one or more targeted queries for its search providers. A follow-up such as “which of these works with HubSpot” means nothing to a search engine on its own, so when ChatGPT does search again, the rewritten query has to carry the conversation’s subject along with the new constraint. None of the engines publish when a follow-up triggers a fresh search and when it does not.
That gives a follow-up two paths. If the engine answers from the list already on the screen, a brand missing from the first answer has almost no way in. If it searches again, a page that states the constraint plainly, such as a named integration, a team size or a region, can bring a new brand into the reply. Which path your buyers’ follow-ups take is something you can observe, and the fix differs for each.
Test the follow-up two ways, then track it as its own question
Take three follow-ups your sales team hears most often. Run each as a two-turn conversation: the opening question, then the follow-up. Then run it again as a single question that combines both, which mirrors the paper’s own advice to users to consolidate everything into one instruction in a new conversation. Repeat each version several times on each engine, because answers vary from run to run.
Read the results in pairs. If you appear in the combined question but drop out after the follow-up, the first answer is anchoring the conversation, and your problem is the opening shortlist. If you are missing from both, no page the engine can find answers that constraint for you. If you appear in both, that follow-up is not where you lose buyers.
Obility’s Answer Engine Insights groups the buying questions you track and keeps the answer, the brands it names and the sources it cites with each observation, so a combined follow-up question can sit in the prompt set beside its opener and show which page each engine relied on. The engines covered for your programme are confirmed at the demo. One limit stands whatever you measure: no published study has yet tracked brand presence across the turns of real buying conversations, so treat any single figure for it, including your own early test, as a starting point.