Obility Editorial · · 4 min read
A form stops the crawlers that feed AI answers
B2B teams put their best research behind a form: the benchmark report, the buyer’s guide, the survey with real numbers in it. That trade made sense when the job of the landing page was to rank and convert. It works less well when buyers ask an AI assistant the question the report answers.
An answer engine can only use what its systems retrieved. OpenAI’s crawler documentation describes OAI-SearchBot as the crawler used to surface websites in ChatGPT’s search features, and ChatGPT-User as the agent that visits a page when someone asks ChatGPT a question. Both request a URL and read what comes back. A crawler collecting pages for search does not type a work email into your form. What it reads is the landing page: a headline, three teaser bullets and a field asking for a phone number. That is everything the answer can learn about your research.
There is a common exception worth checking first. Many “gated” PDFs sit at a public URL that is linked from the thank-you page or a confirmation email, and once that link appears anywhere public, the file can be crawled like any other. Search for the file name before assuming it is hidden. If the PDF is already indexed, you have the drawbacks of a gate without the protection.
Google allows a gate, but only on its own terms
Google has written rules for this case, and they are narrower than most advice suggests. Its spam policies say a paywall or content gate is not treated as cloaking if Google can see the full content behind it, just as a person with access would, and if the site follows its Flexible Sampling guidance. That guidance states that “paywall” covers barriers that require registration, not only subscription, so a lead form counts. It describes two acceptable patterns: metering a number of free reads, and a lead-in that shows the opening of a piece before the barrier.
Google also publishes structured data for the case: mark the page with isAccessibleForFree set to false and point a cssSelector at the gated section. Its AI features documentation says a page only needs to be indexed and eligible for a snippet to appear as a supporting link in AI Overviews or AI Mode, with no additional requirements. So a properly marked registration wall can still feed Google’s AI answers.
That arrangement is Google’s alone. OpenAI’s crawler documentation describes its user agents and robots.txt controls and says nothing about recognising paywall markup. Serving the full text to bots by user agent while people see a form, outside that marked-up arrangement, is close to the example Google gives in its own cloaking definition: inserting content only when the requester is a search engine.
A citation that lands on a form is a poor first visit
Suppose the bots can read the full report and a buyer cannot. An AI answer cites your report for a striking figure. The buyer clicks the citation to check it, which is exactly what a careful buyer does, and meets a form asking for a company size and phone number. The answer has made a claim on your behalf that the reader cannot verify without becoming a lead. Some will fill in the form. Many will go back and click the next source, which is often a competitor’s summary of the same topic.
The opposite failure is quieter. If nothing about the research is readable, answers about your category are built from whoever published openly, and your findings are absent from the comparison entirely.
Publish the findings and gate the working material
The split that holds up is to make the conclusions public and keep the form in front of what someone would actually work with. Put the question, the method, the sample, the headline findings and the limitations on an ordinary HTML page. Write each finding as a sentence that stays true when lifted out of context, with the number and what it was measured on in the same sentence. Then gate the dataset, the full workbook, the templates or the calculator.
As an illustration, a 30-page benchmark becomes an open page with eight findings and a method note, plus a form for the raw data and the scoring model. Buyers comparing vendors get a citable answer, and the people who fill in the form are the ones who want to use the data rather than read a summary of it. Avoid putting the findings only in a PDF; an HTML page is easier to retrieve, quote and link to a specific section.
Be honest about what this changes. Opening the findings makes them available to answer engines. It does not make them cited, and nothing any assistant publishes promises that it will. Form completions on that asset may fall. Treat it as a test on one report before rewriting the whole library.
Measure the open page, the citations and the leads together
Run the test with three signals kept apart. Crawler requests to the open page show it is being fetched, though user-agent identification is a signal and not verified bot identity. Answer observations on the prompts the report answers show whether your page is cited, or whether a third party summarising your work is cited instead. Referral visits from answer engines show whether buyers arrive at the open page and what they do next, including the gated download.
In Obility, Agent Analytics keeps crawler activity separate from referral visits, and Answer Engine Insights records the sources each answer cites for a fixed set of prompts, so a before and after comparison on one report is a like-for-like reading. If citations and AI referrals rise while gated downloads hold, open the next report. If neither moves after a reasonable period, the problem was the research or the question, not the form.