OAI-SearchBot

OAI-SearchBot is operated by OpenAI and documented in its crawler reference at developers.openai.com, where its purpose is stated plainly: surfacing websites in the search results inside ChatGPT. It is the search index crawler in OpenAI's three-bot fleet, alongside GPTBot, which collects training data, and ChatGPT-User, which fetches individual pages live during conversations. Each bot has its own robots.txt token and its own published IP list, so each is a separate decision for site owners.

The index OAI-SearchBot builds is what ChatGPT queries when a user's question triggers web search. ChatGPT search launched on Bing's index, but OpenAI has been building out its own crawl since, and a Botify analysis found OpenAI roughly tripled its web crawling since August 2025. That growth is largely the index buildout, which means the window in which OAI-SearchBot discovers and indexes your site is now.

The full user agent string documented by OpenAI is: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot. Version numbers change over time, so filter logs on the OAI-SearchBot token, and verify authenticity against the IP ranges OpenAI publishes at openai.com/searchbot.json, since scrapers spoof famous bot names.

OAI-SearchBot honors robots.txt rules addressed to its token. To allow it explicitly, add User-agent: OAI-SearchBot followed by Allow: /. To block it, use User-agent: OAI-SearchBot followed by Disallow: /. Because OpenAI separated its bots precisely so these choices are independent, allowing OAI-SearchBot does not opt you into training, and blocking GPTBot does not touch your ChatGPT search presence.

The most common publisher configuration reflects that split: GPTBot disallowed as a content licensing position, OAI-SearchBot and ChatGPT-User allowed so pages stay retrievable and citable. Check that nothing upstream overrides your intent, because security plugins, managed hosting defaults and CDN bot rules frequently block AI crawlers wholesale, including the ones you meant to welcome. A blanket User-agent: * Disallow rule inherited from a staging setup will also stop OAI-SearchBot cold.

Discovery works the way classic search crawling does, so classic hygiene pays off. OAI-SearchBot finds pages by following links and reading sitemaps, and it can only index what it can fetch cleanly: correct status codes, reasonable response times, content present in the served HTML rather than assembled entirely by client-side JavaScript. Sites that are already well built for Googlebot rarely need anything bespoke for OAI-SearchBot beyond confirming the robots.txt group and letting the crawler reach the sitemap.

One subtlety worth knowing: blocking OAI-SearchBot removes your pages from ChatGPT's search index, but ChatGPT can still mention your brand from model memory or cite third-party pages that describe you. Your robots.txt controls whether your own pages can be sources, and controls nothing about the rest of the web's account of your brand.

Blocking OAI-SearchBot means your pages cannot be retrieved or cited when ChatGPT search answers questions in your category. Estimates put ChatGPT search in the range of hundreds of millions of queries per week, and a growing share of buying research starts as a question to an assistant rather than a keyword search. Every one of those answers draws on sources the crawler could reach, so a blocked site forfeits its citation slots to competitors by default.

Unlike blocking a training crawler, which is a defensible licensing stance with delayed and diffuse effects, blocking a search index crawler has immediate and direct commercial consequences. Citations in ChatGPT answers carry attribution and links, which makes them the closest AI equivalent of ranking in classic search. For a brand whose economics depend on being discovered, refusing the crawler that enables those citations is self-inflicted invisibility.

Access is necessary rather than sufficient. Once OAI-SearchBot can read your pages, ChatGPT still chooses a handful of sources per question, favoring content structured to be quotable. The Princeton GEO research measured that effect: statistics, quotations and cited sources raised visibility in generative engine responses by up to 40 percent. So the working sequence is: confirm crawl access, structure pages to be citable, then measure what answers actually say.

Both ends of that sequence are checkable. The free tool at /bot-access tests whether OAI-SearchBot and its peers can read your site, and Reachroller tracks live ChatGPT answers through the official API with web search enabled to show whether crawled pages are turning into brand mentions.