AI crawler

Sorting AI crawlers by job is the single most useful mental model for site owners, because the bots arriving in your server logs feed very different systems on very different timelines. Training crawlers collect text that teaches future models. Whatever they gather today shapes what a model released next year knows by heart, and once a model ships, that influence is blended beyond attribution and cannot be recalled.

Search index crawlers are the direct analog of classic search engine spiders. They build the retrieval indexes that AI search products query at answer time, and they are the type that produces citations, the visible, linked appearances of your pages inside AI answers. Their effect shows up in days to weeks rather than in model release cycles.

User-triggered fetchers retrieve one specific page in real time because a person asked for it, for example when a buyer pastes your pricing page into a chat and asks for a comparison. OpenAI's ChatGPT-User, Anthropic's Claude-User and Perplexity's Perplexity-User all identify these on-demand requests. They move in seconds and matter most in the final stages of a purchase decision.

Most public arguments about AI crawlers go wrong by collapsing the three types into one decision. A publisher angry about uncompensated training use blocks everything and silently disappears from AI search answers too. Treating each type as its own policy choice keeps the tradeoffs clear.

OpenAI operates the clearest example of the three-type model, with a documented bot for each job: GPTBot for training data, OAI-SearchBot for the ChatGPT search index, and ChatGPT-User for live fetches. Each has its own robots.txt token and its own published IP ranges, so each can be allowed or refused independently. A Botify analysis found OpenAI roughly tripled its web crawl since August 2025 as it builds out its own index.

Anthropic mirrors the structure with ClaudeBot for training, Claude-SearchBot for search indexing and Claude-User for user-directed fetches, all documented as honoring robots.txt. Perplexity runs PerplexityBot to feed its own search index, publicly reported at more than 50 billion pages, plus Perplexity-User for requests a person triggers directly.

Google is the case that trips people up. Its AI features, including AI Overviews, cite from Google's regular organic index, which is crawled by plain Googlebot. There is a separate robots.txt token, Google-Extended, but it is a data-use control for Gemini training rather than a crawler you will ever see in logs. Blocking Googlebot removes you from Google search itself, so the AI decision at Google is really a Google-Extended decision.

Beyond the majors, a long tail of smaller AI bots grows monthly. Two hygiene habits cover it: verify heavy crawlers against each operator's published IP ranges, since user agent strings are trivially spoofed, and review your logs or CDN bot dashboard quarterly, because who reads your site is now a strategic input.

Control lives in robots.txt, the plain text file at the root of your domain. Each compliant bot checks for rules addressed to its token before crawling. To block OpenAI's training crawler, for example, you add the line User-agent: GPTBot followed by Disallow: /. To welcome a bot explicitly, use Allow: / instead. Rules are per bot, so you can block GPTBot while allowing OAI-SearchBot on the same site.

robots.txt is a convention rather than an enforcement mechanism. The documented bots from OpenAI, Anthropic, Google and Perplexity's index crawler state that they honor it, but nothing at the protocol level forces any bot to comply, and Perplexity documents that Perplexity-User generally ignores robots.txt because a human initiated the request. Where compliance matters commercially, CDN-level bot management is the harder backstop.

Misconfiguration is common and usually invisible until traffic is already lost. Sites inherit blanket Disallow rules from staging, security plugins ship default AI blocklists, and CDNs enable bot challenges that stop legitimate crawlers. Reachroller offers a free checker at /bot-access that tests which AI crawlers can actually read your site.

For search index crawlers and live fetchers, access is the precondition for existing in AI answers at all. Engines can only retrieve and cite pages their bots can read, and buyers increasingly ask assistants which products to choose before they ever type a query into classic search. Blocking OAI-SearchBot or PerplexityBot hands your citation slots to competitors whose robots.txt is one line shorter.

Training crawlers are a genuinely two-sided call. Large publishers block them as a licensing position while negotiating content deals, which is a coherent commercial argument. For most small and mid-size brands, whose content is not a licensable asset but whose discoverability is existential, allowing everything remains the default that matches their incentives.

Crawling buys eligibility rather than results. Every competitor is crawled by the same fleet, and the engines then choose a handful of sources per question. Whether crawled pages become cited answers is a separate measurement problem, which is why crawler policy is the start of AI visibility work rather than the end of it.