robots.txt for AI crawlers

robots.txt predates AI by three decades. It began as a 1994 convention and was formalized as RFC 9309 in 2022: a plain text file at the domain root where each rule group names a user agent token and lists what that agent may or may not fetch. AI operators adopted the same mechanism, so controlling AI crawlers requires learning which tokens exist and what each one feeds rather than learning any new syntax.

The grammar is three lines deep. User-agent: GPTBot followed by Disallow: / blocks OpenAI's training crawler everywhere. Disallow with a path, such as Disallow: /internal/, blocks a section. Allow: / grants explicit permission, useful when a wildcard group elsewhere in the file is restrictive. An empty Disallow: line permits everything for that agent. Rules bind per token, so a file can welcome OAI-SearchBot in one group while refusing GPTBot in the next.

Two inherited behaviors cause most AI-related accidents. First, a bot with no rule group of its own falls back to the wildcard User-agent: * group, so a legacy blanket Disallow silently blocks every AI bot you never mentioned. Second, robots.txt must be reachable and valid: a file returning errors, or one left over from staging with Disallow: /, takes whole sites out of AI indexes with a single line.

Training crawlers collect content to train future models: GPTBot (OpenAI), ClaudeBot (Anthropic), and Google-Extended, which is a data-use token applied by Googlebot rather than a crawler you will see in logs. Blocking these keeps your future content out of training corpora and changes nothing about today's AI search answers. This group is where a licensing stance belongs, if you have one.

Search index crawlers build the retrieval indexes behind AI answers: OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude, PerplexityBot for Perplexity. Blocking these removes your pages from the source pools those engines cite, which is a direct commercial cost. User-triggered fetchers, ChatGPT-User, Claude-User and Perplexity-User, retrieve single pages when a person asks. Blocking them breaks the moment a buyer pastes your URL into a chat, and Perplexity states Perplexity-User generally ignores robots.txt anyway because a human initiated the request.

A sane default policy for a brand that wants AI visibility, written as rule groups: allow OAI-SearchBot, PerplexityBot, Claude-SearchBot, ChatGPT-User and Claude-User with Allow: /, then decide the training trio, GPTBot, ClaudeBot and Google-Extended, on your actual licensing position, blocking them only if withholding training data serves a real strategy. Most small and mid-size brands allow everything, since being known to future models is itself a visibility surface.

The asymmetry to respect: blocking Googlebot itself removes you from Google Search, AI Overviews and everything built on the search index, while blocking Google-Extended is a narrow, reversible training opt-out. Confusing those two tokens is the most expensive robots.txt mistake in the AI era.

robots.txt is a convention rather than an enforcement mechanism. Nothing at the protocol level compels compliance; the documented bots from OpenAI, Anthropic and Google state that they honor it, but scrapers wearing famous bot names do not, and Cloudflare published research in August 2025 alleging that even Perplexity reached blocked content through undeclared crawlers, which Perplexity disputed. Treat the file as a policy statement the honest actors respect.

Verification closes half the gap. Each major operator publishes the IP ranges its bots crawl from, OpenAI at openai.com/gptbot.json and its sibling files, Anthropic at claude.com/crawling/bots.json, Perplexity at perplexity.com/perplexitybot.json. Before writing policy around a heavy crawler in your logs, confirm its IPs match the operator's list. When enforcement genuinely matters, CDN-level bot management and emerging pay-per-crawl schemes are the harder backstop, since they act on requests rather than requesting cooperation.

The reverse failure is more common for brands: intending to allow AI crawlers while something upstream blocks them. Security plugins ship AI blocklists switched on by default, CDNs enable bot challenges that stop legitimate crawlers, and inherited wildcard rules override good intentions. The result is invisible in analytics and shows up only as absence from AI answers. Reachroller's free checker at /bot-access tests which AI crawlers can actually read your site, which is the fastest way to confirm your file does what you decided.