Robots.txt for AI: which bots to allow, block, and watch

Robots.txt was designed in 1994 for one kind of visitor, the search spider, and it now referees three. Training crawlers collect pages that become model weights: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Applebot-Extended. Search crawlers build the retrieval indexes that AI answers draw from: OAI-SearchBot, Claude-SearchBot, PerplexityBot, plus the classic Googlebot and Bingbot whose indexes several engines piggyback on. User-triggered agents fetch a single page in real time because a human asked about it: ChatGPT-User, Claude-User, Perplexity-User.

Every entry in your robots.txt is therefore a statement about one of three different bargains. A training rule trades future model knowledge against content protection. A search rule trades citation eligibility against crawl load. A user-agent rule trades live usability of your pages inside assistants against nothing much at all, which is why blocking that tier is almost always a mistake. The strategic question of whether to block anything is a business-model call we argue through in should you block AI crawlers; this playbook is the implementation layer.

A quick mechanical refresher, since AI rules fail on syntax more than strategy. Rules group under a User-agent line and apply to the most specific matching agent. A bare Disallow with no value allows everything. An Allow directive can carve exceptions from a broader Disallow. And the file must live at the root of each host, https version included, or the bots you are addressing will never see it.

Specificity is the trap worth restating: a bot obeys the rule group for its own name if one exists, otherwise the wildcard group. That means a site with a permissive wildcard and one forgotten AI-specific disallow behaves opposite to what the owner remembers, and a site that added a blanket AI block in 2024 may be silently refusing search bots that did not exist when the rule was written. Date your rules in comments. Files that explain themselves survive team turnover; files that do not become archaeology.

The allow tier earns its place through a single mechanism: retrieval. When ChatGPT search, Perplexity or Claude answers a current question, it fetches candidate pages and composes from what it gets. OAI-SearchBot builds and refreshes OpenAI's index of citable pages. PerplexityBot does the same for an engine whose entire interface is citations. Claude-SearchBot, which Anthropic split from ClaudeBot in 2025 precisely so sites could allow search without permitting training, covers Claude's web answers. Block any of these and you are not expressing a preference; you are removing yourself from that engine's pool of quotable evidence.

The crawl cost of this tier is modest and worth paying. Search crawlers fetch on index schedules rather than training-scale sweeps, and the pages they want most, comparison pages, pricing, docs, FAQs, are exactly the pages you want quoted. If bandwidth genuinely hurts, rate limiting beats blocking: a slower yes preserves citations that a fast no forfeits.

The user-triggered agents belong in the same tier for a more visceral reason: they act on behalf of a specific human in the middle of a conversation about you. When a prospect pastes your pricing page into ChatGPT or asks Perplexity to compare your plans against a rival's, ChatGPT-User or Perplexity-User makes the fetch. A 403 at that moment is a sales call that hangs up on the buyer. These agents also ignore cached indexes and read the live page, which makes them the channel through which your freshest content reaches answers first.

Keep Googlebot and Bingbot mentally in this tier even though they predate the AI era. Google's AI Overviews and AI Mode cite from Googlebot's index, and Bing's index feeds Copilot along with parts of other engines' retrieval. The supply chain from these indexes into composed answers is mapped in how AI picks its sources.

The training tier is where reasonable sites genuinely diverge, and your robots.txt should record a decision rather than a default. GPTBot, ClaudeBot, Google-Extended, CCBot and Applebot-Extended collect content that shapes what future models know. Blocking them protects your archive from becoming free training material, which is why more than 2.5 million sites disallow AI training. Allowing them keeps your brand's own words in the corpus that models answer from when they respond from memory rather than live retrieval.

For publishers whose content is the product, blocking this tier is rational and increasingly comes paired with licensing leverage or pay-per-crawl pricing, the economics covered in our Cloudflare Pay Per Crawl guide. For brands, the calculus tilts the other way: models that never read your site still learn about you from reviews, forums and competitors, so opting out mostly removes the one source you control. Either choice is defensible; making it accidentally is what this section exists to prevent.

Two entries deserve special notes. CCBot feeds Common Crawl, an open dataset reused across many labs' training runs, so one rule there has unusually wide reach. Google-Extended is an opt-out token rather than a crawler: Googlebot still fetches your pages either way, and the token only tells Google not to use them for Gemini training and grounding, leaving rankings and AI Overviews untouched.

The watch list holds everything the file cannot fully govern. First, bots with documented compliance problems: Cloudflare's August 2025 report found undeclared crawling from Perplexity reaching sites that had disallowed it, and Bytespider's reputation for ignoring rules predates that. State your policy in robots.txt anyway, because the file is what licensing arguments and legal claims point at, then enforce with bot management or a WAF.

Second, mixed-use crawlers, the ones that train, index and browse under a single user agent, giving you no way to allow one job and refuse another. Cloudflare's September 15, 2026 policy blocks these by default on ad-carrying pages precisely to force the split, so expect new, more specific user agents to appear through late 2026 and check vendor docs each quarter.

Third, agentic browsers, assistants that operate a real browser session to complete tasks. Their requests often look human, may not honor robots.txt, and blur the line between a bot and a user's deputy. No stable governance standard exists for them yet as of August 2026. The practical posture: watch your logs, keep the tier-one allows in place so legitimate agents get clean paths, and let enforcement tools handle impostors.

The template below implements the brand-side policy: answer engines allowed explicitly, training bots recorded as a deliberate opt-out. Flip the training section to Allow if you make the opposite call; the structure is the point. Explicit allows also protect you when a broader wildcard rule exists elsewhere in the file.

# Answer engines: allow (these decide AI citations)
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

# Training bots: your deliberate choice (example: opted out)
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# Never add rules against Googlebot or Bingbot here

Three deployment checks: confirm the file serves at the root with a 200 status, test agent-specific behavior with a fetch tool rather than trusting your eyes, and diff the file in version control so a rule change is never a mystery six months later. If a page must never appear anywhere, remember robots.txt is a visibility tool with an ironic edge: disallowed URLs can still be indexed from external links. Access control belongs at the server, robots.txt just directs traffic.

The publisher variant inverts the middle tier and often scopes by path instead of by bot. A news site might disallow every AI agent across the archive while carving out Allow rules for its about pages, subscription landing pages and product announcements, keeping the commercial surface citable while the journalism stays walled or priced. Path-scoped rules per user agent are fully supported syntax and underused: most sites treat the AI decision as all or nothing when the file happily expresses these sections yes, those sections no. Whichever variant you deploy, keep one principle fixed: the file should read as a policy someone could defend in a meeting, with each rule traceable to a reason, because within a year some rule in it will be blamed for either a traffic problem or a licensing argument, and the defensible file wins both conversations.

Robots.txt states intent; it stops nothing by itself. For the compliant majority of named bots that is enough, and for everything else you have three escalation layers. The first is verification: major vendors publish IP ranges and support reverse DNS checks for their crawlers, so a request claiming to be OAI-SearchBot from an unlisted network is an impostor you can refuse without any policy debate. User-agent strings are free to fake; network identity is not.

The second layer is managed bot control. Cloudflare, and equivalents at other CDNs, classify crawler traffic behaviorally and let you allow, block, challenge or charge by category, which is how a disallow rule becomes an actual closed door. This is also where the economics now live: pay-per-crawl pricing turns the block into a meter for sites whose content justifies it. The third layer is rate limiting and path protection for the bots you allow, because an allowed crawler that hammers your infrastructure or wanders into faceted search URLs is a cost problem even when it is a visibility asset. Allow does not have to mean unlimited.

Close the loop with your logs, monthly. Three questions cover it: which AI bots are actually visiting, do their paths match your rules, and are the answer-engine bots reaching the pages you most want cited? A sitemap-listed comparison page that OAI-SearchBot has never fetched is a finding, and usually a fixable one: internal links, crawl priority or an accidental noindex are the usual suspects. Log analysis tells you what the bots consumed; it cannot tell you what the answers did with it, which is the gap the final section addresses.

You will meet the suggestion that a modern AI policy needs llms.txt, a proposed markdown manifest of your best content for language models. The evidence says treat it as optional. As of Q1 2026, no major AI company reads llms.txt in production: OpenAI, Google, Anthropic, Meta and Mistral have made no commitment to it, and Google's search team has dismissed it publicly, with John Mueller comparing it to the keywords meta tag. Ahrefs analyzed 137,000 sites and found 97 percent of llms.txt files were never fetched at all, with zero measurable effect on citations.

The file is cheap to ship and some developer-tool companies publish it for coding agents that do read documentation manifests, so cargo-culting it does no harm. Just never let it substitute for the file that bots actually obey. Robots.txt governs access; llms.txt is, for now, a memo nobody is required to read. The full story, including where the proposal came from and the narrow cases where it earns its keep, is in our llms.txt guide.

The same skepticism discipline applies in reverse to robots.txt itself: it controls the input side only. Whether the engines still cite you after a rules change is an empirical question about outputs. Reachroller closes that loop by asking your buyers' questions across engines on a schedule and recording, with stored raw answers, whether your brand gets named. Change the file, watch the mention rate, and you have a feedback loop instead of a folk belief.