ClaudeBot

ClaudeBot is operated by Anthropic, the company behind the Claude family of AI models, and documented in Anthropic's support center article on web crawling. Its job is training data collection: gathering public web content that may be used to train and improve future Claude models. In Anthropic's wording, the crawl helps enhance the utility and safety of its generative models, and the content it collects shapes what a future Claude knows about your category from memory.

Anthropic runs a three-bot fleet that mirrors the structure OpenAI established. ClaudeBot handles training. Claude-User fetches a specific page when a Claude user asks a question that requires reading it, the live retrieval path. Claude-SearchBot navigates the web to improve search result quality, the indexing path behind Claude's web search features. Each bot has its own robots.txt token, so a site can make three independent decisions rather than one blanket call about Anthropic.

The distinction matters because the three feeds work on different timelines. Training influence surfaces in models released months later and cannot be attributed or recalled. Search indexing and user fetches are what let Claude ground answers in your actual pages today, with your current pricing and claims rather than a snapshot from the training corpus.

Anthropic documents that all three of its bots respect robots.txt directives addressed to their tokens. To block the training crawler from an entire site, add User-agent: ClaudeBot followed by Disallow: /. To block only Anthropic's search indexing or live fetches, address Claude-SearchBot or Claude-User with their own rules instead. To allow ClaudeBot explicitly, use User-agent: ClaudeBot followed by Allow: /.

Anthropic also supports the Crawl-delay extension, which is worth knowing because ClaudeBot drew complaints from site operators in 2024 for aggressive crawl volume. A Crawl-delay line under the ClaudeBot user agent rate-limits the crawler without cutting access entirely, a middle path for sites where server load rather than licensing is the concern.

One housekeeping note for older files: robots.txt rules across the web still address legacy Anthropic tokens such as anthropic-ai and claude-web, which predate the current lineup. Anthropic's documentation today centers on ClaudeBot, Claude-User and Claude-SearchBot, so new rules should target those three tokens, and an audit of an inherited robots.txt should check that a stale legacy rule is not standing in for the decision you actually want to make now.

As with every documented crawler, verify before you act on log entries. Anthropic publishes the IP addresses its crawlers operate from in a machine-readable list at claude.com/crawling/bots.json, and provides a contact address for crawl issues. A request wearing the ClaudeBot user agent from an unlisted IP is an imitator, and blocking policies written around imitators punish the wrong party.

Blocking ClaudeBot keeps your future content out of Anthropic's training data. Anthropic states that sites which disallow ClaudeBot are excluded from future training runs. What it does not do is erase your brand from Claude, because models already trained retain what they learned, and because Claude's account of your brand also comes from every third-party page that mentions you, which your robots.txt cannot reach.

The commercial calculus splits by bot. Blocking Claude-SearchBot or Claude-User cuts the retrieval paths, which means Claude cannot read your pages when a user asks about you directly or when its search feature looks for sources. Anthropic notes that blocking its search crawler may reduce a site's visibility and accuracy in results. For a brand, less accuracy is often the worse half of that trade: Claude answering from stale memory instead of your live pages is how wrong pricing and dead product names end up in front of buyers.

Claude's reach makes the stakes concrete. Anthropic reports Claude serves tens of millions of users, with particular strength among developers and technical evaluators, an audience that habitually asks assistants for tool and vendor recommendations. A brand invisible to Claude's crawlers has no way to influence how those recommendation answers describe it, while competitors who stay crawlable keep their pages in the source pool.

The pragmatic default for most brands is to allow all three Anthropic bots, then verify the configuration held, since CDN bot rules and security plugins often block AI crawlers silently. The free checker at /bot-access tests whether ClaudeBot and its siblings can actually read your site.