Free tool
Robots.txt generator for AI crawlers
Pick allow or block for each AI crawler, add your own disallow paths, and get a correct robots.txt to copy or download. The tool covers the nine user agents that matter for AI visibility, from GPTBot and ClaudeBot to Google-Extended and CCBot. Before you block anything, know the difference: some of these bots feed model training, others feed the live answers your buyers read.
AI crawlers
Unchecked = allowed. Blocking a search bot removes you from that engine's live answers.
robots.txt
User-agent: GPTBot Disallow: User-agent: OAI-SearchBot Disallow: User-agent: ChatGPT-User Disallow: User-agent: ClaudeBot Disallow: User-agent: Claude-User Disallow: User-agent: PerplexityBot Disallow: User-agent: Perplexity-User Disallow: User-agent: Google-Extended Disallow: User-agent: CCBot Disallow: User-agent: * Disallow:
Training bots and answer bots are separate decisions
The nine crawlers in this tool do different jobs. GPTBot collects pages for OpenAI model training. OAI-SearchBot builds the index behind ChatGPT search, and ChatGPT-User fetches a page live when someone asks about it. Anthropic splits the same way: ClaudeBot gathers training data while Claude-User fetches on a user's request. Perplexity runs PerplexityBot for its index and Perplexity-User for on-demand fetches. Google-Extended is an opt-out for Gemini training and grounding, and CCBot crawls for Common Crawl, a public dataset many labs train on.
Blocking training bots is a business judgment about whether your content should teach future models. Blocking search and user-triggered bots is a different call entirely: it removes your pages from the live answers where buyers actually see brands. A publisher with licensing leverage might block everything. A company that wants ChatGPT to recommend it should usually keep OAI-SearchBot and the user-triggered agents open.
How to deploy the file
Upload the generated file to the root of your domain so it resolves at yourdomain.com/robots.txt, served as plain text. Each subdomain needs its own file. If you already have a robots.txt with rules for Googlebot or Bingbot, merge the AI crawler blocks into it instead of replacing it, because a fresh file silently drops your existing directives.
Changes apply on the next crawl, and OpenAI says updates can take around 24 hours to propagate on its side. Confirm the result by watching user agents in your server logs, or run your domain through our free bot access checker to see what each crawler can currently reach.
Honest limits
robots.txt is a request, and compliance is voluntary. OpenAI, Google and Anthropic publish their user agents and state that they honor it. The record elsewhere is mixed: in August 2025 Cloudflare published evidence of Perplexity fetching pages from sites that had blocked it, which Perplexity disputed. If you need a guarantee rather than a request, block the user agents and published IP ranges at your firewall or CDN.
Two more caveats. Blocking GPTBot today does nothing about content already inside trained models. And robots.txt only controls access; it has zero influence on whether an engine chooses to mention or cite you once it can read your pages.
Frequently asked questions
Should I block GPTBot?+
Blocking GPTBot stops OpenAI from using your future pages for model training. It does not remove you from ChatGPT search results as long as OAI-SearchBot stays allowed. Large publishers with licensing ambitions often block it. Most businesses trying to get recommended in AI answers leave it open, since broad exposure is the point.
Will blocking Google-Extended hurt my Google rankings?+
No. Google states that Google-Extended only controls whether your content is used for Gemini training and grounding. It is separate from Googlebot, so blocking it changes nothing about crawling, indexing or ranking in Google Search, and it does not remove you from AI Overviews, which are governed by normal search indexing.
Do AI companies actually obey robots.txt?+
The major labs say they do, and server logs generally back that up for OpenAI, Google and Anthropic. Exceptions exist: Cloudflare documented Perplexity accessing sites that had blocked it in 2025. Treat robots.txt as the polite layer. For enforcement, block user agents and published IP ranges at the CDN or firewall level.
Does a correct robots.txt improve my AI visibility?+
It sets the precondition. Engines can only cite pages their bots can fetch, so an accidental block is a common own goal. Beyond access, visibility depends on whether answers actually name you. Reachroller tracks that, checking whether ChatGPT mentions your brand on real buyer questions, with a free 3-day trial from $29 a month.
More free tools and reading
Tools find the gaps. Tracking closes them.
Reachroller asks the questions your buyers ask and shows which brands ChatGPT names. Three days free, no card.
Check my brand free