GPTBot

GPTBot is operated by OpenAI and documented in the company's crawler reference at developers.openai.com. Its stated purpose is to crawl content that may be used in training OpenAI's generative AI foundation models, which means the pages it collects shape what future GPT models know about your category, your competitors and you. It launched in August 2023 as one of the first publicly documented AI training crawlers, and it quickly became the reference case for the block-or-allow debate.

The critical fact about GPTBot is what it does not feed. OpenAI operates three distinct bots: GPTBot for training data, OAI-SearchBot for the index behind ChatGPT search, and ChatGPT-User for live page fetches during conversations. GPTBot's crawl has no role in retrieving or citing your pages when ChatGPT answers a question today. Its influence arrives later and diffusely, as baked-in knowledge inside models trained on the corpus it gathered.

That timeline matters for strategy. Training data moves on retraining schedules you cannot see, and content absorbed now surfaces months or years later, blended beyond attribution. A Botify analysis found OpenAI roughly tripled its overall web crawl since August 2025, so GPTBot's reach is expanding alongside the rest of the fleet, but its job has stayed the same: reading the web so future models arrive already knowing it.

GPTBot checks the robots.txt file at the root of your domain before crawling and honors rules addressed to its token. To block it from the whole site, add User-agent: GPTBot followed by Disallow: /. To block only a section, scope the rule, for example Disallow: /drafts/. To allow it explicitly while other rules exist, use User-agent: GPTBot followed by Allow: /.

Because rules are per token, the common publisher configuration is split: block GPTBot as a licensing position while allowing OAI-SearchBot, which keeps pages citable in ChatGPT search answers while staying out of future training runs. The reverse split makes little commercial sense, since it contributes training data while refusing the surface that returns attribution and clicks.

Two timing details keep expectations realistic. The opt-out is forward-looking: blocking GPTBot governs future crawls, while content gathered in earlier crawls sits in datasets already assembled, and models already trained keep what they learned. And the version number in the user agent string moves over time, from GPTBot/1.0 at launch to GPTBot/1.4 in the current documentation, so log filters and firewall rules should match the GPTBot token rather than a full frozen string that will silently stop matching after the next revision.

User agent strings can be spoofed by scrapers borrowing GPTBot's name, so authenticate before making policy around log entries. OpenAI publishes the IP ranges GPTBot crawls from at openai.com/gptbot.json, and a reverse lookup against that list separates the real crawler from imitators. If you need enforcement rather than convention, pair the robots.txt rule with CDN-level bot management, since robots.txt compliance is voluntary at the protocol level.

Blocking GPTBot keeps your future content out of OpenAI's training corpus. It does not remove your brand from ChatGPT, and this is the most misunderstood point in the entire AI crawler debate. ChatGPT search answers are grounded through retrieval, which uses the index built by OAI-SearchBot and live fetches by ChatGPT-User. A site can block GPTBot and remain fully retrievable and citable in ChatGPT search.

Your robots.txt also has no reach over what others publish. If reviewers, directories, forums and news sites describe your brand, models train on those descriptions regardless of your own crawler policy. Blocking GPTBot controls one input, your own pages, and leaves the rest of the web's account of you untouched, which is an argument for making sure the third-party record is accurate before worrying about training opt-outs.

The real cost of blocking is subtler and long term. Future models' built-in knowledge is itself a visibility surface: when a model answers from memory rather than live retrieval, brands present in the training corpus have an edge. A brand absent from training data is betting its entire AI presence on retrieval. For large publishers with licensable archives, blocking GPTBot is leverage in content negotiations. For most small and mid-size brands, discoverability is worth more than the licensing position, and allowing GPTBot is the default that fits their incentives.

Whichever way you decide, verify the decision actually took effect. Security plugins and CDN defaults sometimes block GPTBot without the site owner knowing, and the free checker at /bot-access tests which AI crawlers can read your site right now.