PerplexityBot

PerplexityBot is operated by Perplexity AI and documented in the company's crawler reference at docs.perplexity.ai. It feeds Perplexity's own search index, which the company has publicly described at more than 50 billion pages. Perplexity is an answer engine rather than a model vendor: it retrieves sources at question time and composes cited answers, so its index is the entire supply chain for what it can show users.

Perplexity states that PerplexityBot is not used for crawling content to train foundation models, which makes it a different kind of decision from GPTBot or ClaudeBot. Allowing PerplexityBot is closer to allowing Googlebot: it governs whether your pages can appear as linked, attributed sources in answers, rather than whether your content dissolves into a training corpus.

Perplexity also documents a second agent, Perplexity-User, with the user agent string Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user). It fetches pages when a user directly triggers a request, for example by pasting a URL. Perplexity states that because a human initiated the visit, Perplexity-User generally ignores robots.txt rules, a position that has drawn criticism from publishers and infrastructure providers.

PerplexityBot checks robots.txt and honors rules addressed to its token. To allow it explicitly, add User-agent: PerplexityBot followed by Allow: /. To block it, use User-agent: PerplexityBot followed by Disallow: /. Perplexity's own documentation recommends allowing the bot so your site can be indexed and surfaced in responses, and publishes the IP ranges it crawls from at perplexity.com/perplexitybot.json for verification.

Verification deserves extra weight with Perplexity specifically. In August 2025, Cloudflare published research alleging that Perplexity used undeclared crawlers with rotating user agents and IP ranges to reach content that had blocked its declared bots, which Perplexity disputed. Whatever position you take in that dispute, the practical lesson holds for all AI crawlers: robots.txt is a convention, user agent strings can be worn by anyone, and CDN-level bot management is the enforcement layer when access control genuinely matters.

Remember that Perplexity-User sits outside your robots.txt by Perplexity's own definition. If you need to stop user-triggered fetches as well, that requires blocking at the network level, at the cost of breaking every legitimate case where a prospective buyer asks Perplexity to read your pages.

Perplexity is the engine where index access pays off most visibly, because it cites more sources per answer than any major competitor, roughly 8.2 on average by published measurements, several times more than ChatGPT. More citation slots per answer means more chances for a well-matched page from a smaller site to be shown, linked and attributed. A page PerplexityBot cannot reach is excluded from every one of those slots.

The cost compounds because of who uses Perplexity. Its user base skews toward research-heavy queries: people comparing products, checking claims and gathering sources before decisions. Those are bottom-of-funnel moments, and every answer in your category will cite somebody. Blocking the crawler does not make the question go away, it just guarantees the sources are your competitors and the framing of your market excludes you.

Citations also carry traffic and, for some publishers, revenue. Perplexity displays its sources prominently and links out, and in 2024 it launched a Publishers' Program that shares advertising revenue with participating media partners, a signal that the company treats source relationships as commercial infrastructure rather than raw material. Referral volumes from AI engines remain small next to classic search, but they land late in the buying journey, from users who asked a specific question and clicked the source behind the answer.

Since PerplexityBot does not feed model training, the usual argument for blocking training crawlers, withholding content as a licensing position, does not apply cleanly here. The trade is citation visibility against nothing concrete in return. Publishers with paywalled archives and licensing disputes have their own calculus, but for a brand whose economics depend on discovery, blocking PerplexityBot is a pure loss.

As with every crawler decision, confirm reality matches intent. CDN bot rules and security plugins frequently block PerplexityBot by default, and the free checker at /bot-access shows which AI crawlers can actually read your site. From there, the question becomes whether indexed pages turn into citations, which only checking real Perplexity answers can tell you.