Thousands of sites blocked AI crawlers to protect content. They may have accidentally opted out of the next generation of search.
Summary
In 2023–24, blocking GPTBot became standard practice. Now it’s a liability.
AI-driven answer engines (ChatGPT Search, Perplexity, Gemini) cite sources and skip blocked ones.
Our audit of 4,200 domains found sites with GPTBot blocked had 61% lower AI citation rates.
This post shows you how to check your robots.txt, what to fix, and how to measure your AI citation score.
What changed (and why it matters now)
When OpenAI launched GPTBot in August 2023, the advice from SEOs and legal teams was near-unanimous: block it. The concern was reasonable — why let a model train on your content without compensation or credit?
That calculus has flipped. The major AI platforms have shifted from training crawlers to citation-driven answer engines. ChatGPT Search, Perplexity, Google’s AI Overviews, and Bing Copilot all actively retrieve, read, and attribute sources in real time. Blocking their bots no longer just stops training — it removes your brand from the conversation entirely.
Key shift: GPTBot (and its peers) now serve two functions — model training AND real-time retrieval for AI answers. Blocking both with one robots.txt rule is like removing yourself from Google’s index to avoid PageRank.
The data: what we found across 4200 domains
We ran structured AI citation audits across 4,200 domains in our database — spanning SaaS, media, e-commerce, and B2B. We measured how frequently each domain was cited in response to relevant queries across four major AI answer engines.
The pattern is consistent across industries. Sites that blocked all AI crawlers scored an average of 18/100 — effectively invisible. Sites that carefully blocked training crawlers while allowing retrieval bots scored 71/100, nearly as strong as fully open sites.
Who is most at risk
The highest-risk profiles are: publishers who added blanket AI blocks in 2023, enterprise sites where legal teams added catch-all rules, and any site using a CDN-level WAF rule that wasn’t updated after the training/retrieval distinction became clear.
How to audit and fix your robots.txt in 10 minutes
Step 1 Find your current rules
Go to yourdomain.com/robots.txt. Look for any of these user-agents: GPTBot, OAI-SearchBot, PerplexityBot, GoogleOther, Amazonbot, anthropic-ai, ClaudeBot.
Step 2 Distinguish training from retrieval
The key separation: GPTBot and CCBot are historically training-focused. OAI-SearchBot, PerplexityBot, and GoogleOther are retrieval bots — they power live AI answers. Blocking retrieval bots is almost always the wrong call.
Step 3 A recommended robots.txt pattern
# Training crawlers — block if you prefer
User-agent: GPTBot
Disallow: /
# Retrieval bots — allow for AI citation visibility
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GoogleOther
Allow: /
User-agent: anthropic-ai
Allow: /
Step 4 Measure your AI citation score
Making the robots.txt change is step one. The next step is tracking whether AI systems are actually citing your content for relevant queries which is a separate signal from traditional organic rank. Use an AI citation audit tool to baseline your current score and monitor changes weekly.
FAQ
If I allow GPTBot, won’t my content be used for training?
Possibly, though OpenAI has stated training data collection is increasingly separate from search retrieval. If you want to block training but remain visible in AI answers, allow OAI-SearchBot while blocking GPTBot. This is currently the safest split.
Does this affect my traditional Google rankings?
No. AI retrieval bots and Googlebot are entirely separate crawl systems. Changes to your AI-bot rules have no direct effect on your organic search positions.
What if my legal team required the blocks?
The original concern (protecting content from training) may still apply. The conversation to have with legal is the training vs. retrieval distinction citation in an AI answer is arguably analogous to being quoted in a news article, not having content reproduced wholesale.
How do I know if AI systems are citing me?
Manual spot-checks on Perplexity and ChatGPT Search work at small scale. For systematic tracking across hundreds of queries, you need an AI citation audit tool that tests your visibility programmatically.
Not sure if your site is blocked? Run a full AI citation audit in under 2 minutes — check which bots can reach you, how often you’re cited, and what to fix first.Run your own audit at serp.fyi/register →
