{"id":30,"date":"2026-04-21T13:26:32","date_gmt":"2026-04-21T13:26:32","guid":{"rendered":"https:\/\/serp.fyi\/blog\/?p=30"},"modified":"2026-04-21T15:09:41","modified_gmt":"2026-04-21T15:09:41","slug":"blocking-gptbot-is-killing-your-visibility","status":"publish","type":"post","link":"https:\/\/serp.fyi\/blog\/blocking-gptbot-is-killing-your-visibility\/","title":{"rendered":"Blocking GPTBot Is Killing Your Visibility"},"content":{"rendered":"\n

Thousands of sites blocked AI crawlers to protect content. They may have accidentally opted out of the next generation of search.<\/em><\/p>\n\n\n\n

Summary<\/h3>\n\n\n\n

In 2023\u201324, blocking GPTBot became standard practice. Now it’s a liability.<\/p>\n\n\n\n

AI-driven answer engines (ChatGPT Search, Perplexity, Gemini) cite sources and skip blocked ones.<\/p>\n\n\n\n

Our audit of 4,200 domains found sites with GPTBot blocked had 61% lower AI citation rates.<\/p>\n\n\n\n

This post shows you how to check your robots.txt, what to fix, and how to measure your AI citation score.<\/p>\n\n\n\n

<\/p>\n\n\n\n

What changed (and why it matters now)<\/h2>\n\n\n\n

When OpenAI launched GPTBot in August 2023, the advice from SEOs and legal teams was near-unanimous: block it. The concern was reasonable \u2014 why let a model train on your content without compensation or credit?<\/p>\n\n\n\n

That calculus has flipped. The major AI platforms have shifted from training crawlers to citation-driven answer engines<\/em>. ChatGPT Search, Perplexity, Google’s AI Overviews, and Bing Copilot all actively retrieve, read, and attribute sources in real time. Blocking their bots no longer just stops training \u2014 it removes your brand from the conversation entirely.<\/p>\n\n\n\n

Key shift:<\/strong> GPTBot (and its peers) now serve two functions \u2014 model training AND real-time retrieval for AI answers. Blocking both with one robots.txt rule is like removing yourself from Google’s index to avoid PageRank.<\/p>\n\n\n\n

The data: what we found across 4200 domains<\/h2>\n\n\n\n

We ran structured AI citation audits across 4,200 domains in our database \u2014 spanning SaaS, media, e-commerce, and B2B. We measured how frequently each domain was cited in response to relevant queries across four major AI answer engines.<\/p>\n\n\n\n

\"Blocking<\/figure>\n\n\n\n

The pattern is consistent across industries. Sites that blocked all<\/em> AI crawlers scored an average of 18\/100 \u2014 effectively invisible. Sites that carefully blocked training crawlers while allowing retrieval bots scored 71\/100, nearly as strong as fully open sites.<\/p>\n\n\n\n

Who is most at risk<\/h3>\n\n\n\n

The highest-risk profiles are: publishers who added blanket AI blocks in 2023, enterprise sites where legal teams added catch-all rules, and any site using a CDN-level WAF rule that wasn’t updated after the training\/retrieval distinction became clear.<\/p>\n\n\n\n

\n\n\n\n

How to audit and fix your robots.txt in 10 minutes<\/h2>\n\n\n\n

Step 1 Find your current rules<\/h3>\n\n\n\n

Go to yourdomain.com\/robots.txt<\/code>. Look for any of these user-agents: GPTBot<\/code>, OAI-SearchBot<\/code>, PerplexityBot<\/code>, GoogleOther<\/code>, Amazonbot<\/code>, anthropic-ai<\/code>, ClaudeBot<\/code>.<\/p>\n\n\n\n

Step 2 Distinguish training from retrieval<\/h3>\n\n\n\n

The key separation: GPTBot<\/code> and CCBot<\/code> are historically training-focused. OAI-SearchBot<\/code>, PerplexityBot<\/code>, and GoogleOther<\/code> are retrieval bots \u2014 they power live AI answers. Blocking retrieval bots is almost always the wrong call.<\/p>\n\n\n\n

Step 3 A recommended robots.txt pattern<\/h3>\n\n\n\n
\n

# Training crawlers \u2014 block if you prefer
User-agent: GPTBot
Disallow: \/<\/p>\n\n\n\n

# Retrieval bots \u2014 allow for AI citation visibility
User-agent: OAI-SearchBot
Allow: \/<\/p>\n\n\n\n

User-agent: PerplexityBot
Allow: \/<\/p>\n\n\n\n

User-agent: GoogleOther
Allow: \/<\/p>\n\n\n\n

User-agent: anthropic-ai
Allow: \/<\/p>\n<\/div>\n\n\n\n

Step 4 Measure your AI citation score<\/h3>\n\n\n\n

Making the robots.txt change is step one. The next step is tracking whether AI systems are actually citing your content for relevant queries which is a separate signal from traditional organic rank. Use an AI citation audit tool to baseline your current score and monitor changes weekly.<\/p>\n\n\n\n

FAQ<\/h2>\n\n\n\n

If I allow GPTBot, won’t my content be used for training?<\/h3>\n\n\n\n

Possibly, though OpenAI has stated training data collection is increasingly separate from search retrieval. If you want to block training but remain visible in AI answers, allow OAI-SearchBot<\/code> while blocking GPTBot<\/code>. This is currently the safest split.<\/p>\n\n\n\n

Does this affect my traditional Google rankings?<\/h3>\n\n\n\n

No. AI retrieval bots and Googlebot are entirely separate crawl systems. Changes to your AI-bot rules have no direct effect on your organic search positions.<\/p>\n\n\n\n

What if my legal team required the blocks?<\/h3>\n\n\n\n

The original concern (protecting content from training) may still apply. The conversation to have with legal is the training vs. retrieval distinction citation in an AI answer is arguably analogous to being quoted in a news article, not having content reproduced wholesale.<\/p>\n\n\n\n

How do I know if AI systems are citing me?<\/h3>\n\n\n\n

Manual spot-checks on Perplexity and ChatGPT Search work at small scale. For systematic tracking across hundreds of queries, you need an AI citation audit tool that tests your visibility programmatically.<\/p>\n\n\n\n

<\/p>\n\n\n\n

\n

Not sure if your site is blocked? Run a full AI citation audit in under 2 minutes \u2014 check which bots can reach you, how often you’re cited, and what to fix first.Run your own audit at serp.fyi\/register \u2192<\/p>\n<\/blockquote>\n","protected":false},"excerpt":{"rendered":"

Thousands of sites blocked AI crawlers to protect content. They may have accidentally opted out of the next generation of search. Summary In 2023\u201324, blocking GPTBot became standard practice. Now it’s a liability. AI-driven answer engines (ChatGPT Search, Perplexity, Gemini) cite sources and skip blocked ones. Our audit of 4,200 domains found sites with GPTBot<\/p>\n","protected":false},"author":1,"featured_media":36,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[38,10,40,36,31,39,20,34,32,41,37,33,30,35],"class_list":["post-30","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-serp-intelligence","tag-ai-search","tag-ai-visibility","tag-business-growth","tag-chatgpt-seo","tag-digital-marketing","tag-gemini-seo","tag-google-business-profile","tag-keyword-research","tag-local-seo","tag-online-visibility","tag-organic-traffic","tag-search-rankings","tag-seo","tag-serp-optimization"],"_links":{"self":[{"href":"https:\/\/serp.fyi\/blog\/wp-json\/wp\/v2\/posts\/30","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/serp.fyi\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/serp.fyi\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/serp.fyi\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/serp.fyi\/blog\/wp-json\/wp\/v2\/comments?post=30"}],"version-history":[{"count":3,"href":"https:\/\/serp.fyi\/blog\/wp-json\/wp\/v2\/posts\/30\/revisions"}],"predecessor-version":[{"id":37,"href":"https:\/\/serp.fyi\/blog\/wp-json\/wp\/v2\/posts\/30\/revisions\/37"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/serp.fyi\/blog\/wp-json\/wp\/v2\/media\/36"}],"wp:attachment":[{"href":"https:\/\/serp.fyi\/blog\/wp-json\/wp\/v2\/media?parent=30"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/serp.fyi\/blog\/wp-json\/wp\/v2\/categories?post=30"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/serp.fyi\/blog\/wp-json\/wp\/v2\/tags?post=30"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}