Most robots.txt files in 2026 have at least one AI crawler rule that is quietly costing the site visibility. The problem is not bad intent. It is that the people configuring bot policy and the people responsible for AI search visibility rarely sit in the same meeting.

A security team blocks GPTBot to stop training crawls. In the process, the retrieval bot gets blocked, too. A few months later, the site disappears from ChatGPT answers, and no one traces it back to that robots.txt change.

This piece is the reference for every AI crawler that matters, what each one does, and the AI crawler robots. txt decisions that affect whether your brand appears in AI search.

The Three Classes Of AI Crawler

AI crawlers split into training (feeds model weights on a schedule), retrieval (builds the live index AI search features query at answer time), and user-triggered (fetches a page because a person asked the assistant to read it right now). 

The distinction matters because blocking a training crawler is a low-cost business decision, while blocking a retrieval crawler removes you from that product's cited answers immediately. Most crawl volume sits in training. Most visibility impact sits in retrieval. That mismatch is why blanket blocking decisions go wrong.

Every AI Crawler That Matters In 2026

The table below covers the crawlers with the most visibility impact. Each operator runs multiple bots with different purposes, and confusing them is the single most common AI crawler mistake.

OpenAI: Three Bots, Three Purposes

The user agent GPTBot is training-only. It collects content that may end up in model weights but does not feed ChatGPT's search feature. OAI-SearchBot is the retrieval crawler that produces ChatGPT search citations. ChatGPT-User handles live page reads when a person pastes a URL into a conversation. Blocking GPTBot while allowing the other two is a common and defensible configuration. Blocking all three because they share an operator is where visibility disappears.

Anthropic: The Same Split Applies

ClaudeBot is training. Claude-SearchBot is a retrieval. Claude-User is user-triggered. Claude over-indexes on research and comparison queries, so blocking Claude-SearchBot has an outsized impact for B2B brands whose buyers use Claude for vendor evaluation.

Google: One Real Crawler, One Control Token

Googlebot is the real crawler. It gates eligibility for Google Search, AI Overviews, and AI Mode. Google-Extended is not a fetching crawler. It is a robots.txt control token only. Disallowing Google-Extended opts your content out of Gemini training and grounding, but does not affect Search ranking, AI Overviews, or AI Mode. Those are governed entirely by Googlebot access.

Perplexity: The Robots.txt Exception

PerplexityBot is the retrieval crawler for Perplexity Search. Perplexity-User handles user-triggered fetches and, notably, generally ignores robots.txt. Perplexity documents this on the grounds that the fetch is user-initiated. If you want to block Perplexity-User traffic, a WAF rule is the only reliable method.

Training-Only Crawlers: CCBot, Bytespider, And Others

CCBot feeds Common Crawl, the open dataset many downstream models train on. Blocking it has no immediate visibility cost but affects multiple models indirectly over time. Bytespider (ByteDance) is frequently blocked for aggressive crawl behaviour. Meta-ExternalAgent and Applebot-Extended are training control tokens with low blocking cost.

How To Configure Robots.txt For AI Search Visibility

The principle: allow every retrieval and user-triggered bot, then decide about training separately.

A Default Configuration That Preserves Citations

Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, ChatGPT-User, Claude-User, and Perplexity-User. These produce citations and serve live page reads.

For training crawlers (the GPTBot crawler, ClaudeBot, Google-Extended, CCBot, Bytespider, Meta-ExternalAgent), the decision is a business one. Blocking them has no immediate visibility cost but means the model's knowledge of your brand decays over successive training runs. Allowing them means your content trains the models that may cite you in the future.

The most common mistake is a blanket rule. A "block unknown bots" WAF policy written before AI search existed will catch every bot introduced since, including OAI-SearchBot. A rate limit tuned for browser traffic returns 429 responses that crawlers read as unavailability. Both silently degrade AI search visibility without triggering any alert. 

How To Verify What Is Actually Reaching Your Server

Robots.txt describes intent. Your WAF describes reality. When they disagree, the WAF wins.

Three checks, in order. First, filter server logs on the user agent tokens in the table above. A retrieval bot with zero hits in seven days is being stopped upstream. Second, check response codes: a bot receiving 403 or 429 is being blocked in a way that request-count charts report as traffic. Third, verify bot identity against published IP ranges. OpenAI, Anthropic, and Perplexity all publish JSON files of their crawler IPs. User agent strings are trivially spoofed, so verify before building an allowlist.

Crawler access is the foundation. Once the bots can reach you, the question is whether the pages they reach are structured for extraction. See how Passionfruit's GEO service builds citation-ready content on top of solid SEO foundations, and talk to the team about auditing your crawler access alongside your AI visibility.

Frequently Asked Questions

If I Block GPTBot, Do I Still Appear In ChatGPT Answers?

Yes, provided OAI-SearchBot and ChatGPT-User remain allowed. GPTBot is training-only. ChatGPT search uses OAI-SearchBot for retrieval and ChatGPT-User for live reads. Blocking GPTBot while allowing the other two is a standard configuration that preserves visibility.

What Is The Difference Between Google-Extended And Googlebot?

Googlebot fetches pages. Google-Extended is a robots.txt control token with no fetching behaviour. Disallowing Google-Extended opts out of Gemini training and grounding. It does not affect Search ranking, AI Overviews, or AI Mode. Those are governed by Googlebot access only.

Which AI Crawlers Ignore Robots.txt, And How Do I Confirm From Logs?

Perplexity-User is the most documented case. To confirm, filter logs by user agent, check whether requested paths fall under your Disallow rules, and count violations. Any bot consistently hitting disallowed paths is ignoring robots.txt. For Perplexity-User, a WAF rule is the only reliable block.

Does Blocking CCBot Remove Content From Common Crawl Retroactively?

No. Blocking CCBot prevents future snapshots from including your content. Content already captured remains in the dataset and in models trained on it. The effect is gradual: your representation fades over successive training cycles as newer snapshots exclude you.

Should I Allow Training Crawlers For Maximum AI Visibility?

There is no definitive answer yet. Retrieval access (OAI-SearchBot, Claude-SearchBot, PerplexityBot) drives the citations you can measure today. Training access may strengthen parametric knowledge long-term. Allowing both covers both paths. Blocking training while allowing retrieval is the minimum viable configuration.