Why Your AI Crawlers Can See Your Site But Still Won't Cite You: The robots.txt and llms.txt Configuration Guide
AI crawlers can access your site but still won't cite you. Here is how to configure robots.txt for GPT, Claude, and Perplexity Bot, and what llms.txt does.
In this article
Your robots.txt says "Allow." Your pages are indexed. GPTBot, ClaudeBot, and PerplexityBot can all reach your content. But when someone asks ChatGPT or Perplexity about your category, your brand does not appear. The assumption most teams make is that crawler access equals citation eligibility. It does not.
Access is the first gate, not the only one. This guide covers what each file actually controls, which bots to allow, which to block, and why fixing your robots.txt is necessary but rarely sufficient.
What Robots.txt Controls (and What It Does Not)
robots.txt is an access control file. It tells crawlers which pages they can and cannot fetch. It is not a ranking signal, a citation signal, or a trust signal. A perfectly configured robots.txt for AI crawlers opens the door. It does not make the room worth entering.
The Two Types of AI Crawlers
Every major AI company now runs at least two crawlers with different purposes. This distinction is the foundation of a sensible configuration.
| Company | Training crawler (block if you want) | Search/retrieval crawler (allow for citations) |
|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot, ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot, Claude-User |
| Google-Extended | Googlebot (also powers AI Overviews) | |
| Perplexity | PerplexityBot (dual-purpose) | PerplexityBot |
| Meta | Meta-ExternalAgent | (no search product) |
| ByteDance | Bytespider | (no search product) |
The split is supported explicitly by OpenAI and Anthropic. You can opt out of training while remaining eligible for live citations. This has become the fastest-growing configuration pattern across Cloudflare's network.
The Recommended Configuration
For a site that wants maximum AI search visibility while opting out of model training:
# Search and retrieval crawlers (ALLOW for citations)
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Googlebot
Allow: /
# Training crawlers (BLOCK if you want to opt out)
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: Bytespider
Disallow: /This configuration signals: do not use our content to train future models, but do retrieve and cite us when users ask live questions.
What llms.txt Does (and Does Not Do)
llms.txt is a newer file that tells AI systems which pages on your site are most important and how your content is organized. It sits alongside robots.txt but serves a different function.
The Honest Position on llms.txt
Four independent studies (Ahrefs, SE Ranking, Trakkr, and Google's own June 2026 guidance) found llms.txt has no measurable effect on AI citations. Google's guidance on llms.txt is clear: there is no confirmed citation benefit.
The difference between the two files:
Swipe the table to see every column →
| File | What it does | Effect on citations | Priority |
|---|---|---|---|
| robots.txt | Controls which crawlers can access which pages | Direct: blocked crawlers cannot cite you | Fix first |
| llms.txt | Guides AI systems on content hierarchy and structure | Unproven: no confirmed citation lift | Ship last, if convenient |
If you have time for one file, fix robots.txt. If you have time for both, ship llms.txt as a five-minute addition after robots.txt is correct.
Why Fixing Crawler Access Does Not Fix Citation
This is the part most configuration guides skip. You can have perfect robots.txt and still get zero citations. Here is why.
The Three Layers Beyond Access
Layer 1: Renderability. AI crawler access does not execute JavaScript. An analysis of over 500 million GPTBot fetches found no evidence of JavaScript rendering. If your page content requires JavaScript to display (common in React, Angular, and Vue applications), the crawler sees an empty shell. Run curl on your own page and check the raw HTML before trusting any robots.txt fix.
Layer 2: Content quality and structure. A crawler that can access your page still needs a reason to cite it. Front-loaded answers, clear headings, self-contained FAQ sections, and structured data all increase the probability of citation. Crawler access without citation-worthy content produces indexation without visibility.
Layer 3: Entity authority. AI engines cross-reference your page against other sources. If your brand is mentioned on review platforms, cited in industry publications, and discussed in community forums, the engine has multiple signals confirming your authority. If the only source is your own domain, access alone does not build the trust needed for citation.
This is where generative engine optimization picks up where technical configuration stops: building the content quality, entity signals, and external authority that turn crawler access into actual citations.
The Configuration Audit Checklist
Run this checklist before assuming your crawler configuration is correct.
- [ ] robots.txt explicitly allows OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, and Googlebot
- [ ] CDN and WAF bot rules do not override robots.txt (Cloudflare, Akamai, and AWS WAF can block bots independently)
- [ ] Server logs confirm AI crawlers are actually fetching pages (intent is not the same as access)
- [ ] Key pages render meaningful content without JavaScript (curl test)
- [ ] No accidental noindex tags on pages you want cited
- [ ] Subdomains have their own robots.txt (decisions are per-host)
- [ ] XML sitemap is current and references all pages you want crawled
For a deeper look at how GPTBot works and whether to block it, see the detailed breakdown.
Access Is the Floor, Not the Ceiling
Fixing robots.txt is a 30-minute task that removes a hard blocker. If AI crawlers cannot reach your pages, nothing else matters. But once access is confirmed, the work that earns citations is content structure, entity authority, schema markup, and external validation. Configuration opens the door. Everything after that determines whether AI engines walk through it.
See how Passionfruit's GEO service runs crawler audits as the first step of every engagement, then builds the content and authority layer that turns access into citations. Talk to an Expert to start with a technical diagnostic.
Frequently asked questions
How do I configure robots.txt for ChatGPT and Perplexity crawlers?
Allow OAI-SearchBot and ChatGPT-User (for ChatGPT citations) and PerplexityBot (for Perplexity citations). Block GPTBot if you want to opt out of OpenAI model training. The search crawlers power live citations; the training crawlers build future models.
What is the difference between llms.txt and robots.txt?
robots.txt controls crawler access: which bots can fetch which pages. llms.txt guides AI systems on content hierarchy and importance. robots.txt has a direct effect on citations (blocked crawlers cannot cite you). llms.txt has no confirmed citation benefit as of 2026.
Should I block or allow GPTBot, ClaudeBot, and PerplexityBot?
Allow the search and retrieval variants (OAI-SearchBot, Claude-SearchBot, PerplexityBot) for citation eligibility. Block the training variants (GPTBot, ClaudeBot) if you want to opt out of model training. This split is the most common configuration in 2026.
Why is my site still not cited after fixing robots.txt?
Crawler access is necessary but not sufficient. AI engines also need JavaScript-free content rendering, structured and front-loaded answers, schema markup, and external authority signals before citing a page. Check server logs to confirm crawlers are actually fetching your pages.
Does llms.txt help with AI search visibility?
No confirmed benefit. Four independent studies found no measurable citation lift from llms.txt. Ship it if convenient, but prioritize robots.txt, content structure, and schema markup first.
