Which AI crawlers visit your site? (full list)
This list of AI crawlers covers the user-agent tokens you'll see in your server logs and use in robots.txt. Descriptions are based on each company's own documentation (see sources).
| OAI-SearchBot | OpenAI | Indexes pages for ChatGPT search | Yes | You don't appear in ChatGPT search answers | |
|---|---|---|---|---|---|
| ChatGPT-User | OpenAI | Visits a page when a ChatGPT user's request needs it | May not apply | Limited effect; OpenAI says robots.txt may not apply | |
| GPTBot | OpenAI | Collects content to train future models | Yes | Opts you out of training only | |
| Claude-SearchBot | Anthropic | Indexes pages to improve Claude's search results | Yes | Lower visibility in Claude's web answers | |
| Claude-User | Anthropic | Fetches a page when a Claude user asks | Yes | Claude can't open your page for users | |
| ClaudeBot | Anthropic | Collects content for model training | Yes | Opts you out of training only | |
| PerplexityBot | Perplexity | Indexes pages for Perplexity search results | Yes | You're less likely to be shown and linked | |
| Perplexity-User | Perplexity | Visits a page when a user asks | Generally no | Little effect; Perplexity says it generally ignores robots.txt | |
| Googlebot | Google Search, including AI Overviews and AI Mode | Yes | You disappear from Google, AI features included | ||
| Google-Extended | Controls Gemini's use of your content for training and grounding | Yes (a control token) | Gemini can't use your pages to train or ground answers; Search is unaffected | ||
| Bingbot | Microsoft | Bing search, which powers Copilot | Yes | You disappear from Bing and Copilot | |
| CCBot | Common Crawl | Open web archive used to train many models | Yes | Opts you out of Common Crawl |
robots.txt templates you can copy
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xmlThe simplest setup. Every search and AI crawler can read the site, including for training.
# Training crawlers: opt out
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
# Search and answer crawlers: welcome
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xmlKeeps you citable in ChatGPT, Claude, Perplexity, Google and Copilot. Add Google-Extended to the training group only if you accept that Gemini may use your pages less in its answers.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: *
Allow: /Removes you from most AI answers. Sensible for some publishers with paid content; costly for brands that want to be recommended.
Your firewall may be blocking AI bots anyway
robots.txt only works if the crawler reaches it. Many CDNs, hosts and security plugins block or rate-limit AI crawlers on their own. In July 2025 Cloudflare began blocking AI crawlers by default for new sites. When we audited our own site, requests identifying as GPTBot got a 429 error from our host while robots.txt allowed everything.
- 1Check your CDN's bot settings
On Cloudflare, look at AI Crawl Control (AI crawler blocking) and Bot Fight Mode. Allow the search and answer crawlers you want.
- 2Check your host and security plugin
Some hosts rate-limit unknown bots. Whitelist the verified AI crawlers you allow in robots.txt.
- 3Test as the bot, not as a browser
Request your pages with each crawler's user-agent and compare the response. The Website AI Audit does this for GPTBot, ClaudeBot and PerplexityBot automatically.
Common robots.txt mistakes with AI crawlers
- Blocking OAI-SearchBot along with GPTBot. You opt out of ChatGPT search, not just training. If you only want to block GPTBot, block it by name.
- Typos in user-agent names. "GPT-Bot" or "Claude-Bot" match nothing. Copy tokens exactly.
- A leftover "Disallow: /" under User-agent: *. Common after a staging site goes live; it blocks everyone.
- Relying on robots.txt for private content. It's a public request, not access control. Use logins for anything private.
- Forgetting the sitemap line. List your XML sitemap so crawlers find every page.
- Assuming llms.txt replaces robots.txt. llms.txt is an informal guide some tools read; it doesn't grant or block access.
Frequently asked questions
Should I block GPTBot?
Block GPTBot if you don't want OpenAI to use your content for training future models. It won't remove you from ChatGPT search, as long as you leave OAI-SearchBot allowed.
What's the difference between GPTBot and OAI-SearchBot?
GPTBot collects content for training OpenAI's models. OAI-SearchBot indexes pages for ChatGPT search answers. OpenAI treats the two settings independently.
Does blocking Google-Extended affect Google Search?
No. Google says Google-Extended doesn't affect inclusion or ranking in Google Search. It controls whether Gemini can use your content for training and for grounding its answers.
Do AI crawlers respect robots.txt?
The main search and training crawlers from OpenAI, Anthropic, Google and Perplexity say they follow robots.txt. User-triggered fetchers such as ChatGPT-User and Perplexity-User may not, because a person asked for the page.
How long until robots.txt changes take effect?
OpenAI says changes for ChatGPT search can take about 24 hours. Other crawlers re-read robots.txt on their own schedules, usually within a day or two.