What are AI crawlers?
AI crawlers are bots that fetch web content on behalf of AI systems – for training models, powering live search answers, or both. Each announces itself with a user-agent token you can allow or block in robots.txt, which makes crawler policy a strategic decision, not just a technical one.
The main crawlers
| Crawler | Operator | Primarily feeds |
|---|---|---|
| GPTBot | OpenAI | Model training |
| OAI-SearchBot | OpenAI | ChatGPT Search answers |
| ClaudeBot | Anthropic | Claude's models and tools |
| PerplexityBot | Perplexity | Live answer citations |
| Google-Extended | Gemini training (control token) | |
| CCBot | Common Crawl | Open datasets used in training |
The visibility trade-off
Blocking AI crawlers protects content from reuse – but it also removes you from the answers those systems generate. For most businesses that sell expertise, visibility is worth more than exclusivity: an assistant that cannot read you cannot cite you, and an assistant that cannot cite you sends you no AI-referred traffic. Publishers with paywalled archives may reasonably decide the other way. The point is to decide deliberately – many sites blocked everything in 2023 and forgot.
Common questions
Should I block GPTBot?
Only if protecting content from reuse matters more to you than appearing in AI answers – a system that can't read you can't cite you. For most businesses selling expertise, visibility wins; decide deliberately either way.
How do I block or allow AI crawlers?
Standard robots.txt rules per user-agent token – for example 'User-agent: GPTBot' followed by Allow or Disallow directives.
Do AI crawlers respect robots.txt?
The major operators state that they do. Smaller or newer crawlers vary, which is why some sites add firewall-level rules on top.