What are AI crawlers?
AI crawlers are bots that fetch web content on behalf of AI systems – for training models, powering live search answers, or both. Each announces itself with a user-agent token you can allow or block in robots.txt, which makes crawler policy a strategic decision, not just a technical one.
The main crawlers
| Crawler | Operator | Primarily feeds |
|---|---|---|
| GPTBot | OpenAI | Model training |
| OAI-SearchBot | OpenAI | ChatGPT Search answers |
| ClaudeBot | Anthropic | Claude's models and tools |
| PerplexityBot | Perplexity | Live answer citations |
| Google-Extended | Gemini training (control token) | |
| CCBot | Common Crawl | Open datasets used in training |
The visibility trade-off
Blocking AI crawlers protects content from reuse – but it also removes you from the answers those systems generate. For most businesses that sell expertise, visibility is worth more than exclusivity: an assistant that cannot read you cannot cite you, and an assistant that cannot cite you sends you no AI-referred traffic. Publishers with paywalled archives may reasonably decide the other way. The point is to decide deliberately – many sites blocked everything in 2023 and forgot.