{"id":152525,"date":"2026-11-06T16:21:43","date_gmt":"2026-11-06T16:21:43","guid":{"rendered":"\/ca\/tutorials\/ai-crawlers-list\/"},"modified":"2026-11-06T17:03:55","modified_gmt":"2026-11-06T17:03:55","slug":"ai-crawlers-list","status":"publish","type":"post","link":"\/ca\/tutorials\/ai-crawlers-list\/","title":{"rendered":"AI crawlers list: 8 bots to know and manage"},"content":{"rendered":"

The main AI crawlers include bots such as OAI-SearchBot, GPTBot, CCBot, Claude-SearchBot, ClaudeBot, PerplexityBot, Meta-ExternalAgent, and DuckAssistBot. <\/p>

AI crawlers are bots that AI companies use to collect web content for model training, index pages for AI search, or build open datasets that other AI developers reuse. <\/p>

Each crawler serves a different purpose, so you control them separately in robots.txt, the file that tells bots which parts of your website they can crawl. <\/p>

You can verify their visits in your server logs and choose which to allow based on AI search visibility, content protection, and server resources.<\/p>

Here is the AI crawlers list with each bot’s operator, identifier, and primary purpose:<\/p>

Crawler<\/strong><\/p><\/td>

Company\/organization<\/strong><\/p><\/td>

User-agent or robots.txt token<\/strong><\/p><\/td>

Primary purpose<\/strong><\/p><\/td><\/tr>

OAI-SearchBot<\/span><\/p><\/td>

OpenAI<\/span><\/p><\/td>

OAI-SearchBot<\/span><\/p><\/td>

AI search<\/span><\/p><\/td><\/tr>

GPTBot<\/span><\/p><\/td>

OpenAI<\/span><\/p><\/td>

GPTBot<\/span><\/p><\/td>

AI model training<\/span><\/p><\/td><\/tr>

CCBot<\/span><\/p><\/td>

Common Crawl<\/span><\/p><\/td>

CCBot<\/span><\/p><\/td>

Open web dataset collection<\/span><\/p><\/td><\/tr>

Claude-SearchBot<\/span><\/p><\/td>

Anthropic<\/span><\/p><\/td>

Claude-SearchBot<\/span><\/p><\/td>

AI search<\/span><\/p><\/td><\/tr>

ClaudeBot<\/span><\/p><\/td>

Anthropic<\/span><\/p><\/td>

ClaudeBot<\/span><\/p><\/td>

AI model development<\/span><\/p><\/td><\/tr>

PerplexityBot<\/span><\/p><\/td>

Perplexity<\/span><\/p><\/td>

PerplexityBot<\/span><\/p><\/td>

AI search<\/span><\/p><\/td><\/tr>

Meta-ExternalAgent<\/span><\/p><\/td>

Meta<\/span><\/p><\/td>

meta-externalagent<\/span><\/p><\/td>

AI model development and indexing<\/span><\/p><\/td><\/tr>

DuckAssistBot<\/span><\/p><\/td>

DuckDuckGo<\/span><\/p><\/td>

DuckAssistBot<\/span><\/p><\/td>

AI-assisted answers<\/span><\/p><\/td><\/tr><\/tbody><\/table><\/figure>

1. OAI-SearchBot – OpenAI crawler for AI search<\/h2>

OAI-SearchBot is OpenAI’s crawler for ChatGPT search. It visits web pages so OpenAI can discover content to surface in ChatGPT search results.<\/p>

In robots.txt, the crawler uses the OAI-SearchBot token.<\/p>

You can find your crawler rules at yourdomain.com\/robots.txt<\/code>. Look for OAI-SearchBot<\/code> under User-agent<\/code> to see whether your website allows or blocks it.<\/p>

\"OAI-SearchBot