OpenAI training crawler
Free tool ยท Checks every AI bot
robots.txt checker for AI bots
We check whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and 10 other AI crawlers are allowed on your site - then hand you a UK-SMB-friendly recommended robots.txt.
What we check
Every AI crawler that matters
OpenAI search index crawler
OpenAI on-demand fetch (when a user asks ChatGPT to read your page)
Anthropic training crawler
Anthropic on-demand fetch
Anthropic search index crawler
Perplexity search index crawler
Perplexity on-demand fetch