Free tool · Checks every AI bot
robots.txt checker for AI bots
We check whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and 10 other AI crawlers are allowed on your site - then hand you a UK-SMB-friendly recommended robots.txt.
Analysis
Found a robots.txt at
Found a robots.txt at https://kawakawabalm.co.nz/robots.txt
| AI crawler | Status | What it does |
|---|---|---|
| GPTBot | ✓ Allowed | OpenAI training crawler |
| OAI-SearchBot | ✓ Allowed | OpenAI search index crawler |
| ChatGPT-User | ✓ Allowed | OpenAI on-demand fetch (when a user asks ChatGPT to read your page) |
| ClaudeBot | ✓ Allowed | Anthropic training crawler |
| Claude-User | ✓ Allowed | Anthropic on-demand fetch |
| Claude-SearchBot | ✓ Allowed | Anthropic search index crawler |
| PerplexityBot | ✓ Allowed | Perplexity search index crawler |
| Perplexity-User | ✓ Allowed | Perplexity on-demand fetch |
| Google-Extended | ✓ Allowed | Google AI Overviews / Gemini training opt-in token |
| CCBot | ✓ Allowed | Common Crawl (used by many AI training pipelines) |
| Applebot | ✓ Allowed | Apple search index + Siri live generative answers (opt out of Siri answers via <meta name="nosnippet">) |
| Applebot-Extended | ✓ Allowed | Apple Intelligence AI-training opt-out token (controls whether Applebot-crawled content trains Apple Intelligence) |
| Amazonbot | ✓ Allowed | Amazon AI training crawler |
| Google-NotebookLM | ✓ Allowed | Google NotebookLM live citation fetches |
| Google-CloudVertexBot | ✓ Allowed | Google Cloud Vertex AI grounding |
| Google-Agent | ✓ Allowed | Google Gemini agent runs |
| Bytespider | ✓ Allowed | ByteDance / Doubao training crawler |
| Meta-ExternalAgent | ✓ Allowed | Meta AI training crawler |
| DuckAssistBot | ✓ Allowed | DuckDuckGo AI assistant |
Sitemap declarations found:
- https://kawakawabalm.co.nz/wp-sitemap.xml
Recommended
Drop-in robots.txt
UK-SMB-friendly defaults: every AI crawler explicitly allowed by name; admin / cart / checkout paths blocked. Tweak to taste.
# robots.txt - generated by llmsubmitter.com # UK-SMB-friendly defaults: AI crawlers explicitly allowed by name, # admin / transactional paths blocked. Edit to taste. User-agent: * Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: GPTBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: OAI-SearchBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: ChatGPT-User Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: ClaudeBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Claude-User Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Claude-SearchBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: PerplexityBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Perplexity-User Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Google-Extended Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: CCBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Applebot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Applebot-Extended Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Amazonbot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Google-NotebookLM Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Google-CloudVertexBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Google-Agent Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Bytespider Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Meta-ExternalAgent Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: DuckAssistBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= Sitemap: https://kawakawabalm.co.nz/sitemap.xml # llms.txt: https://kawakawabalm.co.nz/llms.txt
Your current robots.txt
For reference
# robots.txt for https://kawakawabalm.co.nz/ User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php # WordPress system paths Disallow: /cgi-bin/ Disallow: /trackback/ Disallow: /feed/ Disallow: /comments/ # Internal search Disallow: /?s= Disallow: /search/ Disallow: /*?s=* # WooCommerce Disallow: /cart/ Disallow: /checkout/ Disallow: /my-account/ Disallow: /add-to-cart= Disallow: /?add-to-cart= # Parameter URLs Disallow: /*?orderby= Disallow: /*?filter_ Disallow: /*utm_* # Archive pages Disallow: /archive/ ################################################## # Bing / Microsoft AI Crawlers ################################################## User-agent: Bingbot Allow: / User-agent: BingPreview Allow: / User-agent: CCBot Allow: / ################################################## # Google Search ################################################## User-agent: Googlebot Allow: / ################################################## # Google Images ################################################## User-agent: Googlebot-Image Allow: / ################################################## # Google Search Console URL Inspection ################################################## User-agent: Google-InspectionTool Allow: / ################################################## # Google Ads ################################################## User-agent: AdsBot-Google Allow: / User-agent: AdsBot-Google-Mobile Allow: / ################################################## # Google Merchant Center / Google Shopping ################################################## User-agent: Storebot-Google Allow: / ################################################## # Google rendering resources ################################################## User-agent: Googlebot Allow: /wp-content/uploads/ Allow: /wp-content/plugins/ Allow: /wp-content/themes/ Allow: /wp-includes/ Allow: /wp-includes/js/ Allow: /wp-includes/css/ ################################################## # Image crawling ################################################## User-agent: Googlebot-Image Allow: /wp-content/uploads/ ################################################## # AI Crawlers ################################################## User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-Web Allow: / User-agent: PerplexityBot Allow: / User-agent: Amazonbot Allow: / User-agent: Meta-ExternalAgent Allow: / ################################################## # Block unnecessary crawlers ################################################## User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: omgili Disallow: / User-agent: Baiduspider Disallow: / User-agent: Yandex Disallow: / User-agent: PetalBot Disallow: / ################################################## # Sitemap ################################################## Sitemap: https://kawakawabalm.co.nz/wp-sitemap.xml