Free tool · Checks every AI bot
robots.txt checker for AI bots
We check whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and 10 other AI crawlers are allowed on your site - then hand you a UK-SMB-friendly recommended robots.txt.
Analysis
Found a robots.txt at
Found a robots.txt at https://oora-home.com/robots.txt
Issues:
- No "Sitemap:" line in robots.txt - AI crawlers can't discover your sitemap from here.
| AI crawler | Status | What it does |
|---|---|---|
| GPTBot | ✓ Allowed | OpenAI training crawler |
| OAI-SearchBot | ✓ Allowed | OpenAI search index crawler |
| ChatGPT-User | ✓ Allowed | OpenAI on-demand fetch (when a user asks ChatGPT to read your page) |
| ClaudeBot | ✓ Allowed | Anthropic training crawler |
| Claude-User | ✓ Allowed | Anthropic on-demand fetch |
| Claude-SearchBot | ✓ Allowed | Anthropic search index crawler |
| PerplexityBot | ✓ Allowed | Perplexity search index crawler |
| Perplexity-User | ✓ Allowed | Perplexity on-demand fetch |
| Google-Extended | ✓ Allowed | Google AI Overviews / Gemini training opt-in token |
| CCBot | ✓ Allowed | Common Crawl (used by many AI training pipelines) |
| Applebot | ✓ Allowed | Apple search index + Siri live generative answers (opt out of Siri answers via <meta name="nosnippet">) |
| Applebot-Extended | ✓ Allowed | Apple Intelligence AI-training opt-out token (controls whether Applebot-crawled content trains Apple Intelligence) |
| Amazonbot | ✓ Allowed | Amazon AI training crawler |
| Google-NotebookLM | ✓ Allowed | Google NotebookLM live citation fetches |
| Google-CloudVertexBot | ✓ Allowed | Google Cloud Vertex AI grounding |
| Google-Agent | ✓ Allowed | Google Gemini agent runs |
| Bytespider | ✓ Allowed | ByteDance / Doubao training crawler |
| Meta-ExternalAgent | ✓ Allowed | Meta AI training crawler |
| DuckAssistBot | ✓ Allowed | DuckDuckGo AI assistant |
Recommended
Drop-in robots.txt
UK-SMB-friendly defaults: every AI crawler explicitly allowed by name; admin / cart / checkout paths blocked. Tweak to taste.
# robots.txt - generated by llmsubmitter.com # UK-SMB-friendly defaults: AI crawlers explicitly allowed by name, # admin / transactional paths blocked. Edit to taste. User-agent: * Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: GPTBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: OAI-SearchBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: ChatGPT-User Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: ClaudeBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Claude-User Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Claude-SearchBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: PerplexityBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Perplexity-User Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Google-Extended Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: CCBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Applebot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Applebot-Extended Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Amazonbot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Google-NotebookLM Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Google-CloudVertexBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Google-Agent Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Bytespider Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Meta-ExternalAgent Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: DuckAssistBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= Sitemap: https://oora-home.com/sitemap.xml # llms.txt: https://oora-home.com/llms.txt
Your current robots.txt
For reference
# robots.txt automatically generated by PrestaShop e-commerce open-source solution # https://www.prestashop.com - https://www.prestashop.com/forums # This file is to prevent the crawling and indexing of certain parts # of your site by web crawlers and spiders run by sites like Yahoo! # and Google. By telling these "robots" where not to go on your site, # you save bandwidth and server resources. # For more information about the robots.txt standard, see: # https://www.robotstxt.org/robotstxt.html User-agent: * # Allow Directives Allow: */modules/*.css Allow: */modules/*.js Allow: */modules/*.png Allow: */modules/*.jpg Allow: */modules/*.gif Allow: */modules/*.svg Allow: */modules/*.webp Allow: /js/jquery/* # Private pages Disallow: /*?order= Disallow: /*?tag= Disallow: /*?id_currency= Disallow: /*?search_query= Disallow: /*?back= Disallow: /*?n= Disallow: /*&order= Disallow: /*&tag= Disallow: /*&id_currency= Disallow: /*&search_query= Disallow: /*&back= Disallow: /*&n= Disallow: /*controller=addresses Disallow: /*controller=address Disallow: /*controller=authentication Disallow: /*controller=cart Disallow: /*controller=discount Disallow: /*controller=footer Disallow: /*controller=get-file Disallow: /*controller=header Disallow: /*controller=history Disallow: /*controller=identity Disallow: /*controller=images.inc Disallow: /*controller=init Disallow: /*controller=my-account Disallow: /*controller=order Disallow: /*controller=order-slip Disallow: /*controller=order-detail Disallow: /*controller=order-follow Disallow: /*controller=order-return Disallow: /*controller=order-confirmation Disallow: /*controller=pagination Disallow: /*controller=password Disallow: /*controller=pdf-invoice Disallow: /*controller=pdf-order-return Disallow: /*controller=pdf-order-slip Disallow: /*controller=product-sort Disallow: /*controller=registration Disallow: /*controller=search Disallow: /*controller=statistics Disallow: /*controller=attachment Disallow: /*controller=guest-tracking # Directories for oora-home.com Disallow: /app/ Disallow: /cache/ Disallow: /classes/ Disallow: /config/ Disallow: /controllers/ Disallow: /download/ Disallow: /js/ Disallow: /localization/ Disallow: /log/ Disallow: /mails/ Disallow: /modules/ Disallow: /override/ Disallow: /pdf/ Disallow: /src/ Disallow: /tools/ Disallow: /translations/ Disallow: /upload/ Disallow: /var/ Disallow: /vendor/ Disallow: /webservice/ # Files Disallow: /recuperation-mot-de-passe Disallow: /adresse Disallow: /adresses Disallow: /connexion Disallow: /inscription Disallow: /panier Disallow: /reduction Disallow: /historique-commandes Disallow: /identite Disallow: /mon-compte Disallow: /suivi-commande Disallow: /avoirs Disallow: /commande Disallow: /recherche Disallow: /suivi-commande-invite Disallow: /confirmation-commande