Should your store block AI crawlers in robots.txt?
Many stores added “Disallow: /” for AI crawlers in 2023 and 2024 to stop their content being used to train models. Since then, the same companies launched AI search — and a store that blocks their search crawlers cannot be read or cited there. The answer depends on which crawler you mean.
Check your site free
Training crawlers and search crawlers are different
Most AI companies now run separate crawlers. Some collect content to train models: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl). Others fetch pages to answer searches and link to them: OAI-SearchBot for ChatGPT search, Claude-SearchBot, PerplexityBot. And some fetch a page only when a user asks about it, such as ChatGPT-User.
So a store can decline training and still be cited: disallow the training crawlers, allow the search crawlers. Blocking everything with one rule removes you from AI answers that could send shoppers your way.
Google is a special case. Google-Extended controls whether your content is used for Gemini models; it does not affect Google Search, rankings or AI Overviews, which use the normal Googlebot. Blocking Googlebot to avoid AI Overviews would remove you from Google entirely.
The main AI crawlers
- GPTBot: OpenAI — collects content for training.
- OAI-SearchBot: OpenAI — powers ChatGPT search results and citations.
- ClaudeBot / Claude-SearchBot: Anthropic — training, and search for Claude.
- PerplexityBot: Perplexity — builds its search index.
- Google-Extended: Google — use of content for Gemini; not Search ranking or AI Overviews.
- CCBot: Common Crawl — an open crawl many models are trained on.
A sensible robots.txt for a store
- Decide on training: if you do not want your content used, disallow GPTBot, ClaudeBot, CCBot and Google-Extended.
- Allow the search crawlers you want to be cited by: OAI-SearchBot, Claude-SearchBot, PerplexityBot.
- Never block Googlebot or Bingbot to control AI features — that removes you from search itself.
- Check the result with a site audit, then watch where you are cited with the AI Overview tracker.
Questions people ask
- Will blocking GPTBot remove me from ChatGPT search?
- Not on its own — ChatGPT search uses OAI-SearchBot. Blocking OAI-SearchBot is what removes you.
- Does blocking Google-Extended hurt my Google rankings?
- No. It only covers use for Gemini models; Search uses Googlebot.
- Does NicheSEO Pro check this?
- Yes. The site audit reads your robots.txt and flags it when AI search crawlers — OAI-SearchBot, Claude-SearchBot, PerplexityBot or ChatGPT-User — are blocked.