Guides / robots.txt for AI agents
robots.txt for AI shopping agents, without feeding the training crawlers
Many stores block every AI bot in robots.txt, including the ones that shop and search for real people. Here’s which user agents do what, and a robots.txt that lets shoppers in and keeps training crawlers out.
Check yours in seconds: our free robots.txt checker shows which AI agents your robots.txt lets in.
After 2023 a lot of stores added “block all AI bots” to robots.txt to keep their content out of model training. Many of those rules also block the bots that search and shop for real people. In our audit, 2 of 28 stores told AI shopping agents to stay out entirely.
Three kinds of AI bot
The big AI companies now use separate user agents for separate jobs, so you can treat them differently:
| Company | Training | Search | Fetching for a person |
|---|---|---|---|
| OpenAI | GPTBot |
OAI-SearchBot |
ChatGPT-User |
| Anthropic | ClaudeBot |
Claude-SearchBot |
Claude-User |
| Perplexity | none listed | PerplexityBot |
Perplexity-User |
Google-Extended (a token, not a crawler) |
Googlebot |
none listed |
- Training bots collect content for future models. Blocking them doesn’t change whether you appear in AI answers.
- Search bots index your pages so AI assistants can find and cite them. Block these and you disappear from AI search.
- Fetching for a person happens when someone asks an assistant to look at, or buy from, a page. OpenAI and Perplexity both say robots.txt may not apply to these, because a person started the request. Blocking them still sends a clear “not welcome” signal, and some agents respect it.
Personal agents like Meta’s Muse or Instinct mostly don’t identify themselves at all. They look like a normal browser. For those, robots.txt doesn’t matter, but your firewall does: see Cloudflare and AI agents.
A robots.txt for a store
This keeps training crawlers out and lets search and shopping in:
# Keep AI training crawlers out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /
# Let AI search and agents shopping for people in
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
Disallow: /wp-admin/
# Everyone else
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yourstore.com/sitemap.xml
Don’t disallow /cart/ or /checkout/ for the agent user agents above. They need to reach them to buy.
On WordPress
WordPress generates a virtual robots.txt, and SEO plugins (Yoast, Rank Math) let you edit it. Some security plugins and Cloudflare’s Managed robots.txt also add AI rules on top of yours, so check the live file at yourstore.com/robots.txt, not just your settings.
Tallygator Pro manages these rules for you: shopping agents allowed, training crawlers kept out, and your own rules left in place.