Skip to content

Guides / robots.txt for AI agents

robots.txt for AI shopping agents, without feeding the training crawlers

Many stores block every AI bot in robots.txt, including the ones that shop and search for real people. Here’s which user agents do what, and a robots.txt that lets shoppers in and keeps training crawlers out.

Updated November 13, 2026 · 5 min read

Check yours in seconds: our free robots.txt checker shows which AI agents your robots.txt lets in.

After 2023 a lot of stores added “block all AI bots” to robots.txt to keep their content out of model training. Many of those rules also block the bots that search and shop for real people. In our audit, 2 of 28 stores told AI shopping agents to stay out entirely.

Three kinds of AI bot

The big AI companies now use separate user agents for separate jobs, so you can treat them differently:

Company Training Search Fetching for a person
OpenAI GPTBot OAI-SearchBot ChatGPT-User
Anthropic ClaudeBot Claude-SearchBot Claude-User
Perplexity none listed PerplexityBot Perplexity-User
Google Google-Extended (a token, not a crawler) Googlebot none listed
  • Training bots collect content for future models. Blocking them doesn’t change whether you appear in AI answers.
  • Search bots index your pages so AI assistants can find and cite them. Block these and you disappear from AI search.
  • Fetching for a person happens when someone asks an assistant to look at, or buy from, a page. OpenAI and Perplexity both say robots.txt may not apply to these, because a person started the request. Blocking them still sends a clear “not welcome” signal, and some agents respect it.

Personal agents like Meta’s Muse or Instinct mostly don’t identify themselves at all. They look like a normal browser. For those, robots.txt doesn’t matter, but your firewall does: see Cloudflare and AI agents.

A robots.txt for a store

This keeps training crawlers out and lets search and shopping in:

# Keep AI training crawlers out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

# Let AI search and agents shopping for people in
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
Disallow: /wp-admin/

# Everyone else
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yourstore.com/sitemap.xml

Don’t disallow /cart/ or /checkout/ for the agent user agents above. They need to reach them to buy.

On WordPress

WordPress generates a virtual robots.txt, and SEO plugins (Yoast, Rank Math) let you edit it. Some security plugins and Cloudflare’s Managed robots.txt also add AI rules on top of yours, so check the live file at yourstore.com/robots.txt, not just your settings.

Tallygator Pro manages these rules for you: shopping agents allowed, training crawlers kept out, and your own rules left in place.

Find out before a customer’s agent does.

Free. No signup. The agent never pays.