Robots.txt: Block AI Training Bots, Keep Ecommerce Search Open

Yes, robots.txt is the standard way to tell AI crawlers whether they can index your pages or use them for training, but compliance is voluntary. Use named tokens like GPTBot to block training crawlers while leaving search-focused tokens allowed if you still want your pages cited in AI search results. The file won’t physically stop a rogue bot, but every major AI vendor documents and honors it.

TL;DR:

• Place robots.txt at the domain root, create a separate file for every subdomain, and capitalize User-agent and Disallow exactly because directive names are case sensitive.
• OpenAI separates its search crawler from its training crawler, so you can allow search citations while blocking training; changes may take about 24 hours.
• Google’s AI training control does not affect Googlebot search crawling; Anthropic exclusion requires separate rules for both of its training crawlers.
• A disallowed page cannot expose its noindex tag to crawlers, so use meta directives for HTML pages and HTTP headers for PDFs and images.
• Robots.txt cannot protect sensitive content or stop noncompliant bots; use authentication for private material and CDN or WAF blocking when requests continue.

Copyable Robots.txt Snippets to Block or Allow AI Crawlers

A robots.txt file lives at the root of your domain (yoursite.com/robots.txt), never in a subfolder, and each subdomain needs its own copy. Directive names and user-agent tokens are case-sensitive, so Disallow and User-agent must be written exactly as shown, though the paths you list can vary by server configuration.

A minimal setup that blocks training crawlers but keeps search visibility looks like this:

• User-agent: GPTBot followed by Disallow: / to opt out of OpenAI’s training crawler.
• User-agent: OAI-SearchBot followed by Allow: / to keep appearing in ChatGPT search results.
• User-agent: * followed by Allow: / so general crawlers, including search engines, keep full access.

For sites that want to opt out of AI training entirely, a fallback block list simply repeats Disallow: / under each named training token (GPTBot, Google-Extended, ClaudeBot, and others) while leaving the wildcard group open for search engines. A conservative “block nothing” template skips the named groups altogether and uses only User-agent: * with Allow: /, which is the default state if you never add a robots.txt file at all.

Keep the file short. Search Engine Land recommends disallowing only the specific training tokens you want to exclude rather than trying to explicitly allow every crawler you don’t mind, since broad allow lists are harder to maintain and more prone to accidental site-wide blocks.

Pro Tip: Test every new robots.txt edit on a staging subdomain first, since a misplaced wildcard can accidentally block Googlebot along with the AI crawlers you meant to exclude.

How User-Agent Tokens and Precedence Rules Work

Robots.txt parsing follows strict rules, and getting them wrong is the easiest way to lock out search engines by mistake. Under RFC 9309, user-agent tokens can only contain letters, underscores, and hyphens, so wildcards inside a token name itself are invalid. You must list each crawler you want to target by its exact name rather than trying to pattern-match several at once.

A few rules decide which group and rule actually apply:

• A named group like User-agent: GPTBot takes precedence over the wildcard User-agent: * group for that specific crawler.
• When multiple Disallow or Allow rules could match a URL, the longest matching path wins, regardless of which rule appears first in the file.
• A trailing $ anchors a rule to the exact end of a path, letting you block a specific file type without affecting an entire directory.

Getting the token name wrong, even by a character, means the crawler simply falls back to the wildcard group instead of the rule you intended.

Vendor-Specific Robots.txt Entries Worth Knowing

Each major AI vendor publishes its own tokens, and they don’t always map directly to a single bot. OpenAI uses OAI-SearchBot for search indexing and GPTBot for training data collection, and the two are controlled independently so you can allow one and disallow the other. Changes to either directive can take about 24 hours to take effect, according to OpenAI’s own documentation, so don’t expect an instant shift in crawl behavior after editing your file.

Google’s Google-Extended is a control token, not a separate HTTP request agent. It governs whether Google’s AI products can use your content for training and generative features, while Googlebot continues crawling for search under its own rules.

• User-agent: GPTBot with Disallow: / opts out of OpenAI training while leaving OAI-SearchBot untouched.
• User-agent: Google-Extended with Disallow: / opts out of Google’s AI training use without affecting standard search indexing.
• User-agent: anthropic-ai and User-agent: ClaudeBot cover Anthropic’s crawlers, and both typically need an explicit Disallow if you want to exclude training.
• User-agent: Applebot-Extended controls Apple’s AI training use separately from the main Applebot used for Siri and Spotlight.
• Perplexity’s crawlers are worth pairing with edge-level enforcement, since reports have described noncompliance incidents where a crawler kept fetching pages despite a disallow rule.

Layering Robots.txt With Meta Tags and Edge Enforcement

Robots.txt is a request, not a wall, and Google’s own documentation confirms that compliance is voluntary rather than a binding security measure. Disallowed pages can still surface as bare URLs in search results, and because a Disallow rule stops the page from being fetched at all, any noindex meta tag you placed on that page never gets read.

That’s why layering matters. HTML pages can carry noai or noimageai meta directives for page-level opt-outs, while non-HTML resources like PDFs, images, and feeds need an X-Robots-Tag HTTP header instead, since there’s no page body to hold a meta tag.

• Use robots.txt as the first layer for well-behaved crawlers that read and respect it.
• Add meta tags or X-Robots-Tag headers for granular, page-by-page or file-type control.
• Turn to authentication or password protection when content is genuinely sensitive, since robots.txt was never designed as a security measure.
• Consider Cloudflare’s Bot Preference Sync, which can prepend managed directives to your existing robots.txt and pair those preferences with edge-enforced blocking by category (Search, Agent, Training) so your stated preferences and actual enforcement stay aligned.

Pro Tip: If a crawler ignores your robots.txt entirely, edge enforcement through your CDN or WAF is the only layer that actually stops the request.

Testing and Verifying That Your Directives Are Working

Confirming that a crawler actually honors your robots.txt takes a short checklist rather than guesswork.

1. Check your server logs for requests to /robots.txt from the specific vendor user agents you’ve targeted, and confirm they appear before any crawl of disallowed paths.
2. Allow for vendor update windows. OpenAI’s documentation notes changes can take roughly 24 hours to propagate, so re-check after a full day rather than immediately.
3. Use a live fetch tool or validator, such as BabyLoveGrowth’s robots.txt checker, to confirm your syntax parses correctly and matches your intent.
4. Review CDN or WAF reports alongside your robots.txt rules to catch any crawler that keeps fetching blocked paths despite your directives.
5. Set up a targeted test page with a unique disallow rule and monitor whether it appears in that vendor’s output over the following days.

Balancing AI Visibility Against Training Opt-Out

For ecommerce sites, the decision usually comes down to three factors: how much revenue depends on AI-driven referrals, how sensitive your product content is, and how often your catalog changes. If AI search citations send meaningful traffic, allow the search tokens and disallow only the training ones. If your descriptions, pricing, or imagery are easily scraped and repackaged elsewhere, disallowing training crawlers protects that investment. Merchants who lean toward visibility still need a way to measure whether it’s working, which is where a dedicated optimization tool earns its place.

— Xhurian

Ecentic: Measure and Optimize Your AI Shopping Visibility

Choosing to allow search tokens is only half the job. The harder question is whether AI shopping agents like ChatGPT and Gemini actually recommend your products once they can see them. We built ecentic to answer that question directly: we connect to your Shopify or WooCommerce store, simulate how AI agents evaluate your product pages, and show you exactly where you win or lose against competitors.

If your goal is more AI-driven traffic rather than less, optimization beats blocking. We turn our diagnostics into concrete rewrite suggestions and publish them back to your store with one click, so the gap between “visible to AI” and “recommended by AI” can close.

• Run simulation-driven scans to see how various AI shopping agents read your listings.
• Get plain-English, actionable recommendations tied to specific product pages, not generic SEO advice.
• Publish fixes directly to Shopify or WooCommerce without leaving your dashboard.

Start with a free product scan to see where your listings stand today.

FAQ

Is robots.txt legal to use for blocking AI bots?

Yes, publishing a robots.txt file and listing any crawler tokens you choose to disallow is a standard, legal part of running a website under the Robots Exclusion Protocol. It simply expresses your crawling preferences; it carries no legal enforcement power on its own, so pair it with authentication for anything that truly needs protection.

Is there a chatroom or forum for discussing AI bot behavior?

There’s no single official chatroom run by AI vendors for robots.txt questions, but each vendor publishes its own documentation, such as OpenAI’s crawler overview and Google’s robots.txt guidance, which are the most reliable places to confirm current token behavior.

Is robots.txt still used and relevant today?

Robots.txt remains the primary, vendor-recognized way to express crawler preferences, including for newer AI training and search bots from OpenAI, Google, and Anthropic. Major AI companies continue to document and update their supported tokens, which is why it’s still the first control most site owners configure.

Do bots make up most of internet traffic?

Bot traffic, including search crawlers and AI agents, makes up a meaningful share of requests hitting most websites, though the exact proportion varies widely by site and industry. Checking your own server logs for crawler user agents is the most reliable way to see how much of your traffic is automated rather than relying on a single industry-wide figure.

Does blocking GPTBot also block ChatGPT search results?

No, blocking GPTBot only opts out of training data collection and has no effect on OAI-SearchBot, which OpenAI controls separately for search indexing. You need to disallow OAI-SearchBot specifically if you also want to exclude your pages from ChatGPT’s search results.

Sources

• Overview of OpenAI Crawlers
• Robots overview (Google Search Central)
• Say it once: introducing Bot Preference Sync | Cloudflare Blog
• RFC 9309
• Robots.txt and SEO: What you need to know in 2026