Agent readiness

Which AI crawlers should you allow? The robots.txt decision most sites make by accident

The publisher version of this question is about compensation. The small-business version is much simpler, and the risk runs the other way: almost every site that is invisible to AI agents got that way without ever deciding to be. Here is the decision, bot by bot, plus the three blocks people set without realizing.

Updated August 21, 2026 · 11 min read

Most writing about AI crawlers is written for publishers, and publishers have a genuine grievance: their product is the content, and a model that ingests it and answers on their behalf takes the visit with it. That fight is real, it is why pay-per-crawl marketplaces exist, and none of it describes your situation if you run a clinic, a bakery, or a plumbing company.

For a business that wants customers, content is the advertisement, not the product. An agent reading your prices and hours and then telling a buyer to come to you is the outcome you want. The failure mode is not being read too much. It is being unreadable, and in our scan of 652 small-business websites almost a third of sites offered agents nothing machine-readable at all. A meaningful slice of that was not a choice anyone made. It was a default.

One of those defaults changes on September 15, 2026, which is a good excuse to do this properly.

There is no such thing as "AI crawlers"

The single biggest reason people get this wrong is treating one label as one decision. At least three different jobs hide under it, and a sensible business answers them differently. Cloudflare formalized the split in July 2026, and the categories are worth borrowing whatever your host is.

Notice that the category people actually have feelings about, Training, is the one with the smallest practical stake for a local business. The two that decide whether a buyer hears about you at all are the two nobody thinks about.

The roster, by who sends it

Every major provider now runs separate user agents for the separate jobs, which is what makes a considered answer possible. These are the tokens you write into robots.txt, taken from each company’s own documentation rather than from an aggregator blog.

Two entries in that table deserve a second look, because they are where the accidents happen.

ChatGPT-User and Perplexity-User are not crawlers. They are a person, right now, asking about you, with an assistant doing the fetching. OpenAI notes that robots.txt may not apply to ChatGPT-User because the request is user-initiated, and Perplexity documents that Perplexity-User generally ignores robots.txt for the same reason. That is exactly why these get blocked at the firewall instead, and blocking them is the closest thing to hanging up on an inbound call that exists on the modern web.

What changes on September 15, 2026

On July 1, 2026, Cloudflare announced new defaults for AI traffic that take effect September 15. For every new domain onboarding to Cloudflare, Training and Agent bots are blocked by default on pages that display advertisements. Search bots stay allowed everywhere. Existing sites keep their current behavior and can set their preference in Security settings before the date.

Read the scope carefully, because the coverage around it has been loose. This is a default for new domains, on ad-bearing pages. If you are an established site that does not run ads, September 15 does nothing to you on its own. The reason to care anyway is the second half of Cloudflare’s note: a customer who chooses to block Training also blocks multi-purpose crawlers that combine training with search, and Cloudflare names Googlebot, Bingbot, and Applebot as examples.

That is the trap in one line. The control is labeled Training. The blast radius includes Google Search. A three-click decision made on principle can quietly cost you the channel that still sends most of your traffic.

The robots.txt a local business should actually publish

For the overwhelming majority of businesses selling a product or a service, the correct file is short and permissive, with the AI agents named explicitly rather than left to a wildcard. Naming them costs nothing and removes the ambiguity that gets you dropped by a cautious parser.

User-agent: *
Allow: /

# Search and agent traffic: these decide whether a buyer hears about you
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Googlebot
User-agent: Storebot-Google
Allow: /

# Training: allow, or drop these three lines if you would rather not
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

This is close to what Nexez publishes for itself, for the obvious reason that a company promising to make businesses legible to agents would look silly blocking them. If you want to see the pattern in the wild, fetch the robots.txt of any site you admire and compare.

Three rules that matter more than the file’s contents. Serve it from the root of every subdomain you use, because robots.txt does not inherit. Keep any path you want quoted out of the disallow list, since a blocked page cannot be cited. And do not disallow your own sitemap, feed, or structured-data endpoints, which is a surprisingly common own goal.

When blocking is the right answer

This guide argues for allowing, and the argument does not generalize to everyone. Blocking Training crawlers is reasonable and sometimes correct when your content is the thing being sold rather than the thing selling something else.

• Publishers, research shops, and anyone whose archive is the product. A model that absorbs the archive competes with the archive.
• Proprietary datasets, original photography, and licensed material where the license terms genuinely constrain redistribution.
• Sites with a real compensation path available. Cloudflare’s pay-per-crawl marketplace, and the pay-per-use successor it announced in July 2026, only pay you if the free door is closed first.
• Anywhere the crawl load itself is a cost problem, though rate limiting is usually the better instrument than an outright block.

Even in those cases, keep the Search and Agent categories open unless you have decided you do not want to be found. Blocking Training while staying discoverable is a coherent position. Blocking everything is a decision to exit the channel.

robots.txt is a request, not a lock

Whichever way you decide, be clear about what the file is. robots.txt is a published preference that well-behaved operators honor voluntarily. The named crawlers from OpenAI, Anthropic, Google, and Perplexity do honor it, which is what makes it useful. Nothing enforces it, and anyone can send a request claiming to be GPTBot.

That cuts both ways, and the practical consequence is the same in either direction: never make a decision based on the user agent string alone. If you are blocking, a string match blocks the honest bots and misses the dishonest ones. If you are allowlisting, a string match hands your allowlist to anyone who can type. OpenAI and Perplexity publish JSON files of their crawler IP ranges for exactly this reason, and Anthropic now publishes ranges too. Verify against those, or against reverse DNS, and treat the string as a hint.

Where this is heading is cryptographic rather than textual. Cloudflare spent 2026 moving from static bot lists toward continuous evaluation of how an agent actually behaves, and signed-request schemes are being standardized so an agent can prove who it is instead of asserting it. Nothing to implement yet. Worth knowing that the string-matching era is ending.

Then measure it, because the answer changes

A robots.txt decision made once and never revisited is how sites end up blocking a crawler that was renamed two years ago while missing three that launched last quarter. The roster in this article will be out of date; the discipline of checking will not be.

The check is a log query, not an analytics dashboard, because AI crawlers do not execute JavaScript and Google Analytics therefore reports zero of them no matter how many arrive. Measuring AI agent traffic covers the full instrument set, but the ninety-second version is: grep your server or CDN logs for the user agents in the table above, and look at the response codes rather than the hit count. A wall of 403s means something in your stack is answering the door and slamming it, and that is worth knowing before you spend another afternoon on structured data nobody is allowed to fetch.

Once the door is open, the work that follows is about being worth reading: JSON-LD that states your facts in a typed form, a feed if you sell anything, and eventually an endpoint an agent can act on rather than just quote. Access is only the precondition. It is a precondition an unsettling number of sites fail.