Cloudflare’s September 15 deadline: what it actually changes for your business

On September 15, Cloudflare flips its AI crawler defaults. Most of the coverage is written for publishers, and if you run a business site the honest answer is that the new default probably does not fire on you. The part that should worry you is a setting you might turn on yourself.

Reviewed September 2, 2026 · 10 min read

On September 15, 2026, Cloudflare changes the default treatment of AI crawlers. Bots classified as Training or Agent get blocked on pages that display ads, while Search crawlers stay allowed. Cloudflare sits in front of roughly a fifth of the web, so when it changes a default, a large slice of the internet changes behaviour without anyone touching a dashboard.

Nearly all the coverage of this is written for publishers, and it reads as alarming. For most businesses it should not. The new default is conditional on one thing: whether the page displays ads. A plumber, a clinic, a spa, or a small retailer typically has no ad inventory at all, so the default has nothing to fire on. If you have been putting off reading about this because it sounded like an emergency, that instinct was roughly correct.

The part that deserves your attention is different, and it points the opposite way from the panic. The riskiest thing here is not Cloudflare’s default. It is a setting you might turn on yourself, in a reasonable attempt to protect your content, that quietly removes you from Google.

What actually changes#

Cloudflare replaced its single AI-bot on/off switch with three categories, defined by what the crawler is doing rather than who operates it:

For each category you can choose to block on all pages, block only on pages displaying ads, or not block at all. Cloudflare’s reasoning for the ad condition is worth understanding because it explains the whole design: an ad on a page is a signal that the owner expected a human to see it. Where a human was expected and a machine arrived instead, Cloudflare treats that as the case for compensation rather than free access.

Who inherits the new defaults: new domains onboarding to Cloudflare, new sites added by existing customers, and all existing customers on the Free plan. Paid customers with an existing configuration keep what they have. If your zone has an explicit AI bot preference saved before September 15, Cloudflare respects it.

The trap: blocking Training can block Google#

This is the part to actually absorb. Some crawlers do more than one job. Googlebot indexes for Search and also collects data that trains models. Cloudflare evaluates these multi-purpose crawlers under every policy that applies to them, and the strictest applicable rule wins.

The tension is real rather than a configuration quirk, and Cloudflare cannot resolve it: Google bundles search indexing and training data collection into one crawler, so there is no setting that separates them from your side. Cloudflare’s three-category taxonomy names the problem clearly but the choice it leaves you is still binary for Googlebot specifically.

For a business that wants customers, the answer is usually straightforward. Publishers whose content is the product may reasonably trade search visibility for training protection. A business whose content exists to sell a service almost never should.

The category merchants should actually care about#

Agent is the quiet one. It covers automated activity acting in real time on a person’s behalf, which is the exact traffic that arrives when a customer asks an assistant about your business and it goes and fetches your page. Blocking Agent does not protect anything from training. It removes you from the moment a real person is being helped to decide.

That is the whole thesis of agentic commerce running in reverse. The crawler guide covers the full roster of which named bots fall into which category, and the short version for a business is: allow Search, allow Agent, and make Training a deliberate decision rather than a default you inherited or a box you ticked in a hurry.

The bug this deadline will expose#

Site audits keep turning up the same failure, and September 15 will create more of it: robots.txt says one thing while the edge does another. Robots.txt is a sign on the door that well-behaved crawlers read and honour, and that some user-triggered fetchers openly say may not apply to them. Cloudflare’s bot rules are not a sign, they are a bouncer. The request gets a 403 whether the bot likes it or not.

The result is an owner who is confident they are visible to AI assistants while every fetch has been dying at the edge for months. Nothing in your analytics reveals this, because blocked crawlers never reach the page and never fire a tag. It shows up only in logs, which is one more reason to instrument them; the measurement guide covers how, including the query that surfaces exactly this by counting status codes served to AI crawlers.

What to do before September 15#

1. Find out whether you are on Cloudflare at all. Many businesses are, through a host or agency, without the owner knowing. If nobody on your team has a Cloudflare login, this deadline is not yours.
2. If you are, open Security, then Settings, then the AI bot policy controls, and record a deliberate choice rather than inheriting one. An explicit preference saved before September 15 is respected.
3. For a normal business site: allow Search, allow Agent. Decide Training on its merits, understanding that blocking it can take Googlebot, Bingbot, and Applebot with it.
4. Prefer explicit allow rules for the crawlers your business depends on over relying on category defaults, since defaults change and yours just did.
5. Check your logs after September 15 for 403s served to AI crawlers. That is the only place a misconfiguration will surface.
6. Make robots.txt and your edge rules tell the same story. Two layers disagreeing is how sites end up invisible while looking fine.

The bigger pattern#

Cloudflare also replaced its per-fetch Pay Per Crawl experiment with Pay Per Use, which pays publishers when their content actually surfaces in an AI answer rather than when it is merely crawled. Read that alongside the rest of what Cloudflare shipped this year and a strategy comes into focus: it is not trying to keep agents off the web, it is building the place where agents pay to come in.

That is worth knowing because it tells you which way this is heading. The long-run equilibrium being built is not blocked or unblocked; it is metered. For a business selling something, that is a better world than it sounds, because the thing being metered is content, while what you actually want from an agent is a customer. Staying readable and reachable is how you end up on the receiving end of that traffic rather than behind the toll booth.

The deadline itself is a small task with a real downside if ignored. Ten minutes in a dashboard, one deliberate decision recorded, and a log check afterwards. The businesses that get hurt on September 15 will overwhelmingly be the ones that blocked something on purpose without knowing what else it took with it.