Cloudflare's September 15 changes: what to check now

Cloudflare's September rollout changed the choices for search, training and agent traffic. The important distinction now is between Disallow AI Training and a full block, especially for crawlers that serve more than one purpose.

Reviewed October 1, 2026 · 6 min read

Cloudflare's September 15, 2026 rollout added Disallow AI Training. Its rollout announcement explains that this setting preserves search access for qualifying mixed-use crawlers, while Block can stop them entirely. Earlier coverage based only on the July plan misses that distinction.

For a business site, start with the settings actually applied to your domain. Do not assume that the absence of ads makes every bot rule irrelevant, or that a saved preference overrides all other security rules.

This guide explains the controls and a practical verification process. The goal is to allow the access your business intends while keeping training preferences and security decisions explicit.

Separate the crawler's purpose from the action you choose#

Cloudflare distinguishes Search, Agent and Training behavior. A crawler can serve more than one purpose, so read the action applied to that category carefully:

In the September announcement, new-domain presets allow Search. Ad-supported sites are offered Disallow AI Training and an ads-only Agent block; non-ad-supported sites are offered Allow for those categories. These are configurable presets, not a description of every existing domain.

Existing preferences are migrated under documented rules. Inspect the resulting values rather than assuming every Free-plan site was reset or every old setting kept its literal label.

Disallow AI Training and Block are different#

Disallow AI Training publishes the supported preference and preserves search access for Cloudflare's qualifying mixed-use crawlers. A full Training block can block mixed-use Googlebot, Bingbot and Applebot requests. Other WAF rules can still affect access.

A preference about content use and a rule refusing a request are different controls. Their effects depend on the provider and implementation. Verify both the published preference and the actual response to legitimate requests.

Document the decision with the person responsible for search and the person responsible for security. That avoids a later change being interpreted as a harmless label adjustment when it changes access.

Review customer-facing agent access#

User-triggered requests can help a customer inspect your services, products or policies. They are still automated requests and may need limits. Decide which public paths should work and which authenticated or sensitive actions require a separate permission check.

The crawler guide separates provider-specific search, training and user-triggered requests. Use it as a starting point, then check current documentation and your own traffic.

Check robots.txt and the edge together#

A crawler may be allowed by robots.txt and still receive an edge challenge or denial. Conversely, a publicly reachable page can carry a robots preference asking a compliant crawler not to fetch it. Neither file inspection nor a single HTTP test gives the whole picture.

Use CDN security events and access logs to identify the rule responsible for a failed request. Origin logs may miss requests served or blocked at the edge. The measurement guide explains which evidence each layer provides.

What to check after the rollout#

1. Confirm who manages your CDN, including a host or agency acting on your behalf.
2. Record the current Search, Agent and Training settings for each relevant domain.
3. Review other WAF, challenge and rate-limit rules that may affect the same paths.
4. Check the published robots.txt and any managed preference synchronization.
5. Inspect recent legitimate crawler requests and the exact rule behind denials or challenges.
6. Test key public pages again after a change and retain a rollback record.

Revisit the policy when your business changes#

An ad-supported publication, a private customer portal and a public service catalog have different needs. Make access decisions by content and workflow rather than assuming one blanket setting serves the whole site.

If you evaluate crawler monetization, review the current product terms and actual demand separately. Do not assume that blocking traffic automatically creates revenue or that a merchant's discovery goals match a publisher's licensing model.

A good outcome is specific: intended public search access works, training preferences are expressed as planned and sensitive actions remain protected. Save evidence of that result instead of relying on the name of a dashboard toggle.