Cloudflare's new default configuration went live today. Mixed-use AI crawlers — bots that bundle search indexing with model training — are now blocked by default on pages that carry display advertising, across all Free tier accounts and any new domain onboarding to Cloudflare.
This affects a significant slice of the web. Cloudflare handles traffic for millions of sites, and its Free tier is the starting point for most of them. The change isn't subtle.
What the policy actually does
Cloudflare now splits AI bot traffic into three categories: Search (indexing content to answer questions later), Training (collecting content to train or fine-tune AI models), and Agent (bots acting in real time on a user's behalf). Under today's defaults, Training and Agent traffic is blocked on ad-carrying pages. Search traffic is allowed through.
The intent is straightforward. Publishers who run ads on their sites shouldn't have to hand over their content to train AI systems without consent or payment. Cloudflare announced in July it was moving toward a Pay Per Use model — publishers can eventually charge AI companies for content that generates value for their systems. September 15 is the first structural enforcement of that principle.
The problem that most publishers haven't absorbed
Google, Apple, and Microsoft run mixed-use crawlers. A single bot does search indexing and training collection simultaneously. Cloudflare's system has to classify each bot as one thing.
If Cloudflare classifies a mixed-use crawler as Training traffic, and your site has blocked Training by default, that crawler gets blocked entirely — including the part of it responsible for indexing your pages in Google Search and Bing.
According to reporting from TechCrunch, AI companies would need to separate their search and training functions into distinct crawlers before publishers can selectively block training without affecting search indexing. As of today, it's unclear which major AI companies have completed that separation.
The upshot: a publisher who configured their Cloudflare settings to "block AI training" may have de-indexed their own site without intending to.
Why paid search teams need to pay attention now
If you're a paid search manager, your first thought is probably "this is an SEO problem." That's accurate but incomplete.
Your Google Ads campaigns run against landing pages that need to be indexed and crawlable for Quality Score to function properly. A page that Google's bot can't access is a page where relevance signals go dark. That doesn't kill a campaign overnight, but a crawl block that persists for weeks degrades the signals Smart Bidding and AI Max use to match your ads to queries.
The more immediate concern is the AI search channel. ChatGPT Search, Perplexity, and Gemini increasingly influence purchase intent before a user even opens Google. If Cloudflare's defaults block the crawlers those platforms use to index your content, you lose the organic AI citation layer that's been driving incremental branded search volume. AI citations in ChatGPT and Perplexity are not something you can buy — they depend on your content being crawlable.
Worth noting: the article process guidelines from AI Weekly and Technology.org suggest that OpenAI and Perplexity have been working to classify their bots clearly as Search rather than Training. But "working toward" is not the same as "completed," and Cloudflare's classifications update as companies update their bot documentation.
What to check today
First, determine whether your site is on Cloudflare and which tier. Free tier accounts inherited the new defaults automatically today. Business and Enterprise customers have more granular controls and may not have been changed.
Second, run a site:yourdomain.com search in Google. Count the indexed pages and compare to what you expect. If the number has dropped significantly in the past week, cross-check with Google Search Console's Coverage and Crawl Stats reports — a sudden spike in crawl errors would confirm a bot access issue rather than a ranking change.
Third, if you find a problem, don't disable all AI protections to fix it. The right fix is to configure Cloudflare's bot settings to explicitly allow verified search crawlers by name or category, while continuing to block the specific crawlers you actually want blocked. Blunt disabling defeats the point.
One caveat: Cloudflare's bot categories aren't finalized across all crawlers. There's a meaningful lag between a company registering a new crawler with Cloudflare and that crawler appearing correctly in your bot management dashboard. If you're seeing unexpected crawl behavior today or tomorrow, it may stabilize over the next few days as Cloudflare processes registrations made before the deadline.
None of this is cause for alarm yet. The window to catch an accidental block before it compounds into a real performance issue is right now. Check your indexation, verify your Cloudflare settings, and flag it to whoever manages your infrastructure if something looks wrong.
If you want a quick read on how your paid campaigns are performing against your landing page health signals, the free audit at Gromerce takes about three minutes.
The crawler policy debate is now operational. The platforms that didn't separate their bots in time will be renegotiating with Cloudflare over the next few weeks.
Sources: TechCrunch, AI Weekly, July–September 2026
