Blog · 7 July 2026 · Updated 20 October 2026

Is Cloudflare blocking AI crawlers on your site without you knowing?

By Dimitris Drakatos, Founder, Wellknown

Cloudflare blocks AI crawlers by default on new sites that carry ads: since 15 September 2026 they start with AI training disallowed and AI agents blocked on pages that show ads. Two more dashboard settings can cut AI agents off from any site, and your analytics will not show it.

What did Cloudflare change?

On 1 July 2025 Cloudflare announced that AI crawlers would be blocked by default: every new domain joining its network was asked at sign-up whether to allow them, flipping the old opt-out model to opt-in. Cloudflare frames this as giving publishers control over how their content feeds AI, which for a publisher monetising page views is a reasonable position.

The scale is what makes it matter. By its own count, Cloudflare handles traffic for 20% of the web, so a default applied at its edge moves a meaningful share of it at once. An individual owner clicking through onboarding makes one decision. The default makes millions of them.

What changed on 15 September 2026?

Cloudflare launched a Training setting called Disallow AI Training, and blocking the Training category now stops Googlebot too. The groundwork was announced on 1 July 2026: bot traffic is now managed in three categories, Search, Agent and Training, with new defaults proposed for 15 September. Cloudflare CEO Matthew Prince put the reasoning plainly: "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge."

The version that launched differs from that proposal. Disallow AI Training writes a no-training rule into your robots.txt and blocks crawlers that only collect training data, such as OpenAI's GPTBot. Crawlers that do both search and training, such as Googlebot, Applebot and Bingbot, keep crawling for search. Choose Block, or Block on pages with ads, and those crawlers are stopped as well, search included. Cloudflare said existing Training blocks would move to Disallow AI Training, so those sites keep their search access.

New domains now get one of two presets. Sites that earn money from ads start with AI training disallowed. AI agents, the ones that fetch a page live because someone asked, are blocked on every page that shows an ad. Sites without ads start with everything allowed. Existing sites that had the older Block AI Bots switch on, and never set the newer controls, move to the same settings as the ad preset.

A Training setting of Block still reads as "keep my content out of AI training", and it now takes your ordinary search visibility with it. If any part of your estate sits behind Cloudflare, check how each category is set now rather than assume the old behaviour still holds.

The three settings that cut AI off, and how we know

We put a new domain through Cloudflare onboarding last week, setting up this site, and counted the switches that would have severed it from AI search. There were three, in three different menus, each presented as a sensible protective measure. Most site owners have never seen these switches.

None of these is a bug, and none is malicious. Each is a defensible security or licensing choice for some sites. The problem is that they are made once, often by whoever set up the CDN years ago, and their cost never appears anywhere the marketing team looks. Google Analytics does not record the AI visits that never happened.

"A page that looks perfect in your browser can be a blank page to the machine deciding whether to recommend you."

How do I check whether my site is affected?

Five checks, ten minutes, no access needed beyond your own Cloudflare login.

1. Open yourdomain.com/robots.txt in a browser and look for disallow rules naming GPTBot, CCBot, ClaudeBot, Google-Extended or similar agents. Rules you did not knowingly write are the first red flag.
2. In the Cloudflare dashboard, check Security for Bot Fight Mode and note whether it is on.
3. Under Scrape Shield, check whether Email Address Obfuscation is on, and decide whether hiding your contact details from AI answers is a trade you want.
4. Review the AI crawler policy under the domain's settings, where Search, Agent and Training each have their own position. For Training, Disallow AI Training keeps you in search, while Block shuts Googlebot out too.
5. Test from the outside. Fetch a key page as the agents themselves and compare what each is served. Our free Agent Parity Check does this in about 30 seconds for any public URL, fetching it as six AI crawlers (GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot, ClaudeBot and CCBot) plus a regular browser.

Should you block AI crawlers?

Sometimes, yes. A publisher whose business is the content itself has a real licensing argument for blocking training crawlers, and Cloudflare built these controls for that case. The distinction that matters is between training and answering, which are handled by different crawlers with different rules. Blocking training crawlers keeps your content out of future model weights. Blocking live agents, the ones fetching a page because a buyer just asked about you, keeps you out of the answer at the moment of decision. Most brands outside publishing want to be in the answer, and many have blocked both without ever making the choice.

Whatever your position, it should be a decision rather than a default someone else shipped. Check the settings, decide deliberately, and re-test from the outside, because what your dashboard says and what an agent is served do not always match.

Updated 20 October 2026: this post now follows what Cloudflare launched on 15 September 2026, not its July proposal. Disallow AI Training keeps Googlebot crawling for search. Choosing Block stops it.