Blog · 7 July 2026

Cloudflare blocks AI crawlers by default: is your site opted out without knowing?

By Dimitris Drakatos, Founder, Wellknown

Since 1 July 2025, Cloudflare has blocked AI training crawlers by default for every newly onboarded domain, and at least three further dashboard settings can cut AI agents off from a site entirely. From 15 September 2026 the defaults tighten again, extending to the agents that fetch pages live. Most site owners have never seen these switches. If your site sits behind Cloudflare, there is a realistic chance parts of it are invisible to the systems your buyers now ask for recommendations, and nothing in your analytics will tell you.

What did Cloudflare change?

On 1 July 2025 Cloudflare announced that new domains joining its network would block AI crawlers used for model training by default, flipping the old opt-out model to opt-in. Site owners choose at onboarding whether search crawlers, AI agents and training crawlers may access the site, and the recommended defaults lean towards blocking. Cloudflare frames this as giving publishers control over how their content feeds AI, which for a publisher monetising page views is a reasonable position.

The scale is what makes it matter. Cloudflare sits in front of roughly one in five websites, so a default applied at its edge moves a meaningful share of the whole web at once. An individual owner clicking through onboarding makes one decision; the default makes millions of them.

What changes on 15 September 2026?

On 1 July 2026 Cloudflare announced the next step: bot traffic is now managed in three categories, Search, Agent and Training, and from 15 September 2026 new domains will block both Training and Agent crawlers by default on pages that carry ads, with Search remaining allowed. Cloudflare CEO Matthew Prince put the reasoning plainly: "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge."

The detail with the widest consequences is how mixed-purpose crawlers are treated. A crawler that combines search with training is judged by all of its behaviours, so blocking the Training category will block crawlers such as Googlebot, Applebot and Bingbot outright, search function included, unless the owner opts out of the new defaults before the deadline. A setting that reads as "keep my content out of AI training" can now carry your ordinary search visibility with it. If any part of your estate sits behind Cloudflare, the weeks before 15 September are the time to make these choices deliberately.

The three settings that cut AI off, and how we know

We put a new domain through Cloudflare onboarding last week, setting up this site, and counted the switches that would have severed it from AI search. There were three, in three different menus, each presented as a sensible protective measure.

None of these is a bug, and none is malicious. Each is a defensible security or licensing choice for some sites. The problem is that they are made once, quietly, often by whoever set up the CDN years ago, and their cost never appears anywhere the marketing team looks. Google Analytics does not record the AI visits that never happened.

"A page that looks perfect in your browser can be a blank page to the machine deciding whether to recommend you."

How do I check whether my site is affected?

Five checks, ten minutes, no access needed beyond your own Cloudflare login.

1. Open yourdomain.com/robots.txt in a browser and look for disallow rules naming GPTBot, CCBot, ClaudeBot, Google-Extended or similar agents. Rules you did not knowingly write are the first red flag.
2. In the Cloudflare dashboard, check Security for Bot Fight Mode and note whether it is on.
3. Under Scrape Shield, check whether Email Address Obfuscation is on, and decide whether hiding your contact details from AI answers is a trade you want.
4. Review the AI crawler policy under the domain's settings: Search, Agent and Training each have their own allow or block position.
5. Test from the outside. Fetch a key page as the agents themselves and compare what each is served. Our free Agent Parity Check does this in about 30 seconds for any public URL, across seven agents including GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot, ClaudeBot and CCBot.

Should you block AI crawlers?

Sometimes, yes. A publisher whose business is the content itself has a real licensing argument for blocking training crawlers, and Cloudflare built these controls for exactly that case. The distinction that matters is between training and answering, which are handled by different crawlers with different rules. Blocking training crawlers keeps your content out of future model weights. Blocking live agents, the ones fetching a page because a buyer just asked about you, keeps you out of the answer at the moment of decision. Most brands outside publishing want to be in the answer, and many have blocked both without ever making the choice.

Whatever your position, it should be a decision rather than a default someone else shipped. Check the settings, decide deliberately, and re-test from the outside, because what your dashboard says and what an agent is actually served do not always match.