The switch that can hide your website from Google changes on September 15
Cloudflare is replacing its single 'Block AI bots' toggle with three categories (Search, Agent, Training) and setting new defaults on September 15, 2026. The part that matters: Googlebot, Bingbot, and Applebot are classified as multi-purpose crawlers, so any setting that blocks AI training now blocks them too. Reports of Googlebot getting 403s started in early August. Here is what changes, who it actually affects, and the five-minute check.
Somewhere in 2024 or 2025, a lot of people flipped a switch in Cloudflare labelled Block AI bots. It was one click, it was free, and the pitch was obvious: stop AI companies from hoovering up your content for training. Plenty of small business sites still have it on. Most of the owners have forgotten it exists.
On September 15, 2026, what that switch does changes. It is being retired and folded into a more granular system, and in the process it starts blocking crawlers it never blocked before. Among them: Googlebot.
This is not a hypothetical. Reports of Googlebot and Bingbot getting refused started appearing in the first week of August. If your site is behind Cloudflare, this is a five-minute check that is worth doing before mid-September, and if it turns out you are fine, you can stop thinking about it entirely.
What Cloudflare announced
On July 1, 2026, Cloudflare replaced the single on/off “Block AI bots” toggle with three categories, each of which you control separately. In their own definitions:
| Category | What it covers |
|---|---|
| Search | Crawlers that collect or index your content so they can answer questions about it later |
| Agent | Automated activity acting in real time on a person’s behalf, such as chat fetch bots and browser-use agents |
| Training | Crawlers taking your content to train or fine-tune a model |
For each one you can block on all pages, block only on pages that display ads, or not block at all. Cloudflare also tracks a longer list of classifications behind the scenes (Transact, Data Collection, Security Testing, SEO, Ads Verification, Social and Link Preview, Feed Fetching, and Monitoring and Operations), but those three are the ones with switches attached.
The split is a real improvement over one toggle. “Do not train on my content” and “do not let ChatGPT read this page when a customer asks about my business” are completely different positions, and the old toggle could not tell them apart.
The two things that happen on September 15
New defaults. For domains newly onboarded to Cloudflare, new sites added to existing accounts, and existing free-plan customers who have not set their own preference by the deadline, Training and Agent get blocked by default on pages that display ads, while Search stays allowed. Cloudflare’s reasoning for scoping it to ad pages is that an ad is a signal the owner meant a person to land there, which makes it monetizable.
If your site does not run display ads, this default does essentially nothing to you. That covers most small business marketing sites, and it is worth saying plainly because the coverage of this change has been aimed at publishers, whose situation is not yours.
Multi-purpose crawlers get reclassified. This is the part that matters, and it is not scoped to ad pages.
Cloudflare’s own documentation puts it this way: crawlers that combine Search and Training “will also be blocked by all configurations to block AI training, including the legacy ‘Block AI bots’ option.” Googlebot, Applebot, and Bingbot are all classified as multi-purpose, because the same crawl feeds both the search index and the AI features built on top of it. Defaults are enforced by the most restrictive applicable rule, so if any of your settings say block Training, they now catch the search engines too.
The legacy “Block AI bots” toggle itself is deprecated on September 15. Its whole design was to exclude mixed-purpose bots, which is exactly the gap the new framework closes.
Put those two facts together and you get the failure mode: a site owner who clicked one button two years ago to keep their content out of training data has, without doing anything else, also told Cloudflare to refuse Google.
It has already started happening
In the first week of August a site owner posted in r/SEO that setting AI Training to Block caused both Googlebot and Bingbot to receive HTTP 403 responses when fetching their sitemap. They confirmed the responses went away when the setting was turned off, that the requests came from legitimate crawlers rather than spoofed user agents, and that Cloudflare’s own dashboard showed the search engines as blocked. Google’s John Mueller responded asking for details so he could look into it.
Whether that specific case was the September change arriving early, a bug, or a misconfiguration was not resolved publicly. The mechanism, though, is documented and intentional, and the date it becomes the norm is a few weeks away.
The consequence of a 403 to Googlebot is not subtle and it is not immediate, which is the bad combination. The request never reaches your server, so Google cannot crawl or refresh anything. Nothing disappears that day. Pages go stale, then they get dropped, and the traffic decays over weeks while everything on your end looks normal.
Who is affected and who is not
Not affected at all: anyone not using Cloudflare. If your site is on Vercel, Netlify, Squarespace, Shopify, Wix, or a managed WordPress host with no Cloudflare in front of it, none of this applies. Check before you assume, because Cloudflare often gets added by whoever set up the DNS and never mentioned again.
Almost certainly fine: a Cloudflare site on a paid plan where nobody ever touched the bot settings. Existing configured domains do not get silently rewritten; the new defaults are aimed at new domains and at free-plan sites that never expressed a preference.
Worth checking today: anything where somebody once turned on “Block AI bots,” or set AI Training to Block, or added a WAF rule or robots.txt entry aimed at AI crawlers. This is the group with real exposure, and it is bigger than people think, because the toggle was heavily promoted and cost nothing to enable.
Also worth checking: free-plan sites, which pick up the new defaults if no preference is set, and any site that runs display ads, which is what the defaults actually target.
The decision most small businesses should make
Publishers and small businesses have opposite incentives here, and most of the advice being written about this change is written for publishers.
A publisher sells attention on their own pages. An AI assistant that reads their article and answers the question has taken the value and left the cost. Blocking is a defensible commercial position for them.
You are not selling attention. You are selling plumbing, or legal work, or physiotherapy appointments. Every one of those AI systems reading your site is a channel through which somebody asking “who does X near me” can be told your name. Being read is the entire point. We have written the checklist version of this and the longer argument behind it, and none of it works if the crawler gets a 403 at the door.
So for a typical small business site, the sensible configuration is:
- Search: allow. This is not really optional. Blocking it means blocking Google.
- Agent: allow. This is the live fetch that happens when a customer asks an assistant about you and it goes to read your actual pages. Blocking it is how you get left out of the answer, or worse, described from stale third-party information.
- Training: your call. This is the only one where “no” is a coherent position. Just understand that with the September change, blocking Training also blocks the multi-purpose search crawlers, which is a price almost no small business should be willing to pay.
If you genuinely want to keep your content out of training data, the mechanism that does that without collateral damage is the signalling layer, not a block. Cloudflare’s managed robots.txt now supports Content Signals with a fourth field expressing intended use (use=immediate, use=reference, which is the default, or use=full). Signals state a preference rather than enforcing one, which is a real limitation, and it is also the difference between declaring a policy and disconnecting your website from Google.
The five-minute check
- Confirm whether you are on Cloudflare. If you do not know, ask whoever manages your DNS, or look at where your nameservers point.
- Open the settings. In the Cloudflare dashboard: select your domain, then Security, then Bots, then the AI Scrapers and Crawlers section. You will see the per-category controls for Search, Agent, and Training.
- Look for anything set to block. Especially the legacy toggle. If Training is blocked and you did not deliberately decide that this month, that is your finding.
- Verify from the outside. Use the URL Inspection tool in Google Search Console and run a live test on a few pages, including your sitemap. A 403 there is unambiguous. While you are in Search Console, check the crawl stats for a rise in failed requests.
- Put a reminder in the calendar for late September. Settings changes and platform defaults have a way of not matching each other. Checking once after the date costs nothing.
Also worth knowing, though not urgent for most people: Cloudflare’s Pay Per Crawl experiment has become Pay Per Use, which pays site owners when their content shows up in an AI answer rather than when a page is fetched, with Ceramic.ai and You.com as early partners. It is a genuinely interesting idea for publishers. For a twenty-page business site it is not a revenue plan, and it should not factor into your settings.
The email to send
If somebody else manages your website or your DNS, this is the whole message:
Are we behind Cloudflare? If so, can you check Security > Bots > AI Scrapers and Crawlers and tell me what Search, Agent, and Training are currently set to, and whether the legacy “Block AI bots” option is on? Cloudflare’s defaults change on September 15 and anything blocking AI training will also block Googlebot and Bingbot from that date. I want Search and Agent allowed. Can you also run a live URL inspection in Search Console to confirm Googlebot is not getting 403s?
A clear answer to that in a day means you are fine and can forget about it. A vague answer means somebody should go look.
The uncomfortable thing about this one is how quiet the failure is. An expired certificate throws a full-page red warning at every visitor and you hear about it within the hour. A blocked crawler produces nothing: no error, no alert, no complaint. Just a slow decline in a number you probably only look at monthly, caused by a button somebody pressed two years ago for a good reason that stopped applying.
If you are not sure what sits in front of your site or who configured it, send us a note. Working out what your setup actually is takes about as long as reading this post, and it is worth doing before mid-September rather than after.