What Cloudflare Actually Changed
In mid-2024, Cloudflare quietly flipped a switch that has significant consequences for anyone who cares about being found by AI search engines. The company introduced a one-click setting called "Block AI Scrapers and Crawlers" inside its dashboard, and then went one step further: it made this setting the default for new accounts and, in many cases, applied it across existing zones.
The setting blocks a wide range of bots used by AI companies to crawl the web and build their training datasets and search indices. That includes crawlers operated by OpenAI (GPTBot), Google's AI data collection agents, Anthropic's ClaudeBot, Common Crawl, and dozens of others. When the rule is active, these crawlers receive a 403 Forbidden response. Your content simply does not get indexed by them.
This is not a subtle change. If your site sits behind Cloudflare and you have never actively checked this setting, there is a real chance your content is being blocked from AI search engines right now. The bots that power ChatGPT's Browse with Bing, Perplexity, and Gemini all depend on being able to crawl your pages. Block them, and you disappear from AI-generated answers entirely.
Why Cloudflare Made This the Default
To be fair to Cloudflare, the decision was not arbitrary. There is a legitimate debate about whether AI companies have the right to scrape web content for training purposes without compensation or consent. Many site owners actively want to block these crawlers, especially publishers, news organisations, and creators concerned about their work being used to train models.
Cloudflare positioned this as a privacy and intellectual property protection feature. From their perspective, defaulting to "block" is the more protective stance, and site owners who want to allow AI crawlers can opt back in.
That framing makes sense for content creators worried about data scraping. It makes much less sense for e-commerce brands, service businesses, and SaaS companies who want to be cited in AI answers. For those businesses, being invisible to AI search is not a protection. It is a competitive disadvantage.
How to Check Whether Your Site Is Affected
Before you change anything, you need to confirm the situation on your own account. Here is what to do:
Step 1: Log into your Cloudflare dashboard
Go to cloudflare.com, log in, and select the domain you want to check. Navigate to Security, then Bots. Look for a setting labelled "Block AI Scrapers and Crawlers" or similar. The exact label has varied slightly across dashboard versions, so look for anything referencing AI bots or AI scrapers.
Step 2: Check your WAF custom rules
The toggle is one way the block gets applied, but Cloudflare also surfaces these controls inside Web Application Firewall rules. Go to Security, then WAF, then Custom Rules. Look for any rule that targets known AI crawlers by user agent. These rules can be added automatically or manually, and they may be active even if the bot settings toggle looks switched off.
Step 3: Test directly with a user agent checker
For a more definitive check, use a tool like httpbin.org or a simple curl command to request a page on your site while spoofing the GPTBot user agent string. If you get a 403, the block is active. This takes two minutes and removes all ambiguity.
The AI Crawlers Worth Allowing
Not all AI crawlers are equal, and you may not want to unblock every single one. Here is a practical breakdown of the crawlers that matter most for AI search visibility:
GPTBot (OpenAI)
This is the crawler OpenAI uses to index the web for ChatGPT's real-time browsing and for training data. Blocking GPTBot means ChatGPT cannot directly read your content. If you want to appear in ChatGPT answers, GPTBot needs access.
PerplexityBot
Perplexity is one of the fastest-growing AI search engines. It crawls the web independently and cites sources directly in its answers. Being indexed by PerplexityBot is one of the more direct routes to getting cited. Allow it.
ClaudeBot (Anthropic)
Anthropic's crawler feeds into Claude's knowledge and its web search functionality. Allowing ClaudeBot keeps you in play as Claude-powered products become more widely used.
Google-Extended
This is Google's specific crawler for Gemini and other AI products, separate from Googlebot. Many sites have inadvertently blocked it through robots.txt or Cloudflare rules. If you want to appear in AI Overviews and Gemini answers, Google-Extended needs to be able to reach your pages.
Common Crawl
A non-profit that builds open datasets used by many AI researchers and smaller models. Less critical for immediate search visibility, but worth allowing if you have no strong reason to block it.
How to Unblock AI Crawlers in Cloudflare
Once you know the block is active, here is how to reverse it:
Using the Bot Management toggle
If the block is applied via the Bot Management section, simply toggle it off. This is the simplest fix. Save the change and give it a few hours to propagate. Cloudflare's edge network is fast, but allow some time before testing again.
Editing WAF custom rules
If the block lives in a custom WAF rule, you will need to either delete the rule entirely or edit it to exclude specific AI bot user agents. You can use Cloudflare's expression editor to write rules like: (http.user_agent contains "GPTBot") then Allow. This gives you granular control, so you can allow the crawlers you want while still blocking others.
Using robots.txt alongside Cloudflare
Cloudflare rules take precedence over robots.txt because they operate at the network level before a request ever reaches your server. Even if your robots.txt says Allow: / for GPTBot, a Cloudflare WAF rule will still block it. You need to fix Cloudflare first. Then make sure your robots.txt aligns with what you want.
A clean robots.txt for AI visibility looks something like this:
User-agent: GPTBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /Crawlability Is Just the Starting Point
Getting AI crawlers through your Cloudflare firewall solves one problem. It does not, on its own, get you cited in AI answers. There is a meaningful difference between being crawlable and being understood.
AI search engines do not just index text. They try to understand what a page is about, who the entity behind it is, what products or services are offered, and whether the source is credible. That understanding is aided enormously by structured data, specifically schema markup implemented in JSON-LD format.
Without structured data, a crawler can read your words but cannot reliably extract meaning. With it, you are handing the AI a structured summary: here is the product, here is the price, here is the rating, here is the organisation behind this page. That structured signal increases the likelihood that the AI will understand, trust, and cite your content.
If you have just unblocked your AI crawlers and want to know what to do next, a good starting point is understanding how WebPage schema helps AI search engines understand your site structure. For e-commerce brands specifically, Offer schema is one of the most direct ways to appear in AI shopping recommendations.
At FlinnSchema, this is what we work on every day. Unblocking crawlers is step one. Making sure the content those crawlers find is properly structured, clearly attributed, and optimised for AI understanding is the real work. If you want an honest assessment of where your site currently stands, our free AI visibility audit is a practical place to start.
A Word on the Ongoing Debate
It is worth acknowledging that some businesses genuinely should block AI crawlers. If you are a publisher, a writer, or a creator whose content has direct commercial value that could be replicated by an AI, blocking training crawlers is a reasonable decision. The conversation about fair compensation for web content used in AI training is real and unresolved.
But for most e-commerce and service businesses, the calculation is different. Your goal is not to protect your content from being read. Your goal is to be found, cited, and recommended. Blocking the bots that make that possible works directly against your interests.
Cloudflare's default is set for publishers. Adjust it for your actual business model.
Frequently Asked Questions
Does blocking AI crawlers in Cloudflare affect my Google SEO?
It depends on how the block is configured. Standard Googlebot for organic search is typically not affected by the AI crawler block, because it uses a different user agent. However, Google-Extended, which powers Gemini and AI Overviews, can be blocked. If you want to appear in Google's AI-generated answers, make sure Google-Extended is explicitly allowed.
If I unblock GPTBot, will I start appearing in ChatGPT immediately?
Not immediately. GPTBot needs to recrawl your pages after you unblock it, which can take days or weeks depending on your site's authority and how frequently it is crawled. Once indexed, your content becomes a candidate for inclusion in ChatGPT answers, but appearing consistently also depends on the quality and structure of your content.
Can I allow some AI crawlers and block others through Cloudflare?
Yes. Using Cloudflare's WAF custom rules, you can write expressions based on specific user agent strings. This lets you, for example, allow GPTBot and PerplexityBot while still blocking crawlers you do not want. It requires a bit of rule-writing but is entirely achievable without developer help.
My site is on Shopify or WordPress, not self-hosted. Does this still apply?
If your Shopify or WordPress site has its DNS routed through Cloudflare (which is common for custom domains), then yes, Cloudflare's bot rules apply to your traffic. Check your Cloudflare account and follow the same steps. If your domain does not use Cloudflare at all, this specific issue does not apply, though you should still check your robots.txt for AI crawler permissions.