Cloudflare has announced a major change to how AI web crawlers access websites on its platform. The company will soon start blocking mixed-use crawlers by default. This will give website owners more control over how their content is used by AI companies.
Until now, Cloudflare allowed customers to manually block AI crawlers. With the new policy, the company is making that protection the default for many websites.
Cloudflare says the internet has changed significantly, with automated bots now making up the majority of web traffic. Many of these bots are not only indexing websites for search engines but also collecting data to train AI models and power AI assistants. The company believes website owners should be able to decide how their content is used.
Starting September 15, 2026, all new Cloudflare customers and new websites added by existing customers will automatically allow traditional search indexing while blocking AI training and AI agent access on pages that display ads. Existing free users will also move to these default settings unless they choose to opt out before the deadline.
Cloudflare says this change is aimed at encouraging AI companies to separate search crawling from AI training and AI agent activities. If a crawler performs both tasks without giving website owners a choice, it will be blocked on supported pages by default.
The company has also updated its monetization program. Its Pay Per Crawl feature, introduced in 2025, has now been renamed Pay Per Use. Instead of charging AI companies whenever they crawl a webpage, the new system will compensate publishers when their content is actually used in AI-generated answers.
At launch, Cloudflare says Ceramic.AI and You.com are participating in the program. The company hopes more AI firms will join over time.
Although Cloudflare did not directly name Google as the target of the policy, its announcement appears to address how some large companies use a single crawler for multiple purposes. Google’s Googlebot is used for search indexing and also supports several AI-powered features. While Google offers Google-Extended for publishers who want to control AI training, there is currently no separate option for publishers who want their content to appear in AI search features without contributing to AI model training.
