The hosting platform wants sites to have more control over how AI companies use their content.
Cloudflare has announced plans to automatically block mixed-use web crawlers that index websites for search engines and act as AI agents and trainers at the same time. The company previously offered its customers the optional ability to prevent crawlers from scraping their sites for AI chatbots, but now Cloudflare’s stance is becoming more defensive by default.
“Now that the majority of traffic on the Internet is non-human, we must move forward and act fast so that a sustainable ecosystem can emerge,” Cloudflare CEO and co-founder Matthew Prince shared in a statement. “Cloudflare’s new tools and partnerships provide visibility and commercial opportunities to website owners and benefit AI companies that have bots with clear and transparent intent. We hope our proposed default changes encourage mixed-use crawlers to separate search from agent use and training.”
Web traffic used to reflect whether people were seeing ads or paying for a website’s subscription, but the popularity of AI models that can visit sites on the user’s behalf and get the latest information has overturned that system. Cloudflare’s new approach is an effort to rebalance the relationship in a way that’s fair to both AI companies and anyone running a website.
Starting September 15, 2026, new websites for new customers and existing Cloudflare customers will by default “allow search but block training and agent use for pages with ads.” Mixed-use crawlers that do not give site owners the option to choose whether their site is used for AI will also be blocked by default on pages containing ads. According to the company, users with free accounts will also be switched to these defaults unless they opt-out before the September 15 deadline.
As part of these changes, Cloudflare is also releasing a new version of the paid crawl feature introduced in 2025, which allows websites to block AI web crawlers by default unless companies pay to scrape their content. This feature is now called pay per use, and instead of paying based on whether a webpage is crawled or not, Cloudflare says site owners will be paid when their content appears in AI chatbots’ responses. The announcement only mentions partnerships with Ceramic.AI and You.com, but Cloudflare expects other AI companies to join as options for its customers.
Also trying to make the relationship between websites and AI companies more fair in general techcrunch Notes, Cloudflare also appears to indirectly target Google. The company’s announcement noted that “the largest search engines have access to nearly 2 times more information than leading AI companies as they make it harder for customers to remain discoverable without using AI.” Google’s main crawler, Googlebot, indexes websites for the company’s various search engines and collects information to train Gemini and power AI features like AI Overview and AI Mode. Google lets websites opt-in to a separate crawler called Google-Extended that only crawls websites for traditional search results, but if a publisher wants to be included in AI mode results but doesn’t want their content to train Google’s model, they don’t have a choice. Cloudflare’s new policy is an attempt to force Google and other companies with mixed-use crawlers to change their strategy.
<a href