• Cloudflare has launched new tools allowing website owners to block AI web crawlers from scraping their content without permission, aiming to protect intellectual property and support fair compensation.
• The company also introduced a “Pay Per Crawl” feature, enabling creators to charge AI bots for content access and establish a sustainable value exchange on the internet.

As the debate over artificial intelligence and content ownership intensifies, Cloudflare has stepped in with a powerful new solution for website owners: the ability to block AI crawlers from scraping web content without permission or compensation.

Launched today, this feature empowers over a million Cloudflare customers to take back control of their intellectual property in the age of generative AI.

The rise of AI models trained on massive datasets, including content from websites, news outlets, and artists, has led to increasing concern among creators. Many feel their work is being used without credit or compensation, leading to lawsuits and rising tension between tech companies and content providers.

Cloudflare’s new default setting now allows all users to block AI crawlers unless they explicitly opt-in.

Matthew Prince, CEO of Cloudflare, emphasized the importance of restoring a fair balance between creators and technology.

AI crawlers have been scraping content without limits, he said. Our goal is to put the power back in the hands of creators while still helping AI companies innovate.

The new setting aligns with Cloudflare’s broader vision for a more sustainable web economy. Publishers can now decide whether to allow AI crawlers to access their content and specify under what conditions. In return, AI companies must disclose how their crawlers are being used, whether for training, search, or inference.

Cloudflare’s platform handles more than 15% of the internet’s traffic, and this move could significantly reshape how AI models access data. Unlike traditional search engines that direct users back to source sites, AI crawlers often extract content without driving traffic or revenue. This change aims to halt that practice and encourage fairer content exchange.

Adding to this initiative, Cloudflare also unveiled Pay Per Crawl, a private beta tool that lets creators charge AI bots for accessing their content. It utilizes the HTTP 402 response code (“Payment Required”) and allows AI companies to either pay for access or be denied. Cloudflare acts as the payment processor, making this system accessible even for smaller creators without technical or legal teams.

Support from leading media voices has been strong. Roger Lynch, CEO of Condé Nast, praised the development, saying it creates “a fair value exchange on the internet.” The Associated Press and International Center for Journalists have also welcomed the tools as essential for sustaining quality journalism in the AI age.

With the introduction of these tools, Cloudflare is leading the charge toward a more ethical and balanced digital content economy, one where creators have a say, and AI innovation doesn’t come at the cost of creativity.