Leading publishers and AI companies will be given the opportunity to stop scraping and unauthorized use of original content
Cloudflare, the leading cloud connectivity company, announced it is the first internet infrastructure provider to block AI crawlers from accessing content without permission or compensation by default. Starting today, website owners can decide whether AI crawlers can access their content and how AI companies can use it. AI companies can also provide information about the purposes for which their crawlers are used—such as training, inference, or search—to help website owners decide which crawlers to grant access. Cloudflare’s new default setting is the first step toward a sustainable future for content creators and AI innovators.
For decades, the internet has operated according to a simple principle: search engines index content and redirect users back to the original websites, generating traffic and advertising revenue for websites of all sizes. This cycle rewards creators of high-quality content both financially and in the form of followers, and helps users find new and relevant information. This model is now obsolete. AI crawlers aggregate content such as text, articles, and images to generate answers without redirecting visitors to the original source. This deprives creators of both revenue and the satisfaction of having their content searched for and seen. When the incentives to create original, high-quality content disappear, society as a whole loses, and the future of the internet is in jeopardy.
“If the internet is to survive the age of AI, we must give publishers back control of their content and create a new economic model that works for everyone—creators, consumers, tomorrow’s AI founders, and the future of the internet itself,” said Matthew Prince, co-founder and CEO of Cloudflare. “Unique content has made the internet one of the greatest inventions of the last century, and it’s imperative that creators continue to create it. Until now, AI crawlers have been able to scrape content unhindered. Our goal is to give control back to developers while empowering AI companies to innovate. In doing so, we aim to secure the future of a free and vibrant internet through a new model that works for everyone.”
“Cloudflare’s innovative approach to blocking AI crawlers is a game-changer for publishers and sets new standards for the respectful treatment of online content. When AI companies no longer get everything for free, it opens the door to sustainable innovation based on consent and partnership,” said Roger Lynch, CEO of Condé Nast. “This is an important step toward a fair exploitation model on the internet that protects creators, promotes quality journalism, and holds AI companies accountable.”
“We’ve long emphasized the need for AI platforms to fairly compensate publishers and authors for using our content. Now we can limit access to our content to those AI partners willing to make fair agreements,” emphasizes Neil Vogel, CEO of Dotdash Meredith. “We’re proud to support Cloudflare and look forward to using their tools to protect our content and the open internet.”
Also Read: Transcend Unveils Workflows to Boost Privacy Automation
“As the largest publisher in the United States, publishing USA TODAY and more than 200 regional publications across the USA TODAY Network, it’s critically important to us to prevent unauthorized copying and use of our original content without proper compensation,” said Renn Turiano, Chief Consumer and Product Officer at Gannett Media. “Given the challenges our industry faces, we’re optimistic that Cloudflare’s technology will help combat the theft of valuable intellectual property.”
“Creators and publishers around the world use Pinterest to grow their businesses, reach new audiences, and directly measure their success. As AI fundamentally changes the digital landscape, we are committed to building an internet infrastructure with integrity, where content is used for its intended purpose, so content creators and publishers can thrive,” said Bill Ready, CEO of Pinterest.
“AI companies, search engines, researchers, and anyone else crawling websites are required to provide accurate identification. And every platform on the internet has a say in who can use their content and for what purposes,” says Steve Huffman, co-founder and CEO of Reddit. “The entire ecosystem of developers, platforms, internet users, and crawlers will benefit from more transparent and controlled crawling practices. Cloudflare’s efforts are a significant step in the right direction for everyone involved.”
“We applaud Cloudflare’s commitment to a sustainable digital ecosystem that benefits all stakeholders – consumers who rely on trusted information, publishers who invest in providing that information, and advertisers who support its distribution,” said Vivek Shah, CEO of Ziff Davis.
Introduction of a permission-based internet model
Cloudflare operates one of the world’s largest networks and is involved in managing and protecting 20% of its traffic. Processing trillions of requests every day, the company has the world’s most advanced bot management solutions that precisely distinguish between human users and AI crawlers. In September 2024, Cloudflare introduced the option to block AI crawlers with a single click. Over one million customers have since chosen this option, designed as a decisive yet simple solution to prevent scraping while they develop their AI strategy.
Now Cloudflare is taking the next step by introducing a permission-based model for AI crawlers. From now on, AI companies must obtain a website’s explicit consent before scraping. When registering a domain with Cloudflare, customers can decide upfront whether to allow or block AI crawlers. With this significant change, every new domain will be enabled by default with the desired default setting, eliminating the need for website owners to manually configure their settings to disable this feature. Customers can easily revise their settings and enable crawling at any time if they want to make their content freely accessible.
Source: Businesswire