Cloudflare Launches Tool To Help Website Owner Monetize Access To Bot AI

Cloudflare launched a new tool that allows website owners to block AI crane bots from accessing their content without permission or compensation, in an effort to help site owners generate revenue from AI companies that want to train their models using web data.

This tool allows site owners to choose whether AI creators can access their content and set prices through the "pay per crane" model, Cloudflare said in a statement on Tuesday, July 1. The approach aims to give content owners greater control over how their data is used and paid.

The increasing use of AI cranes that access content without redirecting users back to their original sources has led to a decrease in traditional search engine traffic, which has been a source of advertising revenue. Now, site owners are looking for new ways to maintain the sustainability of their business.

The Cloudflare initiative has the support of major publishers such as Condé Nast and Associated Press, as well as social media companies such as Reddit and Pinterest.

Stephanie Cohen, Cloudflare's Chief Strategy Officer, stated that the purpose of this tool is to give control to publishers of their content, while creating a sustainable ecosystem for content creators and AI companies.

"The change in traffic pattern is very fast, and something has to change," Cohen said in an interview. "This is just the beginning of a new internet model."

Cloudflare data shows that the ratio between browsing and return visits to the site by Google has now dropped to 18:1, from 6:1 in just the last six months. This decline is believed to have occurred as users are increasingly finding direct answers to Google search results, including through the AI Overviews feature. However, Google's ratio is still much better than other AI companies such as OpenAI which reached 1,500:1.

For decades, search engines have indexed content and redirected users back to the source site, a mechanism that benefits creators. But this model is now disrupted by AI companies that use cranes to collect information without providing traffic back, and instead present this information through chatbots such as ChatGPT, thus harming creators in terms of recognition or income.

Many AI companies are now past the common web standards that publishers use to block their content screening, and claim that they don't violate the law in accessing content for free.

In response, several publishers such as The New York Times have sued AI companies for copyright infringement, while others have opted for a content licensing agreement.

Reddit, for example, sued AI startup Anthropic for allegedly taking comments from Reddit users to train its AI chatbots, but at the same time it also signed a content license agreement with Google.