Image: qurlson/Envato
Cloudflare’s new pay per crawl system, which also applies to site owners, is currently in a private beta.
Cloudflare has introduced a new system that aims to give content creators and website owners more control over the AI crawlers that consume their web content daily. Currently in a private beta, the proposed Cloudflare’s pay-per-crawl system would support the ongoing development of next-gen AI platforms while ensuring fair compensation across the board.
Content creators and website owners have been struggling to come to terms with AI crawlers since their inception. While some choose to make their content freely available for AI training and reference, others opt to block AI crawlers entirely. However, Cloudflare’s new model gives content creators and website owners the option of charging a fee for access.
Cloudflare’s blog post launching pay-per-crawl published today reads, in part: “We believe your choice need not be binary — there should be a third, more nuanced option: You can charge for access. Instead of a blanket block or uncompensated open access, we want to empower content owners to monetize their content at Internet scale.”
Under the new system, content creators and website owners will have three options when dealing with individual AI crawlers:
If an AI crawler’s developer doesn’t have billing set up with Cloudflare, content creators and website owners can still choose to “charge” them. While their current access will effectively be blocked until billing arrangements are made, they’ll be notified of their options.
Cloudflare’s new pay-per-crawl model had to overcome a significant hurdle before it could even enter its private beta phase: spoofing.
With the ability to pick and choose which crawlers have free access, which ones require payment, and which are blocked entirely, the development team at Cloudflare had to prevent hackers from spoofing specific crawlers. They achieved this using web bot authentication. It leverages an Ed25519 key pair and HTTP message signatures to handle each request, ultimately verifying the crawler’s identity and granting or denying access on a case-by-case basis.
If you are a publisher, content creator, AI company, or maybe an AI crawler, Cloudflare provides more details about pay per crawl on its page where you can request to get early access to join the beta.
Cloudflare’s new model is an attempt to put the control of online content back into the hands of its original creators and owners. It follows a similar attempt by Creative Commons, which recently unveiled a new framework meant to compensate creators when their work is used for AI training.
It remains to be seen whether these initiatives will be catalysts for more changes benefitting creators.
Read coverage on our sister site eWeek about recent AI copyright lawsuits with creators and Meta, Anthropic, and Microsoft.
J.R. Johnivan is a technology writer and computer repair professional with 20 years of experience. His work explores emerging technologies, including next-generation LLMs, their societal impact, and how they can improve professional workflows. He began writing while studying computer networking, eventually combining his passion for technology with a career in content. He also brings expertise in project management, HR, and CRM software, giving him a practical, business-focused perspective on today’s tech landscape.
