Cloudflare has introduced a new system that aims to give content creators and website owners more control over the AI crawlers that consume their web content daily. Currently in a private beta, the proposed Cloudflare’s pay-per-crawl system would support the ongoing development of next-gen AI platforms while ensuring fair compensation across the board.
Analyzing the pay-per-crawl model
Content creators and website owners have been struggling to come to terms with AI crawlers since their inception. While some choose to make their content freely available for AI training and reference, others opt to block AI crawlers entirely. However, Cloudflare’s new model gives content creators and website owners the option of charging a fee for access.
Cloudflare’s blog post launching pay-per-crawl published today reads, in part: “We believe your choice need not be binary — there should be a third, more nuanced option: You can charge for access. Instead of a blanket block or uncompensated open access, we want to empower content owners to monetize their content at Internet scale.”
Under the new system, content creators and website owners will have three options when dealing with individual AI crawlers:
- Allow: This gives the AI crawler free access to the selected content.
- Block: Use this option to block an AI crawler’s access entirely. When using this option, the company behind the crawler is never even given the option to pay for access.
- Charge: Content creators and website owners can use this option to charge a fee for access.
If an AI crawler’s developer doesn’t have billing set up with Cloudflare, content creators and website owners can still choose to “charge” them. While their current access will effectively be blocked until billing arrangements are made, they’ll be notified of their options.
More must-read AI coverage
- SS&C Intralinks DealCentre AI vs. Datasite: Which platform is built for the future of dealmaking?
- SS&C Intralinks FundCentre AI vs. Juniper Square: Which platform better supports modern private markets fund managers?
- Why Data, Not Models, Determines AI Success
- The Rise of the AI-Native Factory: How Physical AI Is Transforming Manufacturing
Preventing unauthorized access
Cloudflare’s new pay-per-crawl model had to overcome a significant hurdle before it could even enter its private beta phase: spoofing.
With the ability to pick and choose which crawlers have free access, which ones require payment, and which are blocked entirely, the development team at Cloudflare had to prevent hackers from spoofing specific crawlers. They achieved this using web bot authentication. It leverages an Ed25519 key pair and HTTP message signatures to handle each request, ultimately verifying the crawler’s identity and granting or denying access on a case-by-case basis.
Requesting early access to Cloudflare’s beta
If you are a publisher, content creator, AI company, or maybe an AI crawler, Cloudflare provides more details about pay per crawl on its page where you can request to get early access to join the beta.
Ensuring fair compensation
Cloudflare’s new model is an attempt to put the control of online content back into the hands of its original creators and owners. It follows a similar attempt by Creative Commons, which recently unveiled a new framework meant to compensate creators when their work is used for AI training.
It remains to be seen whether these initiatives will be catalysts for more changes benefitting creators.
Read coverage on our sister site eWeek about recent AI copyright lawsuits with creators and Meta, Anthropic, and Microsoft.