By Muhammad Sal
Cloudflare has launched Pay per Crawl, a new marketplace that lets website owners charge AI bots for scraping their content.
This move gives publishers control over AI crawlers and opens a fresh revenue stream as AI companies increasingly rely on web data.
The platform is currently in private beta and allows site owners to set micropayments for each AI crawl, block bots, or allow free access. This is a response to the surge in AI bots scraping websites without permission, causing high server costs and unlicensed content use. Cloudflare protects about 20% of the internet and has been rolling out tools to help publishers manage AI crawlers.
According to reports, several major publishers have already signed on or expressed support for this initiative, including Gannett, Time, BuzzFeed, The Atlantic, The Associated Press, and Stack Overflow. These big names are backing Cloudflare’s push to block AI crawlers by default and promote a permission-based approach to content access.
Cloudflare CEO Matthew Prince said the goal is to put power back in the hands of creators while still allowing AI innovation. He envisions a future where AI agents could even be given budgets to buy access to the best content automatically, creating a fair marketplace between AI firms and publishers.
For Nigerian content creators and digital businesses, this development could help protect local content and create new income opportunities as AI scraping grows worldwide.
Cloudflare’s Pay per Crawl requires both publishers and AI companies to have Cloudflare accounts to set and negotiate crawl prices. The company acts as the intermediary, charging AI firms and paying publishers. The platform currently does not use cryptocurrencies or stablecoins.
This launch comes at a critical time as many publishers face declining traffic due to AI chatbots providing direct answers instead of sending users to original sites. Cloudflare’s marketplace could redefine how content is shared and monetized in the AI era.
We’d love your thoughts on this article! Was the information relevant to you?
