AI Crawl Control Pay Per Crawl - Discovery API, Custom Pricing, Advanced Configuration
Pay Per Crawl now offers a Discovery API for crawlers to programmatically find participating domains, requires payment headers in Web Bot Auth signatures to prevent tampering, introduces error codes for robust error handling, and allows site owners to configure free pages while charging for other content.
Pay Per Crawl is introducing enhancements for both AI crawler operators and site owners, focusing on programmatic discovery, flexible pricing models, and granular configuration control.
For AI crawler operators
Discovery API
A new authenticated API endpoint allows verified crawlers to programmatically discover domains participating in Pay Per Crawl. Crawlers can use this to build optimized crawl queues, cache domain lists, and identify new participating sites. This eliminates the need to discover payable content through trial requests.
The API endpoint is GET https://crawlers-api.ai-audit.cfdata.org/charged_zones and requires Web Bot Auth authentication. Refer to Discover payable content for authentication steps, request parameters, and response schema.
Payment header signature requirement
Payment headers (crawler-exact-price or crawler-max-price) must now be included in the Web Bot Auth signature-input header components. This security enhancement prevents payment header tampering, ensures authenticated payment intent, validates crawler identity with payment commitment, and protects against replay attacks with modified pricing. Crawlers must add their payment header to the list of signed components when constructing the signature-input header.
New crawler-error header
Pay Per Crawl error responses now include a new crawler-error header with 11 specific error codes for programmatic handling. Error response bodies remain unchanged for compatibility. These codes enable robust error handling, automated retry logic, and accurate spending tracking.
For site owners
Configure free pages
Site owners can now offer free access to specific pages like homepages, navigation, or discovery pages while charging for other content. Create a Configuration Rule in Rules > Configuration Rules, set your URI pattern using wildcard, exact, or prefix matching on the URI Full field, and enable the Disable Pay Per Crawl setting. When disabled for a URI pattern, crawler requests pass through without blocking or charging.
Some paths are always free to crawl. These paths are: /robots.txt, /sitemap.xml, /security.txt, /.well-known/security.txt, /crawlers.json.
Get started
AI crawler operators: Discover payable content | Crawl pages
Site owners: Advanced configuration
Source: original entry ↗
More from Cloudflare
Follow Cloudflare to get its new changes in your feed and email digest.
Cloudflare One Client for macOS 2026.8.2100.0
GA release for macOS Cloudflare One Client with improved split tunnel handling that no longer briefly blocks traffic during reconnects, support for non-RFC 1918 local IPv4 networks, faster connects with lower memory use, and numerous reliability fixes across DNS, reauthentication, and client stability.
Cloudflare One Client for Windows 2026.8.2100.0
This GA release improves split tunnel reliability, adds support for non-RFC 1918 local networks, optimizes connection performance with faster reconnections and lower memory usage, and includes numerous bug fixes for DNS, registration, and network handling. The client now features a service recovery mechanism that automatically restarts on system unlock and better handles large hosts files without blocking traffic.
Cloudflare One Client for Linux 2026.8.2100.0
New GA release for Linux with improved split tunnel handling that no longer briefly blocks traffic during reconnects, support for non-RFC 1918 local IPv4 networks, faster tunnel reconnections, and lower memory usage. Includes numerous stability and reliability fixes for DNS, reconnection behavior, and crash issues.