AI Crawl Control - Enhanced crawler insights and custom 402 responses

We improved AI crawler management with detailed analytics and introduced custom HTTP 402 responses for blocked crawlers. AI Audit has been renamed to AI Crawl Control and is now generally available.

Enhanced Crawlers tab:

• View total allowed and blocked requests for each AI crawler
• Trend charts show crawler activity over your selected time range per crawler

Custom block responses (paid plans): You can now return HTTP 402 "Payment Required" responses when blocking AI crawlers, enabling direct communication with crawler operators about licensing terms.

For users on paid plans, when blocking AI crawlers you can configure:

• Response code: Choose between 403 Forbidden or 402 Payment Required
• Response body: Add a custom message with your licensing contact information

Example 402 response:

HTTP 402 Payment Required
Date: Mon, 24 Aug 2025 12:56:49 GMT
Content-type: application/json
Server: cloudflare
Cf-Ray: 967e8da599d0c3fa-EWR
Cf-Team: 2902f6db750000c3fa1e2ef400000001

{
  "message": "Please contact the site owner for access."
}