AI Crawl Control: New tools for the agentic Internet
AI Crawl Control now provides content format insights showing what content types AI agents request versus what your origin serves, and has renamed the Robots.txt tab to Directives with Agent Readiness scoring to help you optimize your site for both human and AI agent interactions.
AI Crawl Control now includes new tools to help you prepare your site for the agentic Internet—a web where AI agents are first-class citizens that discover and interact with content differently than human visitors.
Content Format insights
The Metrics tab now includes a Content Format chart showing what content types AI systems request versus what your origin serves. Understanding these patterns helps you optimize content delivery for both human and agent consumption.
Directives tab (formerly Robots.txt)
The Robots.txt tab has been renamed to Directives and now includes a link to check your site's Agent Readiness ↗ score.
Refer to our blog post on preparing for the agentic Internet ↗ for more on why these capabilities matter.
Source: original entry ↗
More from Cloudflare
Follow Cloudflare to get its new changes in your feed and email digest.
Cloudflare One Client for macOS 2026.8.2100.0
GA release for macOS Cloudflare One Client with improved split tunnel handling that no longer briefly blocks traffic during reconnects, support for non-RFC 1918 local IPv4 networks, faster connects with lower memory use, and numerous reliability fixes across DNS, reauthentication, and client stability.
Cloudflare One Client for Windows 2026.8.2100.0
This GA release improves split tunnel reliability, adds support for non-RFC 1918 local networks, optimizes connection performance with faster reconnections and lower memory usage, and includes numerous bug fixes for DNS, registration, and network handling. The client now features a service recovery mechanism that automatically restarts on system unlock and better handles large hosts files without blocking traffic.
Cloudflare One Client for Linux 2026.8.2100.0
New GA release for Linux with improved split tunnel handling that no longer briefly blocks traffic during reconnects, support for non-RFC 1918 local IPv4 networks, faster tunnel reconnections, and lower memory usage. Includes numerous stability and reliability fixes for DNS, reconnection behavior, and crash issues.