AI Search adds custom domains, authentication, and advanced crawling

AI Search gets you from a data source to a working search endpoint quickly. This release adds what you need to put that endpoint in front of real users: your own domain, authentication, and one endpoint across several instances. It also adds crawling for sites without a complete sitemap, so your index covers everything you want it to find.

Each of the following is a new option. The previous behavior is still the default, so nothing changes until you change it.

Serve search from your own domain

A public endpoint is a URL that a site or app can query directly, with no authentication in front of it. By default that URL is a generated hostname on search.ai.cloudflare.com. You can now serve the same endpoint from a custom domain, a hostname in a zone that you own:

https://search.example.com/search

Restrict who can query your content

Once your endpoint is on your own domain, you can put Cloudflare Access in front of it. For example, you usually want to give /mcp to specific agents rather than to anyone who finds the URL. Agents authenticate with an Access service token, and people who open the endpoint in a browser sign in through your identity provider.

Search several instances from one URL

A namespace can expose its own public endpoint with /search, /chat/completions, and /mcp paths that fan out across the instances you choose:

curl https://ns-<NAMESPACE_ENDPOINT_ID>.search.ai.cloudflare.com/search \
  --header "Content-Type: application/json" \
  --data '{
    "messages": [{ "content": "How do I configure AI Search?", "role": "user" }],
    "ai_search_options": { "instance_ids": ["docs", "support"] }
  }'

Index your sites without a sitemap

Website data sources support a new discover parse type. It starts at the source URL and collects pages from both your sitemaps and the links it finds while crawling:

curl -X POST "https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/ai-search/instances" \
  -H "Authorization: Bearer <API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "my-ai-search",
    "type": "web-crawler",
    "source": "example.com",
    "source_params": {
      "web_crawler": {
        "parse_type": "discover",
        "discover_options": { "source": "links", "limit": 5000, "depth": 3 }
      }
    }
  }'

To learn more, refer to the AI Search documentation.