Making agent-friendly pages with content negotiation

Agents fetch web pages to answer questions, write code, and complete tasks. When an agent requests a page, it gets everything your browser gets, including navigation menus, stylesheets, JavaScript bundles, tracking scripts, and footer links, when all it needs is the structured text on the page. That extra markup confuses the agent, consumes its context window, and makes every request more expensive.

What agents need is a way to request just the text content of a page, without the browser-specific markup. solves this. It's a standard HTTP mechanism where the client specifies its preferred format via the header, and the server returns the matching representation. Many agents already send when fetching pages, and a server that supports content negotiation can return clean, structured text from the same URL that serves HTML to a browser.Content negotiationAcceptAccept: text/markdown

We've updated many of our pages, including our blog and changelog, to support content negotiation. This post walks through how it works, how we implemented it in Next.js, and how to add markdown sitemaps so agents can discover your content.

When an agent fetches a page, it includes an header with its format preferences:Accept

Accept: text/markdown, text/html, */*

By listing first, the agent signals that markdown is preferred over HTML when available. This works better than hosting separate URLs because content negotiation requires no site-specific knowledge. Any agent that sends the right header gets markdown automatically, from any site that supports it.text/markdown.md

Try it yourself:

curl https://vercel.com/blog/self-driving-infrastructure -H "accept: text/markdown"

The implementation has two parts, a rewrite rule in that detects the header and a route handler that returns markdown.next.config.ts

The rewrite checks the header on every incoming request. When it contains , the request gets routed to a dedicated markdown endpoint instead of the default HTML page:Accepttext/markdown

The serves the markdown. Our blog content lives in our CMS as rich text, so the route handler converts it to markdown on the fly. If your content is already authored in markdown, you can serve it directly without a conversion step.route handler

The rich-text-to-markdown conversion preserves the content's structure. Code blocks keep their syntax highlighting markers, headings maintain their hierarchy, and links remain functional. The agent receives the same information as the HTML version, just in a format optimized for token efficiency.

The HTML version of this page is around 500KB. The markdown version is 3KB, a 99.37% reduction in payload size. For agents operating under token limits, smaller payloads mean they can consume more content per request and spend their budget on actual information instead of markup.

We keep the HTML and markdown versions synchronized using and shared slugs, so when content updates in our CMS, both versions refresh simultaneously.Next.js 16 remote cache

Content negotiation also works for sitemaps. XML sitemaps are flat lists of URLs with no titles, no hierarchy, and no indication of what each page is about. A markdown sitemap gives agents a structured table of contents with human-readable titles and parent-child relationships, so they can understand what content exists on your site and navigate to what they need.

We serve markdown sitemaps for both our blog and documentation.

Here's the route handler we use to generate a markdown sitemap for blog posts:

For documentation or other content with nested sections, a recursive renderer preserves the hierarchy so agents understand which pages are children of which topics:

You can see this in action with the .Vercel documentation sitemap

For agents that don't send the header, a tag in your HTML provides an alternative discovery path:Acceptlink<head>

<link rel="alternate" type="text/markdown" title="LLM-friendly version" href="/llms.txt" />

Content negotiation, markdown sitemaps, and tags give agents three ways to find and consume your content efficiently. You can read this page as markdown by sending to any blog or changelog URL on vercel.com, as shown in the curl example above.link rel="alternate"Accept: text/markdown

For an implementation reference, see in our knowledge base.how to serve documentation for agents

Read more

What is content negotiation

How agents request markdown

Implementing content negotiation in Next.js

Performance benefits

Markdown sitemaps for agent discovery

Making your site agent-friendly