People arrive with context. They have a goal, know what they were doing, and may remember the page they came from. They can skim, follow links, and build a mental model of your product over time.
Agents work differently. They start from the user’s prompt and pull in whatever content they can find for that one request, often a few chunks at a time, without seeing the rest of your site.
This course focuses on agents that read your content in real time to answer questions or complete tasks. Optimizing content for model training is a related but separate concern.
How agents find your content
Depending on the tool, an agent reaches your content in one or more of these ways:- Web search. The agent searches the web, reads the result snippets, and fetches a few of the pages that look relevant. Your page titles and descriptions decide whether yours are among them.
- Direct fetch. The agent requests a URL it already knows, from training data, a user’s message, or a link on another page.
- An index file. The agent reads
llms.txt, a Markdown list of your pages with descriptions, to decide which pages to fetch. - An MCP server. A user connects your documentation’s MCP server to their AI tool. The agent can then search your content and read full pages directly, without going through a web search.
What agents receive
When a person visits your site, the browser renders HTML with navigation, scripts, and styling. Agents don’t need any of that, and it costs them context space they could spend on your content. Mintlify serves agents a Markdown version of each page instead. An agent gets Markdown when it requests a URL ending in.md, asks for Markdown in its request headers, or identifies itself as a known AI agent, like the ones ChatGPT and Claude use to fetch pages. You don’t need to configure anything for this to work.
Components become plain Markdown too. Callouts become labeled blocks, steps become numbered titles, and components without a Markdown equivalent, like tabs, keep their text. That means agents see the content of every tab, not just the first one.
Agents work from fragments
Because an agent may read only one page, or only part of one, it can’t rely on context the way a person can. A sentence like “once you’ve completed the setup above” points to text the agent may never have seen. A page titled “Advanced configuration” gives it little reason to fetch the page in the first place. The next lesson covers how to write pages that hold up under these conditions. The short version: every page should make sense on its own, say clearly what it covers, and use the same terms your users do.Technical problems cause silent failures
Some agent failures have nothing to do with how well you’ve written your content. Technical issues can stop agents from getting it at all:- Very long pages. Agent tools limit how much of a page they’ll read. Some cut off around 50,000 characters, and the limit varies by tool. The agent gets part of the page and doesn’t know the rest exists. Focused, single-topic pages are much less likely to hit a limit.
- Lots of tabbed content. Because agents see every tab, a page with many large tabs can balloon in size. Put content that applies to everyone before the tabs, and keep each tab focused.
- Content rendered in the browser. Most agent fetch tools don’t run JavaScript, so content that only appears after scripts run may be invisible to them. Mintlify renders pages on the server, so this mainly affects custom interactive components.
- Moved content. Agents often have old URLs from training data or earlier conversations. Add redirects whenever you move a page. Moving to a new domain is riskier, because many agent tools don’t follow redirects to a different host, so keep old URLs working on the original domain when you can.