Help agents find and read your content

Good pages are only part of the job. Agents also need a way to find the right pages, read them efficiently, and understand what your product does before they start. Mintlify generates most of this for you. This lesson covers what each piece does, what you can tune, and when it’s worth overriding the defaults.

`llms.txt`: a map of your site

llms.txt is a Markdown file at the root of your site that lists your pages with a short description of each. It works like a sitemap for AI tools: an agent can read it to see what your site covers and pick which pages to fetch, without crawling your navigation. Mintlify generates llms.txt automatically and keeps it up to date as you publish. The generated file includes:

Your site name as a heading
A summary of your site, taken from the description field in docs.json
Every page in navigation order, with the description from its frontmatter
Links to the Markdown version of each page

You can see yours by adding /llms.txt to your site’s URL.

Improve the generated file

Because the file is built from your content, you improve it by improving the sources:

Write a specific site description in docs.json. This is the first thing an agent reads. “Acme docs” tells it nothing. “Documentation for Acme, a payment processing API that handles charges, refunds, and webhook notifications” tells it whether it’s in the right place.
Give every page a specific description. Pages without one appear in llms.txt with only a title, and the previous lesson covered why titles alone aren’t enough.
Order your navigation deliberately. The file follows your navigation, so put your most important content first.
Leave out pages agents shouldn’t use. Pages marked hidden: true or noindex: true in their frontmatter are excluded from llms.txt, along with search and AI context. Use this for deprecated pages you haven’t deleted yet, internal notes, and pre-release content.

Here’s the difference a site description and page descriptions make:

Weak llms.txt

# Acme Docs

## Docs
- [Getting Started](https://docs.acme.com/getting-started.md)
- [Advanced configuration](https://docs.acme.com/advanced.md)
- [Webhooks](https://docs.acme.com/webhooks.md)

Strong llms.txt

# Acme

> Documentation for Acme, a payment processing API for e-commerce platforms that handles charges, refunds, and webhook notifications.

## Docs
- [Quickstart](https://docs.acme.com/quickstart.md): Make your first API call and create a test charge
- [Configure rate limits](https://docs.acme.com/rate-limits.md): Set request limits per API key and handle 429 responses
- [Receive webhooks](https://docs.acme.com/webhooks.md): Register an endpoint and verify webhook signatures

Large sites don’t lose pages to size limits. If the file would exceed 100,000 characters, Mintlify splits it into linked index files that agents follow.

When to write a custom file

You can replace the generated file by adding your own llms.txt to the root of your project. A custom file gives you full control over grouping and wording, but you take on its maintenance: it won’t update when you add, move, or remove pages. Most sites are better served by improving the generated file. If you do write one, it must follow the llms.txt format, starting with your site name as a heading.

`llms-full.txt`

Mintlify also generates llms-full.txt, which contains the full Markdown content of every page in one file. Agents that want everything at once can read it instead of fetching pages one by one. It follows the same rules for hidden and noindex pages. A custom llms.txt doesn’t change it.

Markdown pages

As the first lesson covered, Mintlify serves a Markdown version of every page to agents automatically. Links in the Markdown point to other Markdown pages, so an agent that follows them keeps getting clean content. Two features let you shape what agents see in that Markdown. Agent-only and human-only content. The Visibility component shows content to one audience and hides it from the other. Use it when the best instructions differ: a person clicks a button, while an agent calls an API.

<Visibility for="humans">
  Click **Create account** in the top-right corner.
</Visibility>

<Visibility for="agents">
  To create an account, call `POST /v1/accounts` with a valid email address.
</Visibility>

Site-wide agent instructions. Set markdown.instructions in docs.json to add a block of guidance to every Markdown page and to llms.txt. Use it for rules that apply everywhere, like “Always cite the API version” or “Prefer the TypeScript SDK in examples.”

docs.json

"markdown": {
  "instructions": "Always cite the API version. Prefer the TypeScript SDK in examples."
}

Your MCP server

Mintlify generates a search MCP server for your site. When a user connects it to an AI tool like Claude, Cursor, or ChatGPT, the agent can search your content and read full pages directly, instead of relying on web search or what it remembers from training. Your MCP server also gives agents a way to report problems. If an agent finds a page that’s wrong or outdated, it can submit feedback, which shows up in your analytics alongside feedback from readers. Make it easy for users to connect. The contextual menu adds options to every page for copying the page as Markdown, opening it in an AI tool, and installing your MCP server in Cursor or VS Code.

`skill.md`: what agents can do with your product

llms.txt tells agents where your content is. skill.md tells them what they can accomplish with your product: its capabilities, the inputs they need, and the constraints that apply. Mintlify generates a skill.md for public sites by analyzing your documentation, and keeps it current as your content changes. Agents connected to your MCP server can use it automatically. Developers can also add it to their own agents with npx skills add and your site’s URL. Write custom skills when developers commonly ask agents to complete specific workflows with your product, like “create a charge” or “set up webhooks.” Put each skill in its own folder under .mintlify/skills/, with a SKILL.md file that has a name, a description, and step-by-step instructions:

.mintlify/skills/create-charge/SKILL.md

---
name: create-charge
description: Create a payment charge using the Acme API. Use when a user needs to process a payment.
---

## Instructions

1. Retrieve the customer's payment method with `GET /customers/{id}`.
2. Create a charge with `POST /charges`, passing the payment method ID and amount.
3. Return the charge ID to the user.

Mintlify lists custom skills at a discovery endpoint so agents can find them. If your primary goal is helping agents answer questions, the generated skill.md may be all you need. See the Mintlify skill.md guide before you override it.

Structured sources

If your product has an API, keep its OpenAPI specification accurate. A specification gives agents exact parameter types, required fields, allowed values, and response shapes that prose can leave out. Treat the specification as the source of truth for API structure. Use written pages for concepts, workflows, examples, and decisions the specification can’t explain, and avoid defining the same field by hand in several places.

`robots.txt` and AI crawlers

Mintlify’s default robots.txt allows AI crawlers and includes a Content-Signal line that opts your content in to search indexing, AI answers, and AI model training. If you don’t want your docs used for training, add a custom robots.txt with ai-train=no. Be careful about blocking AI crawlers entirely: if they can’t read your docs, AI tools can’t cite them in answers.

Check your setup

Run mint score from the Mintlify CLI to check how ready your site is for agents. It checks your llms.txt and llms-full.txt, Markdown responses, skill.md, MCP server, robots.txt, and more, then reports a score with each check passed or failed. Next up: Give agents your team’s conventions—tell AI writing tools who you write for and which terms to use.

​llms.txt: a map of your site

​Improve the generated file

​When to write a custom file

​llms-full.txt

​Markdown pages

​Your MCP server

​skill.md: what agents can do with your product

​Structured sources

​robots.txt and AI crawlers

​Check your setup