Skip to main content
The only way to know whether agents can use your content is to ask them. A few questions now and then catch obvious gaps, but they won’t tell you whether last month’s restructure made answers better or worse. For that, you need a small, saved set of questions that you ask the same way every time and compare over time.

​
Start with real user questions

Pull questions from support tickets, search queries, assistant conversations, onboarding calls, and common product tasks. Keep the original wording when you can. Users rarely describe a problem with the exact terms you use to describe your product, so it’s helpful to test their phrasing. Include several kinds of questions. Some types of questions and examples include:
  • Direct lookup: “What is the maximum request size?”
  • Task-based: “How do I rotate an API key without downtime?”
  • Troubleshooting: “Why does my webhook return 401?”
  • Cross-page: “Which authentication method should I use for a browser app?”
  • Unsupported: “Can I deploy this product on a game console?”
  • Ambiguous: “How do I change the limit?”
It might seem counterintuitive to test questions that aren’t answered in your documentation. But you want to know whether an agent asks for context or states that your documentation doesn’t have an answer. A confident guess is a bad answer.

​
Choose which agents to test

Answers depend on the tool and how it reaches your content. The same question can get different answers from:
  • Your site’s Mintlify assistant, which searches your docs directly
  • A general AI tool like ChatGPT or Claude using web search
  • A coding agent connected to your MCP server
Test the tools your users actually use. For each run, record which tool you used and how it accessed your content. Otherwise you can’t tell whether an answer changed because of your content or because you switched tools.

​
Define what a good answer contains

For each question, write down:
  • The facts the answer must include
  • The page or pages that support those facts
  • Any important warning or prerequisite
  • What the answer must not claim
  • Whether the agent should answer, ask a follow-up question, or say the docs don’t cover it
Don’t require the agent to answer the question exactly. You aren’t writing a script for it to recite. Focus on the substance of the answer. Would it help your users succeed with the task they’re trying to accomplish?
Example evaluation case

​
Look beyond correct or incorrect

An answer can get the main fact right and still leave someone stuck. Check for:
  • Correctness: Are the claims accurate?
  • Completeness: Does it include the steps, prerequisites, and risks needed to act?
  • Grounding: Do the cited pages support the claims?
  • Restraint: Does the agent avoid inventing an answer when the content is missing?
  • Usefulness: Can the reader take the next step without guessing?
Phrasing isn’t part of this test unless the prose makes the answer hard to understand.

​
Run the same questions before and after changes

Before a major content or configuration change, save the current answers as a baseline. Run the same questions afterward and compare them. If an answer gets worse, trace the problem back to its source:
  1. Did the agent find the right page?
  2. Did the page contain the answer in a focused section?
  3. Did titles, descriptions, or terminology make the page hard to match?
  4. Did outdated or duplicate content compete with the canonical source?
Fix the content or configuration that caused the failure. Don’t contort one sentence to make a single test pass. Your pages still need to work for questions you haven’t anticipated.

​
Keep the question set manageable

Start with 5-10 questions that cover the tasks your users care about most. Add questions if you discover a new gap in your content and want to test for it. Remove questions that no longer reflect your product. Run the set after major releases, navigation or terminology changes, and updates to llms.txt or agent instructions. Keep reviewing a sample of live questions too. A fixed set helps you catch regressions, but it won’t tell you about every new need. Assistant analytics are a good source: look for topics that come up often or get poor ratings.

​
Check the technical setup too

Content quality and technical setup fail in different ways. Your question set tests the content. To test the setup, run mint score from the Mintlify CLI. It checks whether agents can reach your llms.txt, get Markdown versions of your pages, find your MCP server and skill.md, and get past your robots.txt. Run it when you change any of those settings. Next up: Protect private and sensitive information—make sure agents only access the content they should.