Skip to main content
You should routinely test how well agents can answer questions about your product. Asking a few questions is a useful spot check to see if your content is working as expected. Save a small set of questions and ask them when you make significant changes to your content.

​
Start with real user questions

Pull questions from support tickets, search queries, assistant conversations, onboarding calls, and common product tasks. Keep the original wording when you can. Users rarely describe a problem with the exact terms you use to describe your product, so it’s helpful to test their phrasing. Include several kinds of questions. Some types of questions and examples include:
  • Direct lookup: “What is the maximum request size?”
  • Task-based: “How do I rotate an API key without downtime?”
  • Troubleshooting: “Why does my webhook return 401?”
  • Cross-page: “Which authentication method should I use for a browser app?”
  • Unsupported: “Can I deploy this product on a game console?”
  • Ambiguous: “How do I change the limit?”
It might seem unintuitive to test questions that aren’t answered in your documentation. But you want to know whether an agent asks for context or states that your documentation doesn’t have an answer. A confident guess is a bad answer.

​
Define what a good answer contains

For each question, write down:
  • The facts the answer must include
  • The page or pages that support those facts
  • Any important warning or prerequisite
  • What the answer must not claim
  • Whether the agent should answer, ask a follow-up question, or say the docs don’t cover it
Don’t require the agent to answer the question exactly. You aren’t writing a script for it to recite. Focus on the substance of the answer. Would it help your users succeed with the task they’re trying to accomplish?
Example evaluation case

​
Look beyond correct or incorrect

An answer can get the main fact right and still leave someone stuck. Check for:
  • Correctness: Are the claims accurate?
  • Completeness: Does it include the steps, prerequisites, and risks needed to act?
  • Grounding: Do the cited pages support the claims?
  • Restraint: Does the agent avoid inventing an answer when the content is missing?
  • Usefulness: Can the reader take the next step without guessing?
Phrasing isn’t part of this test unless the prose makes the answer hard to understand.

​
Run the same questions before and after changes

Before a major content or configuration change, save the current answers as a baseline. Run the same questions afterward and compare them. If an answer gets worse, trace the problem back to its source:
  1. Did the agent find the right page?
  2. Did the page contain the answer in a focused section?
  3. Did titles, descriptions, or terminology make the page hard to match?
  4. Did outdated or duplicate content compete with the canonical source?
Fix the content or configuration that caused the failure. Don’t contort one sentence to make a single test pass. Your pages still need to work for questions you haven’t anticipated.

​
Keep the question set manageable

Start with 5-10 questions that cover the tasks your users care about most. Add questions if a you discover a new gap in your content and want to test for it. Remove questions that no longer reflect your product. Run the set after major releases, navigation or terminology changes, and updates to llms.txt or agent instructions. Keep reviewing a sample of live questions too. A fixed set helps you catch regressions, but it won’t tell you about every new need. Next up: Protect private and sensitive information — Make sure agents only access the content they should.