llms.txt help agents find content, but they can’t control access to private pages or protect a sensitive value.
Use access controls
If content is private, protect the page or site with authentication and authorization. Test access with the same roles and groups your users have. Removing a link from navigation orllms.txt only makes a page less visible. Anyone—or any agent—with the URL may still be able to fetch it.
Before publishing, classify content:
- Public: Safe for anyone to read and reuse
- User-only: Available to authenticated users
- Role-restricted: Available only to specific groups
- Internal: Not part of the user documentation site
- Secret: Must not appear in documentation, source control, examples, or agent context
Keep examples safe to copy
Use unmistakable placeholders such asYOUR_API_KEY and explain where readers should supply the real value. Never paste a working credential, private hostname, customer identifier, or production response into an example.
Check screenshots too. Browser tabs, account names, email addresses, tokens, and internal URLs can appear outside the area you meant to document.
Add automated secret scanning to the repository when possible, but don’t rely on automation alone. Reviewers should ask whether every example is safe to publish and copy.
Separate public and private sources
When an agent that helps maintain your documentation has access to both public and private repositories, make the boundaries between them explicit.- State which source is canonical for each audience
- Avoid publishing internal instructions that contradict public guidance
- Keep pre-release terminology out of public answers until the feature ships
Test access by audience
Add access cases to your evaluation set.- Can a public user retrieve only public content?
- Can an authenticated user reach the right private guide?
- Does an unsupported request cause the agent to guess from internal context?
- Can an agent retrieve only public content?