Agent-friendly websites
llms.txt vs APIs vs MCP: An Agent-Ready Website Guide
An implementation guide to agent-friendly websites, with a short llms.txt template, API contract, failure tests and a practical decision framework.
In this guide
- The useful starting point
- What to build—and what each layer does
- Choose the lightest useful interface
- Worked example: a service lookup contract
- Test the contract beyond the happy path
- The six-step implementation
- A short llms.txt you can adapt
- If you need an API, start with a read
- A better audit prompt
- Test what happened, not what the assistant says happened
- Avoid the shortcuts that create new problems
- Questions worth answering
- Optional hosting for a working prototype
- Put one improvement to work
- Sources and editorial notes
- Continue with the right practical resource
- Sources and further reading
The useful starting point
The Route Doctor newsletter example describes a short llms.txt file and two public endpoints, with a successful Claude test reported by its author. That is an instructive pattern, not evidence that every agent will discover the file or that the approach improves rankings. The linked project and checker could not be independently verified during this review, so this guide does not repeat their commands as tested installation instructions.
There is a useful idea inside the excitement about agent-ready websites: stop making software infer every capability from a visually complex page. Give it a short map, a clear contract and a result it can check. The mistake is treating those ingredients as a universal certification or a shortcut to search rankings.
The Route Doctor newsletter example describes a short llms.txt file and two public endpoints, with a successful Claude test reported by its author. That is an instructive pattern, not evidence that every agent will discover the file or that the approach improves rankings. The linked project and checker could not be independently verified during this review, so this guide does not repeat their commands as tested installation instructions.
What to build—and what each layer does
| Layer | Useful job | What it does not establish |
|---|---|---|
| Readable, accessible pages | Explain the offer and make the next action understandable. | That an agent will interpret every detail correctly. |
| A short llms.txt | Provide a concise map for clients that use it. | Permission, universal adoption or a search ranking boost. |
| Documented read-only data | Return structured facts for a defined task. | That those facts are fresh or safe without maintenance. |
| Optional MCP integration | Expose capabilities to compatible clients. | Automatic discovery, authorization or reliable outcomes. |
Choose the layer that solves the observed problem. llms.txt proposal and format describes the file convention; OpenAPI specification describes HTTP API contracts; Model Context Protocol architecture explains MCP’s client/server architecture. These are different responsibilities.
Choose the lightest useful interface
If the task is reading information, a well-structured page may suffice. If the task is retrieving fresh, structured data, a documented read-only API can reduce ambiguity. OpenAPI describes the HTTP contract; it does not run the service. MCP connects compatible clients to exposed capabilities and introduces its own implementation and maintenance work. Add it when a real client workflow benefits. None of these automatically grants the agent permissions.
Worked example: a service lookup contract
Use a fixed public service catalog for the first prototype. GET /api/v1/services returns identifiers, names, scope, pricing basis, source URLs and a checked date. A query for an unsupported identifier returns an explicit error, not an empty success message or invented match. Unknown routes return 404; unsupported methods return 405. Keep the example free of customer data and administrative actions.
Test the contract beyond the happy path
Validate both status and content type. A 200 response containing your application’s HTML shell is not a valid JSON API response. Check malformed input, no matching result, rate limiting, stale data and upstream failure. If a request is retried, ensure it does not create a duplicate action. For real deployments, add appropriate authorization, abuse controls, operational logs and ownership; a documentation file is not a security boundary.
The six-step implementation
- Define the task. Choose one safe useful action.
- Write the contract. Inputs, outputs and errors.
- Implement the read. Keep private data out.
- Publish the map. Link real docs and endpoints.
- Test failure paths. Verify status and response.
- Observe real use. Maintain what clients need.
A short llms.txt you can adapt
This is an illustrative template. Replace every example.com URL with an existing public page on your own domain. Check the content, not just the link. If your site has no public API, leave API instructions out. Do not publish placeholder endpoints, private URLs, access tokens or customer information.
# Service Finder Demo
> Discover a documented read-only service, call it with valid input, and explain an error without inventing a result.
## Useful pages
- [Main information](https://example.com/docs/): The current offering and relevant details.
- [Conditions and process](https://example.com/api-reference/): Eligibility, limitations and how the process works.
- [Next step](https://example.com/status/): The official route for a person to continue.
## Boundaries
This example does not support automated purchases or bookings.
Use the linked pages for current facts; ask when information is missing.
Serve the file at your site root when your hosting setup supports it. In a typical static build, that means placing it in the directory copied to the public root. Verify the deployed URL: a local file alone is not a published file. If your platform already generates one, inspect and maintain that version instead of creating conflicting instructions.
If you need an API, start with a read
The following is a proposed contract, not an endpoint installed on this site. Choose fields that answer the task, document their types and meaning, and return only information you intend to make public. Include a source URL and an update signal where useful. A checked date records a review; it does not guarantee the underlying data is current.
GET https://example.com/api/v1/services
Accept: application/json
{
"example": true,
"checked_at": "2026-11-11",
"items": [{
"id": "intro",
"name": "Introductory service",
"pricing_basis": "Request a current quote",
"source_url": "https://example.com/services/",
"available_actions": ["read_details"]
}]
}Define no-result and error behavior as carefully as success. Keep private data behind proper access controls. For future actions that change records or spend money, design authentication, authorization, explicit confirmation and duplicate-action protection separately. OWASP API Security Project provides a useful security review starting point.
A better audit prompt
Use an assistant with browsing or request capabilities. If it cannot access the site, it should report the limitation; a fluent response is not proof of inspection.
Review this public website: [URL]. Task: Discover a documented read-only service, call it with valid input, and explain an error without inventing a result.
Use browsing or request tools if available; if unavailable, say what you cannot verify. Inspect the relevant public pages and /llms.txt. Record the URLs you actually fetched and the facts you found. Do not infer success from a 200 response alone. Separate observed facts, missing information and suggested changes. Do not sign in, submit forms, change data, add to a cart or purchase.
Draft a concise llms.txt using only verified public URLs. Include an API example only if a documented public endpoint exists and its read operation is appropriate to test. Do not invent endpoints. Rank the three most useful fixes by customer impact and effort. Explain how to retest each one.Test what happened, not what the assistant says happened
| Test | Evidence to keep | A passing result |
|---|---|---|
| Retrieve the map | Final URL, status, content type and response body. | The actual text file appears; no login page or application shell. |
| Follow its links | The final destination and the relevant source facts. | Useful pages resolve and match the stated purpose. |
| Try the customer task | Exact prompt, tool capability, answer and cited URLs. | Required facts and next step are correct. |
| Ask an unsupported question | The answer and what the source actually says. | The assistant names the limit rather than inventing a capability. |
| Repeat after a change | Source version, date and all attempted runs. | Any improvement is reproducible within the tested setup. |
Measure for this site’s use case: Contract-valid responses, factual task completion, error recovery, latency and maintenance burden. Tool adoption and rankings require separate evidence.
Run a baseline before editing. Use fresh sessions and keep the prompt and available tools comparable. Include failed attempts in your denominator. Record whether you explicitly told the agent about llms.txt; that is different from observing spontaneous discovery. A scanner is a diagnostic aid, not a certification.
Avoid the shortcuts that create new problems
- Do not replace readable pages with machine-only content. People still need to evaluate the offer.
- Do not treat robots.txt or llms.txt as authentication. Protect private routes with real access controls.
- Do not tell every crawler it may access everything merely to improve a score.
- Do not duplicate prices and policies into several files without an update process.
- Do not claim a successful booking, lead or sale from a generated answer. Verify the actual system result.
For interface implementation, consult Google web.dev: agent-friendly website UX and W3C: labeling form controls. Preserve descriptive controls, meaningful form labels and visible state changes. Agent testing complements human usability and accessibility testing.
Questions worth answering
Will llms.txt improve Google rankings?
No ranking benefit is established by adding this file. Google’s current guidance says it does not use llms.txt for Search visibility. Treat the file as an optional usability aid for clients that consume it; keep ordinary SEO work and task testing separate.
Do I need MCP or Markdown copies of every page?
Not for every project. Start with the smallest interface the task requires. Add Markdown resources or a compatible tool integration when there is a clear consumer and maintenance plan. Check platform support rather than assuming all assistants behave the same way.
Can I use this without coding?
Yes. You can clarify public pages, check links and run the task exercise with your existing website editor and assistant. Publishing a root file depends on the host. A public API requires additional implementation; ask for help if that is outside your skills.
How often should I review it?
Review whenever an offering, policy, route or API contract changes. Assign an owner. Frequently changing prices or stock need stronger freshness controls than a static guide.
Search guidance: Google Search: optimizing for generative AI. Documentation reviewed November 11, 2026. Provider behavior can change; recheck the linked sources when implementing.
Optional hosting for a working prototype
Your existing host may be sufficient. If a custom read-only service needs deployment, you can evaluate Railway through our referral link. Compare current usage charges and operating requirements before choosing it. Referral disclosure: we may receive a benefit for qualifying activity. No paid host is required to complete the planning and local test exercises.
Put one improvement to work
Start with this task: Discover a documented read-only service, call it with valid input, and explain an error without inventing a result. Save the baseline, make one justified improvement and keep the retest evidence. Use the result to choose the next useful change.
Sources and editorial notes
- llms.txt proposal and format
- Google Search: optimizing for generative AI
- Google web.dev: agent-friendly website UX
- W3C: labeling form controls
- OpenAPI specification
- Model Context Protocol architecture
- OWASP API Security Project
This guide combines primary-source documentation with original implementation advice and a fictional worked example. It is not a report of measured client results, guaranteed traffic or earnings. The examples use reserved example.com URLs and must be adapted and verified before deployment.
Sources and further reading
- llms.txt proposal and format
- Google Search: optimizing for generative AI
- Google web.dev: agent-friendly website UX
- W3C: labeling form controls
- OpenAPI specification
- Model Context Protocol architecture
- OWASP API Security Project
Sources checked November 11, 2026. Tool features and plan availability can change; review the linked publisher documentation before connecting an account.
Put one idea to work
Compare a tool on work you understand, measure review effort, and keep the result only when it improves the task.
Plan a useful tool trial →Explore more from AI SuperpowerKeep practicing
Technology field notesGranola, Figma or Canva? Follow one piece of feedback to the right tool
Three product updates make more sense when you start with the file you need to change.
November 18, 2026 · 2 min read
Technology field notesBefore you subscribe to an AI agent, give it a task it should leave unfinished
An acceptance test for buyers who need to know what an agent does when information runs out.
November 18, 2026 · 2 min read