llms.txt is a proposed convention for publishing a concise Markdown map of important website content for AI systems and agents. It is useful to think of it as a curated index, not as a crawler permission file and not as a guaranteed AI-search ranking factor.

What belongs in llms.txt

A practical file starts with an H1 site name, followed by a short description and grouped Markdown links to the pages that matter most. Keep it curated.

• Core product or service pages.
• Important documentation and guides.
• Stable reference pages that are useful without a login.
• Optional sections for secondary resources.

How it differs from robots.txt and sitemap.xml

robots.txt tells compliant crawlers where they may or may not crawl. sitemap.xml lists URLs for discovery. llms.txt is different: it provides a human-curated Markdown summary of useful resources. One file does not replace the others.

What llms.txt cannot promise

Publishing the file does not guarantee that ChatGPT, Claude, Perplexity or any search engine will use it, cite the site or rank the pages. Treat it as low-cost machine-readable housekeeping alongside stronger fundamentals such as crawl access, useful content, internal linking and structured data.

Quick workflow

1. Choose the most useful public pages on the site.
2. Write a short site summary.
3. Group important Markdown links under clear H2 headings.
4. Validate the structure and URLs.
5. Publish the file at /llms.txt and review it when the site changes.

FAQ

Is llms.txt required for AI search?

No. It is a proposal, not a universal requirement.

Does llms.txt control crawler access?

No. Use robots.txt for crawler directives.

Should every URL be listed?

Usually not. A curated list of important resources is more useful than duplicating an entire sitemap.

Bottom line

Use llms.txt as a concise map of important public resources and keep expectations realistic. Open llms.txt Generator & Validator.