Complete Guide to llms.txt: The Key File That Helps AI Correctly Understand Your Website in the Age of AI Search

As ChatGPT, Claude, Perplexity, Google AI Overviews, and other AI tools become new gateways for users to find information, whether a website can be correctly understood and cited by AI has become just as important as traditional SEO. llms.txt was created for this exact purpose. It is a new standard: a file placed in the root directory of a website that uses a concise, structured format to tell large language models (LLMs) what your website is, which pages are important, and where trustworthy information should be retrieved from.

This article provides a complete explanation of what llms.txt is, including its definition, format, use cases, writing methods, and common misconceptions, helping you implement it correctly and avoid mistakes.

llms.txt is a plain text file written in Markdown and placed in the root directory of a website (/llms.txt). Its purpose is to provide brief background information and important page links to AI at “inference time,” helping large language models quickly understand the content of a website.

This standard was proposed by Jeremy Howard of Answer.AI in September 2024, and the official specification website is llmstxt.org. Its core concept is simple: because an LLM’s context window is limited and cannot read an entire website at once, it needs a “curated map” that guides AI toward the most important and authoritative pages.

In one sentence: llms.txt is a website navigation index for AI, not an encyclopedia for AI.

Traditional webpages are filled with navigation bars, ads, JavaScript, CSS, and all kinds of noise. For LLMs, extracting clean and structured information from them can be costly. llms.txt provides three specific values:

• Reducing AI understanding costs: It uses clean Markdown to directly list key information and links, so AI does not need to parse complex HTML.
• Giving you control over your brand narrative: You can personally define “who we are” and “what we offer,” reducing the chance of AI misunderstanding or piecing together incorrect information.
• Improving the citability of important content: You can proactively present the most authoritative pages, such as product information, pricing, FAQs, and policies, to AI, increasing the chance of being cited correctly.

Within the framework of GEO (Generative Engine Optimization), llms.txt is considered one of the foundational elements for making a website “AI-Ready.”

llms.txt deliberately uses Markdown instead of XML or JSON because Markdown is easy for both humans and AI to read. The official specification defines the structure as follows:

• One H1 heading, which is the only required element: The name of the website or project.
• A blockquote summary: One to two sentences describing the core positioning of the website. This is the most important line because AI mainly relies on it to understand your identity.
• Zero or more explanatory paragraphs: Supplementary background information, such as language, subsystems, or important notes.
• Zero or more H2 sections: Each section contains a “file list” of links.

Each item in the link list should be formatted as a Markdown link [Name](URL), optionally followed by a colon and a one-sentence description. For example:

There is also a conventional section called ## Optional, which is used for secondary or skippable links, allowing AI to deprioritize these items when context space is limited.

Many people confuse the two, but their roles are clearly different:

• llms.txt: A concise “navigation index file” that only includes a brand summary and selected links. It is designed to be small and refined, usually kept within a few hundred words. Its job is to point the way.
• llms-full.txt: A complete “full content file” that compiles the actual content of important pages into a single file, making it easier for AI to read all the content at once. Its job is to provide content.

In other words, if you want AI to read the full content, the correct approach is to create a separate llms-full.txt file instead of stuffing large amounts of text into llms.txt. Mixing the two will make the index file bloated and undermine the purpose of “curation.”

All three files are placed in the root directory of a website, but they serve completely different audiences and purposes:

• robots.txt: Tells search engine crawlers which paths can or cannot be crawled. It is about access control.
• sitemap.xml: Lists “all” page URLs on a website for search engines to index comprehensively. Its goal is coverage.
• llms.txt: Provides AI with selected key pages and a brand summary. Its goal is precision and conciseness.

The key difference: sitemap should be “complete,” while llms.txt should be “curated.” llms.txt is not a sitemap, so you should not list every page on the website indiscriminately.

Here are the practical steps:

1. Write the H1 and brand summary: Clearly define who you are and what you offer in one sentence. This is the most important part of the entire file.
2. Identify core pages: Select the pages you most want AI to rely on, such as main products or services, pricing, FAQs, return and exchange policies, and contact information.
3. Organize sections by topic using H2 headings: Group links by product line or function to keep the structure clear.
4. Write a specific description for each link: A clear one-sentence description is far more helpful to AI than only providing the page title.
5. Place secondary links under Optional: Put skippable content, such as regulations or external references, under ## Optional.
6. Place the file in the website root directory: The file must be located at https://yourdomain.com/llms.txt. AI will only look for it in the root directory.

Recommended items to include:

• A clear brand or website positioning summary, which functions like a “system prompt” for AI.
• Links to core products, services, and category pages, along with meaningful one-sentence descriptions.
• Pages that users commonly ask about, such as FAQs, policies, pricing, and contact information.

Recommended items to avoid:

• Duplicate or highly similar pages, as well as login pages, admin pages, and other internal resources.
• Pure marketing slogans and vague content.
• Frequently changing or highly time-sensitive content, such as short-term promotional pages.
• Stuffing full article content into llms.txt. Full content should remain on the actual page or be placed in llms-full.txt.

One guiding principle: curate ruthlessly. llms.txt is a curated briefing, not a data dumping ground.

Misconception 1: If you put definitions, FAQs, and large amounts of content directly into llms.txt, AI will copy them as answers.
The truth is: LLMs will not treat llms.txt as a standard answer sheet to copy from. Definitions should be placed on glossary pages and marked up with Schema.org DefinedTerm. FAQs should be placed on the page itself and marked up with FAQPage structured data (JSON-LD). These are the mechanisms AI and search engines are more likely to extract. llms.txt is only responsible for linking to those pages.

Misconception 2: llms.txt can replace Schema structured data.
The two are complementary and operate at different levels. Schema (JSON-LD) is embedded in the source code of each page for crawlers to read, while llms.txt links to those pages. The correct approach is “Schema on the page, llms.txt as the guide,” rather than stuffing Schema into llms.txt.

Misconception 3: llms.txt is the same as a sitemap.
It is not. A sitemap aims to list all pages; llms.txt aims to curate key pages. Copying the entire sitemap into llms.txt actually goes against the original design intent of llms.txt.

Before implementation, one thing must be considered honestly: llms.txt is still a “proposed standard.” Currently, it has higher adoption among software documentation and developer tool websites, with well-known adopters including Anthropic, Stripe, Mintlify, Cursor, and Zapier.

However, mainstream AI search engines such as Google, OpenAI, and Anthropic have not officially and publicly committed to reading and using llms.txt for retrieval. Therefore, any claim that “implementing llms.txt will guarantee an increase in AI citation rates within a few months” lacks public evidence and should be treated with caution.

A more practical positioning is this: llms.txt is a low-cost, future-oriented piece of infrastructure. It does not guarantee immediate results, but it is inexpensive to implement, has no obvious downside, and aligns with the broader direction of an “AI-Ready/GEO” strategy, making it worth preparing early.

1. Place it in the domain root directory: The final URL must be https://yourdomain.com/llms.txt. Do not place it in a subdirectory, or AI may not find it.
2. Make sure the server does not intercept it: If the homepage has redirect rules, make sure those rules only apply to the root path “/” and do not also redirect /llms.txt.
3. Use canonical URLs: Links should point to the canonical version of each page whenever possible, including or excluding trailing slashes based on the actual site configuration, to avoid unnecessary redirects.
4. Maintain it regularly: When the website is redesigned or when products are added or removed, update llms.txt accordingly to avoid broken links.

llms.txt represents a shift in mindset: websites should not only be friendly to “humans” and “search engine crawlers,” but also to “AI models.” Its role is to serve as a clear map, not a thick encyclopedia. Clearly state your positioning, point to important pages, and let the actual pages and structured data carry the rest of the content.

Correctly implementing llms.txt, together with on-page Schema structured data and solid content, is the complete approach for helping a website establish a strong position in the age of AI search.