LLMs.txt Generator — Build llms.txt for AI Crawlers
Documentation owners, site maintainers, and SEO staff are the ones who usually write this file, because they already know which pages hold the real answers. The moment to build one is when you want AI crawlers and language models to read your site with guidance instead of inferring everything from raw HTML. This generator produces the markdown: a level-one title, a blockquote summary, an optional details paragraph, then any number of ## sections where every entry becomes a link with a title, a URL, and a short description. The finished file gives model-driven crawlers a short map of your most important pages, in the same way robots.txt tells search bots where they are allowed to go.
How the Generator Works
- Enter a title and a summary
Type the site or project name, then write one sentence of summary, which becomes the blockquote under the heading. Add a details paragraph when versioning notes or auth requirements need to travel with the file.
- Add sections and links
Press Add section for every level-two heading you want, give it a name, then drop links inside it. Each link takes a title, an absolute http or https URL, and a description that may stay empty for a bare bullet.
- Check the output and the stats
The generated file rebuilds on every keystroke, and the stats line reports links, sections, and lines. The red banner names the problem when output cannot be produced, such as a malformed URL or a missing title.
- Copy the file and publish it
Press Copy, save the text as llms.txt, and serve it from the root of your domain as plain markdown. Regenerate it whenever navigation changes so the file matches the site.
Before and after: the same four URLs
A flat bullet list gives a crawler four addresses and nothing else, so it has no idea which page answers which question. The rewritten version below carries the same links with headings and descriptions attached.
Helpful pages
- https://docs.acme.example/start
- https://docs.acme.example/api
- https://www.example.com/about
- https://www.example.com/pricing# Acme Docs
> Product documentation for the Acme platform.
## Core docs
- [Getting started](https://docs.acme.example/start): Install the CLI
- [REST API reference](https://docs.acme.example/api): Every endpoint
## Company
- [About](https://www.example.com/about): Team and history
- [Pricing](https://www.example.com/pricing): Plans and limitsChoosing What Makes the Cut
The file stays short, so selection matters more than formatting. A good starting set is the getting-started page, the primary reference, one or two conceptual guides, and the pricing or contact page. Leave out tag archives, paginated listings, and anything behind a login, because a crawler that hits a wall early gives up on the rest of the list. Each description should say in a few words what the reader gets, since that line decides which page a model fetches next. Ten accurate entries beat a file that lists every URL you own.
Use Cases
| Developer | Production | Workflow |
|---|---|---|
| Documenting an API so human readers and crawlers open the same entry points | Publishing llms.txt at the domain root beside robots.txt and sitemap.xml | An agency keeping one llms.txt for each client documentation site |
| Grouping guides, reference pages, and SDK downloads into separate sections | Regenerating after every navigation change and confirming the file serves text/markdown | Reusing a preset as the starting shape, then copying the result into each deploy |
| Rewriting descriptions when a page title changes but the URL stays put | Keeping the file short enough that a crawler reads it in a single request | Reviewing link counts per client before a monthly reporting pass |
Common Mistakes
- Mirroring the full sitemap into the file. Hundreds of URLs turn a reading list into an inventory, and crawlers skip past lists that long.
- Leaving the summary blockquote empty. The generator refuses to emit output without it, and parsers that follow the proposal expect it in place.
- Writing relative URLs. Use absolute http or https addresses so each link resolves wherever the file is read from.
- Keeping retired pages in a section. A link that returns 404 damages confidence in the entries around it.
- Ignoring the file after a redesign. Update it in the same change that updates navigation so the two stay in agreement.
Frequently Asked Questions
What is llms.txt and who proposed it?
llms.txt is a proposed convention for a plain markdown file that a site publishes at the root of its domain. Jeremy Howard described the idea in 2024 as a machine-readable directory pointing at the pages a site most wants language models to read. The file is optional, plain text, and follows a simple shape rather than a formal standard. Because it is still a proposal, support differs between providers and each crawler decides for itself whether to fetch it.
How is llms.txt different from robots.txt?
robots.txt restricts automated agents with Disallow and Allow rules, telling them which URLs they may not fetch. llms.txt points in the other direction: it lists the pages you do want readers to reach, each one carrying a title and a single-line description. The two files answer separate questions and normally sit side by side at the domain root. Use robots.txt to keep crawlers out of private areas, and llms.txt to show language models where the useful documentation starts.
How does llms.txt compare with sitemap.xml?
A sitemap.xml is an XML inventory built for search engines: it lists every indexable URL, carries lastmod values, and can grow to thousands of entries. llms.txt is a short markdown document for language-model crawlers that names only the handful of pages worth reading first. A sitemap records what exists, while an llms.txt file says what deserves attention first. Most teams keep both files and regenerate them from the same navigation source.
Where do I place llms.txt and how should it be served?
Publish the file at the root of your domain, so a plain request for example.com/llms.txt returns it directly. Serve it as text/markdown or text/plain with UTF-8 encoding, behind no login and no redirect. Documentation subdomains each need their own copy, because a crawler that reads the main host will not automatically check the subdomain. A request that receives HTML instead of markdown counts as a missing file.
What structure does an llms.txt file require?
The file opens with a single level-one heading carrying the site or project name. Directly beneath it sits a blockquote line holding a one-sentence summary of what the site covers. After that comes an optional free-text paragraph, then level-two headings for each section, with markdown link bullets underneath. This generator enforces the heading and the blockquote, and it reports an error when a link URL is malformed.
How do AI crawlers use the file?
A crawler fetches llms.txt once, parses the markdown, and treats it as a list of candidate URLs with context attached to each entry. The descriptions matter, because they let a model choose the right page for a question without downloading every candidate first. Fetching behavior varies: some systems read the file while crawling a domain, others only look for it when an agent is asked about that site. No crawler is required to honor the file, so it works beside your sitemap and internal links.
How often should llms.txt be updated?
Regenerate the file whenever you restructure documentation, rename a section, or retire a page that still appears in the list. A workable rhythm is to update it in the same change that alters navigation, so the file and the site never drift apart. Broken entries cost more than a missing file, because a crawler that meets 404 responses twice stops trusting the remaining entries. Re-running a preset and copying the output takes well under a minute.