# llms.txt generator: build the file from a crawl of your own site

> Generate an llms.txt file for your own website from a crawl: a Python script that reads every page as Markdown and lists it with a one-line summary. 2026 data.

- URL: https://webreaderapi.com/guides/llms-txt-generator/
- Last updated: 2026-10-22
- Publisher: WebReaderAPI (https://webreaderapi.com)

An llms.txt file lists a site's key pages, with a line on each, in Markdown that models read easily. This guide gives honest adoption figures, then a script that crawls your site and writes the file for you.

**Quick answer — How do I generate an llms.txt file for my website?**

Crawl your own site, take each page's title and first paragraph, and write them as a Markdown list under your site's name in a file at `/llms.txt`. **WebReaderAPI** does the crawl: up to 10,000 pages per job, as Markdown with metadata, at one credit per page from $0.07/1K pages as of October 2026. The Python script below writes the file. No KYC; pay in crypto.

## llms.txt in brief

| Attribute | Value |
| --- | --- |
| What it is | A proposed Markdown file at the root of a site that lists its key pages for language models |
| Adoption, top 1,000 sites | 9.3% (93 sites) on 18 September 2026, per Rankability |
| Sites with the file | From 4,088 to 36,120 between June 2025 and May 2026, per Originality.ai |
| Measured use | 97% of llms.txt files got zero requests in an Ahrefs study of 137,000 domains |
| Cost to build with WebReaderAPI | One credit per page crawled; 10,000 pages cost $3.75 |

## Is llms.txt worth making?

It is cheap to make, and its measured effect so far is small. Adoption is rising: Rankability found the file on 9.3% of the top 1,000 sites on 18 September 2026, against 0.3% in June 2025, although its method changed between the two counts. Originality.ai counted sites with the file growing from 4,088 to 36,120 between June 2025 and May 2026.

Use is another matter. An Ahrefs study of 137,000 domains, reported by PPC Land on 2 July 2026, found that 97% of llms.txt files received no requests at all, and that AI retrieval bots made 1.1% of the requests the files did get. Make one if your docs are large and you want a clean index; do not expect it to change how assistants cite you on its own.

## What goes into an llms.txt file?

A title, a one-paragraph summary, then grouped lists of links with a short description each. A minimal file looks like this:

- `# Example Docs` on the first line, the name of the site or project.
- `> One sentence on what the site covers.` as a quoted summary.
- `## Guides`, `## API reference` and other sections, one per part of the site.
- `- [Page title](https://example.com/page): one line on what the page answers.` for each page.

Keep descriptions factual and short. Do not write instructions to AI systems in the file; assistants increasingly treat such text as manipulation.

*Python: crawl your own site and write llms.txt*

```python
import os
import time
from collections import defaultdict
from urllib.parse import urlparse

import requests

API = "https://webreaderapi.com/api/v1"
HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}
SITE = "https://docs.example.com/"


def first_paragraph(markdown):
    for block in markdown.split("\n\n"):
        block = block.strip()
        if block and not block.startswith(("#", "-", "|", "!", ">")):
            return " ".join(block.split())[:160]
    return ""


job = requests.post(API + "/crawl", headers=HEADERS, json={"url": SITE, "limit": 300}, timeout=60).json()
pages, cursor, delay = [], None, 5
while True:
    params = {"cursor": cursor} if cursor else {}
    batch = requests.get(API + "/jobs/" + job["job_id"], headers=HEADERS, params=params, timeout=60).json()
    pages += [p for p in batch.get("results", []) if "markdown" in p]
    cursor = batch.get("next_cursor")
    if batch["status"] == "done" and not cursor:
        break
    if not cursor:
        time.sleep(delay)
        delay = min(delay * 2, 60)

sections = defaultdict(list)
for p in pages:
    path = urlparse(p["final_url"]).path.strip("/")
    section = path.split("/")[0].replace("-", " ").title() if path else "Home"
    summary = p.get("description") or first_paragraph(p["markdown"])
    sections[section].append("- [" + p["title"] + "](" + p["final_url"] + "): " + summary)

lines = ["# Example Docs", "", "> Documentation for the Example product.", ""]
for name in sorted(sections):
    lines += ["## " + name, ""] + sorted(sections[name]) + [""]
with open("llms.txt", "w", encoding="utf-8") as fh:
    fh.write("\n".join(lines))
print(len(pages), "pages written to llms.txt")

```

## How does the script work?

It starts one crawl job, collects the results page by page, then groups pages by the first folder of their path. For each page it uses the meta description when the site provides one, and otherwise the first paragraph of the Markdown, cut to 160 characters.

Change `SITE`, the title line and the summary line to match your project, and raise `limit` for a larger site, up to 10,000 pages per job. Review the output before you publish it: section names come from your URL structure and may need a human touch.

## Does my robots.txt need to allow the crawl?

Yes. WebReaderAPI reads only what robots.txt allows for `WebReaderAPI` or for all robots, and it waits any `Crawl-delay` you set. If your own site blocks robots, add a group that allows our token for the time of the crawl; our [crawler page](https://webreaderapi.com/bot/) shows the exact lines.

Pages that robots.txt disallows are skipped and not charged, so a strict file costs you nothing extra; it only leaves those pages out of llms.txt.

## How much does it cost?

One credit per page read, two for pages that need a browser. A documentation site of 300 pages costs 300 credits, or up to 600 if it is a JavaScript application. That fits easily in 10,000 pages for $3.75, as of October 2026, and the credits left over stay on your account: they never expire.

Rebuild the file when your docs change. If you keep the crawl output, you can re-read only the changed pages with a batch job instead of crawling everything again.

## llms.txt: common questions

### Where do I put the llms.txt file?

At the root of your site, so it is served at `/llms.txt`, as plain text or Markdown. Some sites also publish `llms-full.txt` with the full text of the listed pages.

### Does llms.txt replace robots.txt?

No. robots.txt says which paths robots should skip; llms.txt lists pages you want models to find. Keep both. Our [robots.txt guide](https://webreaderapi.com/guides/robots-txt-for-ai-agents/) explains the difference.

### Will llms.txt make ChatGPT or Claude cite my site?

There is no evidence that it does on its own. The Ahrefs study reported in July 2026 found 97% of files received no requests. Clear pages with direct answers and dated facts matter more.

### Can I generate llms.txt for someone else's site?

Only for your own sites or with the owner's agreement. The file speaks for the site, so it belongs to the site owner to publish.

## Sources

1. [llms.txt adoption data](https://www.rankability.com/data/llms-txt-adoption/) — Rankability (2026-09-18)
2. [llms.txt adoption rises 8.8x but 97% of files get zero AI requests](https://ppc.land/llms-txt-adoption-rises-8-8x-but-97-of-files-get-zero-ai-requests/) — PPC Land (2026-07-02)

---

WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto.