Quick answer
How do I generate an llms.txt file for my website?
Crawl your own site, take each page's title and first paragraph, and write them as a Markdown list under your site's name in a file at /llms.txt. WebReaderAPI does the crawl: up to 10,000 pages per job, as Markdown with metadata, at one credit per page from $0.07/1K pages as of October 2026. The Python script below writes the file. No KYC; pay in crypto.
llms.txt in brief
| What it is | A proposed Markdown file at the root of a site that lists its key pages for language models |
|---|---|
| Adoption, top 1,000 sites | 9.3% (93 sites) on 18 September 2026, per Rankability |
| Sites with the file | From 4,088 to 36,120 between June 2025 and May 2026, per Originality.ai |
| Measured use | 97% of llms.txt files got zero requests in an Ahrefs study of 137,000 domains |
| Cost to build with WebReaderAPI | One credit per page crawled; 10,000 pages cost $3.75 |
Is llms.txt worth making?
It is cheap to make, and its measured effect so far is small. Adoption is rising: Rankability found the file on 9.3% of the top 1,000 sites on 18 September 2026, against 0.3% in June 2025, although its method changed between the two counts. Originality.ai counted sites with the file growing from 4,088 to 36,120 between June 2025 and May 2026.
Use is another matter. An Ahrefs study of 137,000 domains, reported by PPC Land on 2 July 2026, found that 97% of llms.txt files received no requests at all, and that AI retrieval bots made 1.1% of the requests the files did get. Make one if your docs are large and you want a clean index; do not expect it to change how assistants cite you on its own.
What goes into an llms.txt file?
A title, a one-paragraph summary, then grouped lists of links with a short description each. A minimal file looks like this:
# Example Docson the first line, the name of the site or project.> One sentence on what the site covers.as a quoted summary.## Guides,## API referenceand other sections, one per part of the site.- [Page title](https://example.com/page): one line on what the page answers.for each page.
Keep descriptions factual and short. Do not write instructions to AI systems in the file; assistants increasingly treat such text as manipulation.
import os
import time
from collections import defaultdict
from urllib.parse import urlparse
import requests
API = "https://webreaderapi.com/api/v1"
HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}
SITE = "https://docs.example.com/"
def first_paragraph(markdown):
for block in markdown.split("\n\n"):
block = block.strip()
if block and not block.startswith(("#", "-", "|", "!", ">")):
return " ".join(block.split())[:160]
return ""
job = requests.post(API + "/crawl", headers=HEADERS, json={"url": SITE, "limit": 300}, timeout=60).json()
pages, cursor, delay = [], None, 5
while True:
params = {"cursor": cursor} if cursor else {}
batch = requests.get(API + "/jobs/" + job["job_id"], headers=HEADERS, params=params, timeout=60).json()
pages += [p for p in batch.get("results", []) if "markdown" in p]
cursor = batch.get("next_cursor")
if batch["status"] == "done" and not cursor:
break
if not cursor:
time.sleep(delay)
delay = min(delay * 2, 60)
sections = defaultdict(list)
for p in pages:
path = urlparse(p["final_url"]).path.strip("/")
section = path.split("/")[0].replace("-", " ").title() if path else "Home"
summary = p.get("description") or first_paragraph(p["markdown"])
sections[section].append("- [" + p["title"] + "](" + p["final_url"] + "): " + summary)
lines = ["# Example Docs", "", "> Documentation for the Example product.", ""]
for name in sorted(sections):
lines += ["## " + name, ""] + sorted(sections[name]) + [""]
with open("llms.txt", "w", encoding="utf-8") as fh:
fh.write("\n".join(lines))
print(len(pages), "pages written to llms.txt")
How does the script work?
It starts one crawl job, collects the results page by page, then groups pages by the first folder of their path. For each page it uses the meta description when the site provides one, and otherwise the first paragraph of the Markdown, cut to 160 characters.
Change SITE, the title line and the summary line to match your project, and raise limit for a larger site, up to 10,000 pages per job. Review the output before you publish it: section names come from your URL structure and may need a human touch.
Does my robots.txt need to allow the crawl?
Yes. WebReaderAPI reads only what robots.txt allows for WebReaderAPI or for all robots, and it waits any Crawl-delay you set. If your own site blocks robots, add a group that allows our token for the time of the crawl; our crawler page shows the exact lines.
Pages that robots.txt disallows are skipped and not charged, so a strict file costs you nothing extra; it only leaves those pages out of llms.txt.
How much does it cost?
One credit per page read, two for pages that need a browser. A documentation site of 300 pages costs 300 credits, or up to 600 if it is a JavaScript application. That fits easily in 10,000 pages for $3.75, as of October 2026, and the credits left over stay on your account: they never expire.
Rebuild the file when your docs change. If you keep the crawl output, you can re-read only the changed pages with a batch job instead of crawling everything again.
llms.txt: common questions
Where do I put the llms.txt file?
At the root of your site, so it is served at /llms.txt, as plain text or Markdown. Some sites also publish llms-full.txt with the full text of the listed pages.
Does llms.txt replace robots.txt?
No. robots.txt says which paths robots should skip; llms.txt lists pages you want models to find. Keep both. Our robots.txt guide explains the difference.
Will llms.txt make ChatGPT or Claude cite my site?
There is no evidence that it does on its own. The Ahrefs study reported in July 2026 found 97% of files received no requests. Clear pages with direct answers and dated facts matter more.
Can I generate llms.txt for someone else's site?
Only for your own sites or with the owner's agreement. The file speaks for the site, so it belongs to the site owner to publish.
Sources (2)
- llms.txt adoption data — Rankability (2026-09-18)
- llms.txt adoption rises 8.8x but 97% of files get zero AI requests — PPC Land (2026-07-02)