Guide · how-to · October 2026

How to convert a web page to Markdown for an LLM

Copy and paste works for one page. A library works if you run your own crawler. An API works when you want one request per page. Here is how to choose, what good output looks like, and code you can reuse.

Last updated: October 22, 2026 · 5 min read

Quick answer

How do I convert a web page to Markdown for an LLM?

Fetch the page, keep only its main content, then turn headings, lists, tables and links into Markdown and cap the length to your model's budget. You can do it by hand, with open-source libraries, or with one API call: WebReaderAPI returns the Markdown and metadata for one credit per page, from $0.07/1K pages as of October 2026. No KYC; pay in crypto.

Three ways to get Markdown from a web page

By hand	Copy the text from the browser or its reader view; fine for a few pages
Open-source libraries	Readability, Trafilatura, MarkItDown or Crawl4AI inside your own fetcher
A web reader API	`GET https://webreaderapi.com/api/v1/read?url=…` returns Markdown, title, author and date
Length control	`max_chars` on WebReaderAPI, or cut the text yourself after conversion
JavaScript pages	Need a headless browser; WebReaderAPI renders them when needed, for 2 credits

Which method should I use?

Pick by volume and by how much infrastructure you want to run.

Method	Good for	You handle
Copy and paste	A handful of pages, a one-off prompt	Cleaning by hand
Library in your code	Steady volumes, full control	Fetching, browsers, robots.txt, PDFs, retries
Web reader API	Agents and pipelines that read many sites	Only the request and the result

Libraries give you control and no per-page fee. The cost is operational: a headless browser for JavaScript pages, a robots.txt check, character-set and redirect handling, and a separate path for PDFs. An API folds all of that into one HTTP call.

What must good Markdown keep, and what should it drop?

Good Markdown keeps every element that carries meaning and drops everything that only exists for layout or marketing.

Keep headings as #, ## and ###: they tell the model, and your chunker, where each topic starts.
Keep lists and tables as Markdown lists and pipe tables, so steps and numbers stay readable.
Keep links as [text](url): they let an assistant cite and follow sources.
Keep code blocks fenced, with the language when the page shows it.
Drop menus, cookie notices, ads, related-post widgets and footers: they repeat on every page and pollute retrieval.

Check the result on a few of your real pages. Extraction tools disagree most on forums, long tables and pages with several articles.

How much smaller is Markdown than HTML?

Usually much smaller, because page furniture and markup disappear. When Cloudflare launched its Markdown for Agents feature on 12 February 2026, its example page went from 16,180 tokens as HTML to 3,150 tokens as Markdown, about 80% fewer.

Your own ratio depends on the site. Long articles shrink less, in percentage, than short pages wrapped in heavy templates. Measure on your sources before you size a budget.

How do I keep a page within my model's token budget?

Cap the text before it reaches the model, then split what remains along headings. With WebReaderAPI, the max_chars parameter cuts the Markdown at a set length, between 1,000 and 500,000 characters. The word_count field tells you how long the full article was, so an agent can decide whether reading the rest is worth it.

For retrieval, split on headings first and only then cut long sections. A chunk that starts with its own heading keeps its topic; a chunk cut mid-sentence from a long page does not.

Python: read a page as Markdown and split it into heading-based chunks

import os
import re

import requests

API = "https://webreaderapi.com/api/v1/read"
HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}


def page_to_chunks(url, max_chars=60000, chunk_len=1500):
    params = {"url": url, "format": "markdown", "max_chars": max_chars}
    resp = requests.get(API, headers=HEADERS, params=params, timeout=90)
    data = resp.json()
    if resp.status_code != 200:
        print("not read:", data["error"]["code"])
        return []
    chunks = []
    for section in re.split(r"\n(?=#{1,3} )", data["markdown"]):
        section = section.strip()
        while section:
            chunks.append(section[:chunk_len])
            section = section[chunk_len:]
    return [{"text": c, "source": data["final_url"], "title": data["title"]} for c in chunks]


if __name__ == "__main__":
    for chunk in page_to_chunks("https://example.com/guides/compost")[:3]:
        print(chunk["title"], "|", chunk["text"][:80])

What usually goes wrong, and how do I spot it?

Four problems account for most bad conversions:

Empty output. The page builds its content with JavaScript. Look for a low word_count; render the page in a browser, or set render=true.
Cookie or consent text in the result. The extractor kept a banner. Prefer main-content extraction and check the first lines of each result.
Missing tables. Some sites draw tables as images or canvas elements; no text extractor can read those.
A refusal instead of a page. The site disallowed robots in robots.txt or refused the request. Respect it: ask the owner for access, or use an official API or feed if the site offers one.

None of these is fixed by retrying faster. Read the error, then change the method or the source.

Web page to Markdown: common questions

Is Markdown better than plain text for an LLM?

For most pages, yes. Markdown keeps headings, lists, tables and links at a small token cost, so the model can see structure and cite sources. Plain text is slightly shorter but loses that structure.

Can I convert a whole website, not just one page?

Yes. Crawl it and convert each page. WebReaderAPI's crawl API does both in one job: Crawl one website per job, up to 10,000 pages, following its sitemap, its robots.txt and its Crawl-delay.

Can I convert a PDF the same way?

Yes, by sending the PDF's URL to the same endpoint. PDF files up to 20 MB and 100 pages are converted to Markdown, at 1 credit per started block of 10 pages.

Do not convert them through a third-party service unless they are your own and you have exported them yourself. WebReaderAPI reads public pages only and never logs in to accounts.

Is there a free way to do this?

Open-source libraries are free to use and run on your own machines. WebReaderAPI has no free tier; its smallest pack is 10,000 pages for $3.75, paid in crypto with no KYC.

Sources (1)

Introducing Markdown for Agents — Cloudflare (2026-02-12)