# Web page to Markdown in Python

> Convert web pages to Markdown in Python: requests, async httpx with a concurrency cap, batch jobs and RAG chunking by headings. From $0.07/1K pages, no KYC.

- URL: https://webreaderapi.com/docs/python/
- Last updated: 2026-10-22
- Publisher: WebReaderAPI (https://webreaderapi.com)

Four recipes that cover most Python projects: read one page with requests, read many pages concurrently with httpx, hand a list to a batch job, and split the Markdown into chunks for a vector store.

**Quick answer — How do I convert a web page to Markdown in Python?**

Call **WebReaderAPI** with `requests.get("https://webreaderapi.com/api/v1/read", params={"url": ...})` and your key in the `Authorization` header; the JSON answer holds the page's main content in `markdown`. For many pages, use `httpx.AsyncClient` with a semaphore or a batch job. One page costs 1 credit, from $0.07/1K pages as of October 2026. No KYC; pay in crypto.

## What do I need installed?

Python 3.9 or later and one HTTP client. `requests` is enough for scripts that read a page at a time; `httpx` adds async support for many pages in parallel. No package of ours is needed: the API is plain HTTPS and JSON.

Keep your key out of the code. Read it from the `WEBREADER_API_KEY` environment variable, as every example below does, and give each project its own key so you can revoke one without touching the others.

*Install*

```bash
pip install requests httpx
export WEBREADER_API_KEY="your-key"
```

## How do I read one page with requests?

Send `url` and any options as query parameters, raise on an HTTP error, and read the fields you need. The example keeps the main content, caps it at 20,000 characters for a model, and prints what the call cost.

*requests: one page*

```python
import os

import requests

API = "https://webreaderapi.com/api/v1/read"
HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}


def read(url, **options):
    resp = requests.get(API, headers=HEADERS, params={"url": url, **options}, timeout=75)
    if resp.status_code != 200:
        error = resp.json()["error"]
        raise RuntimeError(error["code"] + ": " + error["message"])
    return resp.json()


page = read("https://example.com/docs/getting-started", max_chars=20000)
print(page["title"], page["word_count"], "words", page["credits_used"], "credit(s)")
print(page["markdown"][:500])

```

## How do I read many pages at once with httpx?

Use one `httpx.AsyncClient` and an `asyncio.Semaphore` set at or below your concurrency limit. Every account has a limit on how many pages it reads at once; your dashboard shows it, and support can raise it. Errors from one page should not stop the others, so the example collects them next to the results instead of raising.

If you go above the limit, the API answers `429 concurrency_limit`; nothing is billed, but the page is not read. Keeping the semaphore below the limit is simpler than retrying.

*httpx: up to 10 pages in flight*

```python
import asyncio
import os

import httpx

API = "https://webreaderapi.com/api/v1/read"
HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}


async def read_one(client, sem, url):
    async with sem:
        resp = await client.get(API, params={"url": url, "max_chars": 50000})
    body = resp.json()
    if resp.status_code == 200:
        return url, body, None
    return url, None, body["error"]["code"]


async def read_many(urls, limit=10):
    sem = asyncio.Semaphore(limit)
    async with httpx.AsyncClient(headers=HEADERS, timeout=75) as client:
        return await asyncio.gather(*(read_one(client, sem, u) for u in urls))


if __name__ == "__main__":
    urls = [f"https://example.com/blog/post-{n}" for n in range(1, 21)]
    for url, page, error in asyncio.run(read_many(urls)):
        print(url, "->", error or page["title"])

```

## When should I use a batch job instead?

When you have a list of up to 100 URLs and do not need each page the moment it is ready. A batch job runs on our side, so your script holds no connections open and does no retry logic. Start it with `POST https://webreaderapi.com/api/v1/batch`, then collect results from the job with the cursor loop shown on the [batch and crawl page](https://webreaderapi.com/docs/crawl/).

For a whole site, start a crawl instead and let it follow the sitemap and links; the same page has a complete loop that writes one `.md` file per page.

*Start a batch job*

```python
import os

import requests

urls = [f"https://example.com/changelog/{year}" for year in range(2020, 2027)]
resp = requests.post(
    "https://webreaderapi.com/api/v1/batch",
    headers={"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]},
    json={"urls": urls, "options": {"format": "markdown", "main_content": True}},
    timeout=30,
)
resp.raise_for_status()
print("job:", resp.json()["job_id"])

```

## How do I split the Markdown into chunks for RAG?

Split on headings first, then on size. Headings mark where topics change, so a chunk that starts at a heading is easier for a retriever to match. Keep the page title and the heading path in each chunk's metadata: they help the model cite the right section.

The function below needs no extra library. It cuts at `#` to `###` headings, then splits any long section on blank lines to stay under a character budget. Store `final_url` with each chunk, so two addresses that lead to the same page do not create duplicates.

*Chunk Markdown by headings*

```python
import re


def chunk_markdown(page, max_chars=2000):
    """Split a /read response into chunks with a heading path and the source URL."""
    chunks, path, buf = [], [], []

    def flush():
        text = "\n".join(buf).strip()
        while text:
            part = text[:max_chars]
            cut = part.rfind("\n\n")
            if len(text) > max_chars and cut > max_chars // 2:
                part = part[:cut]
            chunks.append({"text": part.strip(), "section": " > ".join(path), "title": page.get("title"), "source": page["final_url"]})
            text = text[len(part):].strip()
        buf.clear()

    for line in page["markdown"].splitlines():
        m = re.match(r"^(#{1,3}) (.+)", line)
        if m:
            flush()
            level = len(m.group(1))
            path[:] = path[: level - 1] + [m.group(2).strip()]
        buf.append(line)
    flush()
    return chunks

```

## How much does a Python job cost?

Count pages, not requests. 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged. Reading 2,000 documentation pages without a browser uses about 2,000 credits, which 10,000 pages ($3.75) covers five times over. Prepaid credits never expire: unused pages stay on your account until you use them, so a pipeline that runs once a month uses only what it reads.

## Python: common questions

### Is there an official Python package?

No. The code on this page is all you need: WebReaderAPI is a plain HTTPS API, and `requests` or `httpx` speak it directly.

### How do I handle a page that robots.txt refuses?

Check the error code. `blocked_by_robots` and `target_refused` are final and free; record them and continue. The [errors page](https://webreaderapi.com/docs/errors/) has a retry helper that only retries the codes that can change.

### Can I use the API from a Jupyter notebook?

Yes. The `requests` recipe works as is. In a notebook, `await read_many(urls)` replaces `asyncio.run`, because the notebook already runs an event loop.

### Which timeout should my client use?

A little more than the API's own: the default is 30 seconds and the maximum 60, so 75 seconds on the client side leaves room for a slow site and the response.

---

WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto.