Quick answer
How do I convert a web page to Markdown in Python?
Call WebReaderAPI with requests.get("https://webreaderapi.com/api/v1/read", params={"url": ...}) and your key in the Authorization header; the JSON answer holds the page's main content in markdown. For many pages, use httpx.AsyncClient with a semaphore or a batch job. One page costs 1 credit, from $0.07/1K pages as of October 2026. No KYC; pay in crypto.
What do I need installed?
Python 3.9 or later and one HTTP client. requests is enough for scripts that read a page at a time; httpx adds async support for many pages in parallel. No package of ours is needed: the API is plain HTTPS and JSON.
Keep your key out of the code. Read it from the WEBREADER_API_KEY environment variable, as every example below does, and give each project its own key so you can revoke one without touching the others.
pip install requests httpx
export WEBREADER_API_KEY="your-key"How do I read one page with requests?
Send url and any options as query parameters, raise on an HTTP error, and read the fields you need. The example keeps the main content, caps it at 20,000 characters for a model, and prints what the call cost.
import os
import requests
API = "https://webreaderapi.com/api/v1/read"
HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}
def read(url, **options):
resp = requests.get(API, headers=HEADERS, params={"url": url, **options}, timeout=75)
if resp.status_code != 200:
error = resp.json()["error"]
raise RuntimeError(error["code"] + ": " + error["message"])
return resp.json()
page = read("https://example.com/docs/getting-started", max_chars=20000)
print(page["title"], page["word_count"], "words", page["credits_used"], "credit(s)")
print(page["markdown"][:500])
How do I read many pages at once with httpx?
Use one httpx.AsyncClient and an asyncio.Semaphore set at or below your concurrency limit. Every account has a limit on how many pages it reads at once; your dashboard shows it, and support can raise it. Errors from one page should not stop the others, so the example collects them next to the results instead of raising.
If you go above the limit, the API answers 429 concurrency_limit; nothing is billed, but the page is not read. Keeping the semaphore below the limit is simpler than retrying.
import asyncio
import os
import httpx
API = "https://webreaderapi.com/api/v1/read"
HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}
async def read_one(client, sem, url):
async with sem:
resp = await client.get(API, params={"url": url, "max_chars": 50000})
body = resp.json()
if resp.status_code == 200:
return url, body, None
return url, None, body["error"]["code"]
async def read_many(urls, limit=10):
sem = asyncio.Semaphore(limit)
async with httpx.AsyncClient(headers=HEADERS, timeout=75) as client:
return await asyncio.gather(*(read_one(client, sem, u) for u in urls))
if __name__ == "__main__":
urls = [f"https://example.com/blog/post-{n}" for n in range(1, 21)]
for url, page, error in asyncio.run(read_many(urls)):
print(url, "->", error or page["title"])
When should I use a batch job instead?
When you have a list of up to 100 URLs and do not need each page the moment it is ready. A batch job runs on our side, so your script holds no connections open and does no retry logic. Start it with POST https://webreaderapi.com/api/v1/batch, then collect results from the job with the cursor loop shown on the batch and crawl page.
For a whole site, start a crawl instead and let it follow the sitemap and links; the same page has a complete loop that writes one .md file per page.
import os
import requests
urls = [f"https://example.com/changelog/{year}" for year in range(2020, 2027)]
resp = requests.post(
"https://webreaderapi.com/api/v1/batch",
headers={"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]},
json={"urls": urls, "options": {"format": "markdown", "main_content": True}},
timeout=30,
)
resp.raise_for_status()
print("job:", resp.json()["job_id"])
How do I split the Markdown into chunks for RAG?
Split on headings first, then on size. Headings mark where topics change, so a chunk that starts at a heading is easier for a retriever to match. Keep the page title and the heading path in each chunk's metadata: they help the model cite the right section.
The function below needs no extra library. It cuts at # to ### headings, then splits any long section on blank lines to stay under a character budget. Store final_url with each chunk, so two addresses that lead to the same page do not create duplicates.
import re
def chunk_markdown(page, max_chars=2000):
"""Split a /read response into chunks with a heading path and the source URL."""
chunks, path, buf = [], [], []
def flush():
text = "\n".join(buf).strip()
while text:
part = text[:max_chars]
cut = part.rfind("\n\n")
if len(text) > max_chars and cut > max_chars // 2:
part = part[:cut]
chunks.append({"text": part.strip(), "section": " > ".join(path), "title": page.get("title"), "source": page["final_url"]})
text = text[len(part):].strip()
buf.clear()
for line in page["markdown"].splitlines():
m = re.match(r"^(#{1,3}) (.+)", line)
if m:
flush()
level = len(m.group(1))
path[:] = path[: level - 1] + [m.group(2).strip()]
buf.append(line)
flush()
return chunks
How much does a Python job cost?
Count pages, not requests. 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged. Reading 2,000 documentation pages without a browser uses about 2,000 credits, which 10,000 pages ($3.75) covers five times over. Prepaid credits never expire: unused pages stay on your account until you use them, so a pipeline that runs once a month uses only what it reads.
Python: common questions
Is there an official Python package?
No. The code on this page is all you need: WebReaderAPI is a plain HTTPS API, and requests or httpx speak it directly.
How do I handle a page that robots.txt refuses?
Check the error code. blocked_by_robots and target_refused are final and free; record them and continue. The errors page has a retry helper that only retries the codes that can change.
Can I use the API from a Jupyter notebook?
Yes. The requests recipe works as is. In a notebook, await read_many(urls) replaces asyncio.run, because the notebook already runs an event loop.
Which timeout should my client use?
A little more than the API's own: the default is 30 seconds and the maximum 60, so 75 seconds on the client side leaves room for a slow site and the response.