POST /crawl · up to 10,000 pages per job · robots.txt and Crawl-delay respected

Crawl API: turn a whole website into Markdown files

Name: Web reader API credits
Brand: WebReaderAPI

Give a start URL and a page limit. The crawler follows the site's sitemap and links within the same domain, reads each page as Markdown and hands you the results page by page. One credit per page.

Get an API key Crawl and batch docs

Last updated: October 22, 2026 · 6 min read

Quick answer

How can I crawl a whole website and get every page as Markdown?

Use a crawl API: you send a start URL and a limit, and it returns each page of the site as Markdown. WebReaderAPI crawls up to 10,000 pages per job within one domain, follows the sitemap, robots.txt and Crawl-delay, and charges one credit per page read, from $0.07/1K pages as of October 2026. Pages refused by robots.txt are free. No KYC; pay in crypto.

Crawl API at a glance

Start a crawl	`POST https://webreaderapi.com/api/v1/crawl` with `url`, `limit`, `max_depth`, `include_paths`, `exclude_paths`
Get the results	`GET https://webreaderapi.com/api/v1/jobs/{job_id}`, paginated with a cursor
Scope	Crawl one website per job, up to 10,000 pages, following its sitemap, its robots.txt and its Crawl-delay
Many separate URLs	`POST https://webreaderapi.com/api/v1/batch`: Up to 100 URLs per batch job
Billing	1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged
Site owners	We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged
Price	$14.25 for 50,000 pages, $50 for 500,000 pages

How does a crawl job work?

A crawl is a background job, because reading hundreds of pages takes longer than one HTTP request should.

Start: POST https://webreaderapi.com/api/v1/crawl with the start URL and your limits. You get a job_id at once.
Discover: the crawler reads the site's robots.txt and sitemap, then follows links within the same domain, breadth first, up to max_depth.
Read: each allowed page is read with the same options as /read: Markdown, main content, rendering when needed.
Collect: poll GET https://webreaderapi.com/api/v1/jobs/{job_id} and page through the results with the cursor while the job runs, or once it is done.

The job stops at your limit, so you never pay for more pages than you asked for.

Python: crawl a documentation site into Markdown files

import os
import pathlib
import re
import time

import requests

API = "https://webreaderapi.com/api/v1"
HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}

job = requests.post(
    API + "/crawl",
    headers=HEADERS,
    json={"url": "https://docs.example.com/", "limit": 500, "max_depth": 4, "include_paths": ["/guide/*"]},
    timeout=60,
).json()

out = pathlib.Path("docs_md")
out.mkdir(exist_ok=True)
cursor, delay = None, 5
while True:
    params = {"cursor": cursor} if cursor else {}
    page = requests.get(API + "/jobs/" + job["job_id"], headers=HEADERS, params=params, timeout=60).json()
    for item in page.get("results", []):
        if "markdown" in item:
            name = re.sub(r"[^a-z0-9]+", "-", item["url"].lower()).strip("-")[:120]
            (out / (name + ".md")).write_text(item["markdown"], encoding="utf-8")
    cursor = page.get("next_cursor")
    if page["status"] == "done" and not cursor:
        break
    if not cursor:
        time.sleep(delay)
        delay = min(delay * 2, 60)
print("credits used:", page.get("credits_used"))

Which pages does the crawler visit, and which does it skip?

It stays inside the domain of the start URL and skips anything the site asks robots to leave alone. In detail:

Same domain only. Links to other domains are listed in the results but not read.
robots.txt first. Pages disallowed for WebReaderAPI or for all robots are skipped and not charged.
Crawl-delay honoured. If the site sets one, the crawler waits that long between two pages, so a large site may take longer.
Your filters. include_paths and exclude_paths take simple patterns such as /blog/*; use them to avoid tag pages, search pages and archives.
Duplicates removed. The same page reached by two URLs, for example with tracking parameters, is read once when the site declares a canonical URL.

Should I use a crawl or a batch?

Use a crawl when you know the site but not the pages; use a batch when you already have the list of URLs. A crawl discovers pages from the sitemap and links. A batch reads exactly the URLs you send, up to 100 per job, across any number of sites.

For a weekly refresh of a knowledge base, a common pattern is one crawl to build the index, then batches of the pages whose lastmod changed in the sitemap. That keeps the weekly cost to the pages that actually moved.

How much does it cost to crawl a site?

One credit per page read, two for pages that needed a browser. A documentation site of 1,500 pages that renders on the server costs 1,500 credits; the same site built as a single-page app may cost up to 3,000.

Three typical jobs, as of October 2026: a 300-page product manual fits in the 10,000 pages pack ($3.75); ten documentation sites of 2,000 pages each fit in 50,000 pages ($14.25); a 400,000-page archive fits in 500,000 pages ($50). Skipped and failed pages are free.

Which pack fits a crawl?

Size the pack on the number of pages, then add a margin for pages that need a browser. Credits never expire, so the rest serves the next crawl.

Test

10,000 pages

$3.75

$0.38/1K pages

Try it on your real URLs
Same API as the biggest pack
Credits never expire

Choose 10,000 pages

Starter

50,000 pages

$14.25

$0.28/1K pages

An AI agent or a RAG prototype
Crawl a documentation site
Credits never expire

Choose 50,000 pages

Build

100,000 pages

$21.75

$0.22/1K pages

Production agents and RAG pipelines
Batch and crawl jobs
Credits never expire

Choose 100,000 pages

Scale

500,000 pages

$50

$0.10/1K pages

Large knowledge bases
Higher concurrency on request
Credits never expire

Choose 500,000 pages

In every plan: Main-content extraction · JavaScript rendering when needed · Failed and refused pages not charged · Credits never expire

Crawl API: common questions

How many pages can one crawl read?

Up to 10,000 pages per job, within the domain of the start URL. For a larger site, split it into several jobs with include_paths, for example one per section.

Does the crawler use the sitemap?

Yes. It reads the sitemap listed in robots.txt or at the usual location, then follows links on the pages it reads. Pages that only the sitemap lists are still read if robots.txt allows them.

Can I crawl a site that blocks AI crawlers?

Not with WebReaderAPI. If robots.txt disallows our crawler or all robots, those pages are skipped and not charged. If the site declares Content-Signal: ai-input=no, we do not read it for you. We never try to get around such choices.

Can I get the results while the crawl is still running?

Yes. GET https://webreaderapi.com/api/v1/jobs/{job_id} returns the pages read so far with a cursor; keep paging until the job is done and no cursor is left.

Can a crawl read more pages than my limit?

No. Credits are counted page by page, and the job stops when it reaches the limit you set. Pages it skips because of robots.txt, and pages that fail, use no credit.