# Crawl API: turn a whole website into Markdown files

> Crawl API that turns a whole website into Markdown for RAG: up to 10,000 pages per job, sitemap and robots.txt followed, from $0.07/1K pages. Crypto.

- URL: https://webreaderapi.com/crawl-api/
- Last updated: 2026-10-22
- Publisher: WebReaderAPI (https://webreaderapi.com)

Give a start URL and a page limit. The crawler follows the site's sitemap and links within the same domain, reads each page as Markdown and hands you the results page by page. One credit per page.

**Quick answer — How can I crawl a whole website and get every page as Markdown?**

Use a crawl API: you send a start URL and a limit, and it returns each page of the site as Markdown. **WebReaderAPI** crawls up to 10,000 pages per job within one domain, follows the sitemap, robots.txt and Crawl-delay, and charges one credit per page read, from $0.07/1K pages as of October 2026. Pages refused by robots.txt are free. No KYC; pay in crypto.

## Crawl API at a glance

| Attribute | Value |
| --- | --- |
| Start a crawl | `POST https://webreaderapi.com/api/v1/crawl` with `url`, `limit`, `max_depth`, `include_paths`, `exclude_paths` |
| Get the results | `GET https://webreaderapi.com/api/v1/jobs/{job_id}`, paginated with a cursor |
| Scope | Crawl one website per job, up to 10,000 pages, following its sitemap, its robots.txt and its Crawl-delay |
| Many separate URLs | `POST https://webreaderapi.com/api/v1/batch`: Up to 100 URLs per batch job |
| Billing | 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged |
| Site owners | We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged |
| Price | $14.25 for 50,000 pages, $50 for 500,000 pages |

## How does a crawl job work?

A crawl is a background job, because reading hundreds of pages takes longer than one HTTP request should.

1. **Start:** `POST https://webreaderapi.com/api/v1/crawl` with the start URL and your limits. You get a `job_id` at once.
2. **Discover:** the crawler reads the site's robots.txt and sitemap, then follows links within the same domain, breadth first, up to `max_depth`.
3. **Read:** each allowed page is read with the same options as `/read`: Markdown, main content, rendering when needed.
4. **Collect:** poll `GET https://webreaderapi.com/api/v1/jobs/{job_id}` and page through the results with the cursor while the job runs, or once it is `done`.

The job stops at your `limit`, so you never pay for more pages than you asked for.

*Python: crawl a documentation site into Markdown files*

```python
import os
import pathlib
import re
import time

import requests

API = "https://webreaderapi.com/api/v1"
HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}

job = requests.post(
    API + "/crawl",
    headers=HEADERS,
    json={"url": "https://docs.example.com/", "limit": 500, "max_depth": 4, "include_paths": ["/guide/*"]},
    timeout=60,
).json()

out = pathlib.Path("docs_md")
out.mkdir(exist_ok=True)
cursor, delay = None, 5
while True:
    params = {"cursor": cursor} if cursor else {}
    page = requests.get(API + "/jobs/" + job["job_id"], headers=HEADERS, params=params, timeout=60).json()
    for item in page.get("results", []):
        if "markdown" in item:
            name = re.sub(r"[^a-z0-9]+", "-", item["url"].lower()).strip("-")[:120]
            (out / (name + ".md")).write_text(item["markdown"], encoding="utf-8")
    cursor = page.get("next_cursor")
    if page["status"] == "done" and not cursor:
        break
    if not cursor:
        time.sleep(delay)
        delay = min(delay * 2, 60)
print("credits used:", page.get("credits_used"))

```

## Which pages does the crawler visit, and which does it skip?

It stays inside the domain of the start URL and skips anything the site asks robots to leave alone. In detail:

- **Same domain only.** Links to other domains are listed in the results but not read.
- **robots.txt first.** Pages disallowed for `WebReaderAPI` or for all robots are skipped and not charged.
- **Crawl-delay honoured.** If the site sets one, the crawler waits that long between two pages, so a large site may take longer.
- **Your filters.** `include_paths` and `exclude_paths` take simple patterns such as `/blog/*`; use them to avoid tag pages, search pages and archives.
- **Duplicates removed.** The same page reached by two URLs, for example with tracking parameters, is read once when the site declares a canonical URL.

## Should I use a crawl or a batch?

Use a crawl when you know the site but not the pages; use a batch when you already have the list of URLs. A crawl discovers pages from the sitemap and links. A batch reads exactly the URLs you send, up to 100 per job, across any number of sites.

For a weekly refresh of a knowledge base, a common pattern is one crawl to build the index, then batches of the pages whose `lastmod` changed in the sitemap. That keeps the weekly cost to the pages that actually moved.

## How much does it cost to crawl a site?

One credit per page read, two for pages that needed a browser. A documentation site of 1,500 pages that renders on the server costs 1,500 credits; the same site built as a single-page app may cost up to 3,000.

Three typical jobs, as of October 2026: a 300-page product manual fits in the 10,000 pages pack ($3.75); ten documentation sites of 2,000 pages each fit in 50,000 pages ($14.25); a 400,000-page archive fits in 500,000 pages ($50). Skipped and failed pages are free.

## Which pack fits a crawl?

Size the pack on the number of pages, then add a margin for pages that need a browser. Credits never expire, so the rest serves the next crawl.

| Plan | Quantity | Price | Price per unit |
| --- | --- | --- | --- |
| Test | 10,000 pages | $3.75 | $0.38/1K pages |
| Starter | 50,000 pages | $14.25 | $0.28/1K pages |
| Build | 100,000 pages | $21.75 | $0.22/1K pages |
| Scale | 500,000 pages | $50 | $0.10/1K pages |

In every plan: Main-content extraction · JavaScript rendering when needed · Failed and refused pages not charged · Credits never expire

**Before you pay**

- Minimum top-up $25. Top up once in crypto, then buy any plan from your balance.
- Your balance has no end date and can buy page credits at any time. If a pack costs less than the minimum top-up, the amount you added only to reach that minimum can be sent back on request within 30 days of the payment, in USDT or USDC, as long as you have not spent it. Any other balance is not refunded unless the law says otherwise.
- Credited once your payment is checked on the blockchain. Typical network confirmation time: BTC ≈ 20 min · USDT TRC-20 ≈ 1 min · USDT ERC-20 ≈ 5 min · USDC ERC-20 ≈ 5 min · ETH ≈ 5 min · SOL ≈ 1 min · LTC ≈ 15 min

## Crawl API: common questions

### How many pages can one crawl read?

Up to 10,000 pages per job, within the domain of the start URL. For a larger site, split it into several jobs with `include_paths`, for example one per section.

### Does the crawler use the sitemap?

Yes. It reads the sitemap listed in robots.txt or at the usual location, then follows links on the pages it reads. Pages that only the sitemap lists are still read if robots.txt allows them.

### Can I crawl a site that blocks AI crawlers?

Not with WebReaderAPI. If robots.txt disallows our crawler or all robots, those pages are skipped and not charged. If the site declares `Content-Signal: ai-input=no`, we do not read it for you. We never try to get around such choices.

### Can I get the results while the crawl is still running?

Yes. `GET https://webreaderapi.com/api/v1/jobs/{job_id}` returns the pages read so far with a cursor; keep paging until the job is `done` and no cursor is left.

### Can a crawl read more pages than my limit?

No. Credits are counted page by page, and the job stops when it reaches the `limit` you set. Pages it skips because of robots.txt, and pages that fail, use no credit.

---

WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto.