No KYC — no ID, no selfie, no documents. Pay in crypto: BTC, USDT, USDC, ETH, SOL, LTC
Language: English
Get an API key

Docs · jobs

Batch and crawl API: read many URLs or a whole site

Two endpoints start background jobs: /batch for a list of up to 100 URLs, /crawl for one website. Both return a job ID; you collect the pages from /jobs/{job_id}, a page of results at a time.

Last updated: · 7 min read

Quick answer

How do I crawl a website to Markdown with WebReaderAPI?

Send the start URL to POST https://webreaderapi.com/api/v1/crawl with a page limit. WebReaderAPI reads the site in the background, same domain only, following its sitemap, robots.txt and Crawl-delay, up to 10,000 pages per job. Collect the Markdown pages from the job, a page of results at a time. Each page read costs 1 credit (2 with a browser), from $0.07/1K pages as of October 2026. No KYC; pay in crypto.

Jobs at a glance

BatchUp to 100 URLs per batch job: POST https://webreaderapi.com/api/v1/batch
CrawlCrawl one website per job, up to 10,000 pages, following its sitemap, its robots.txt and its Crawl-delay: POST https://webreaderapi.com/api/v1/crawl
ResultsGET https://webreaderapi.com/api/v1/jobs/{job_id}, then ?cursor=… for the next page of results
BillingEach page read is billed like a /read call; pages skipped by robots.txt and failed pages cost nothing
Scope of a crawlOne domain; links to other domains are listed but not read
Cancel a jobNot available in v1: set limit to the most pages you are ready to pay for

Start a batch job

POST/api/v1/batch

Reads up to 100 URLs in the background with the same options. Options use the names and values of /read. The answer is the job ID; values are illustrative.

Authentication: API key in the Authorization: Bearer API_KEY header.

Request body

FieldTypeRequiredDescription
urlsarray of stringsYes1 to 100 http or https URLs. Duplicates are read once.
optionsobjectNoAny /read parameter except url: format, main_content, render, include_links, include_images, country, max_chars, timeout.
Example: three articles as Markdown
{
  "urls": [
    "https://example.com/blog/post-1",
    "https://example.com/blog/post-2",
    "https://example.org/news/launch"
  ],
  "options": {
    "format": "markdown",
    "max_chars": 50000
  }
}

Response 202 Accepted

{
  "job_id": "job_5c1e8a90d2"
}

Code examples

curl
curl -s "https://webreaderapi.com/api/v1/batch" \
  -H "Authorization: Bearer $WEBREADER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com/blog/post-1", "https://example.com/blog/post-2"], "options": {"format": "markdown"}}'
Python
import os
import requests

resp = requests.post(
    "https://webreaderapi.com/api/v1/batch",
    headers={"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]},
    json={"urls": ["https://example.com/blog/post-1", "https://example.com/blog/post-2"], "options": {"format": "markdown"}},
    timeout=30,
)
print(resp.json()["job_id"])
Node.js
const res = await fetch('https://webreaderapi.com/api/v1/batch', {
  method: 'POST',
  headers: { Authorization: `Bearer ${process.env.WEBREADER_API_KEY}`, 'Content-Type': 'application/json' },
  body: JSON.stringify({ urls: ['https://example.com/blog/post-1', 'https://example.com/blog/post-2'], options: { format: 'markdown' } }),
});
console.log((await res.json()).job_id);

Start a crawl job

POST/api/v1/crawl

Reads one website from a start URL, breadth first. The crawler stays on the start URL's domain, uses the sitemap named in robots.txt or found at /sitemap.xml, and respects robots.txt and Crawl-delay. Values are illustrative.

Authentication: API key in the Authorization: Bearer API_KEY header.

Request body

FieldTypeRequiredDescription
urlstringYesThe start page, such as the home page of a documentation site.
limitintegerNoMost pages to read, up to 10,000. The job stops when it reaches this number.
max_depthintegerNoMost link steps away from the start URL.
include_pathsarray of stringsNoSimple path patterns to keep, such as /docs/*.
exclude_pathsarray of stringsNoSimple path patterns to skip, such as /blog/tag/*.
optionsobjectNoAny /read parameter except url, applied to every page.
Example: the docs section of a site, 500 pages at most
{
  "url": "https://docs.example.com/",
  "limit": 500,
  "max_depth": 3,
  "include_paths": [
    "/docs/*"
  ],
  "exclude_paths": [
    "/docs/archive/*"
  ],
  "options": {
    "format": "markdown"
  }
}

Response 202 Accepted

{
  "job_id": "job_8b27f41c0e"
}

Code examples

curl
curl -s "https://webreaderapi.com/api/v1/crawl" \
  -H "Authorization: Bearer $WEBREADER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://docs.example.com/", "limit": 500, "include_paths": ["/docs/*"]}'
Python
import os
import requests

resp = requests.post(
    "https://webreaderapi.com/api/v1/crawl",
    headers={"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]},
    json={"url": "https://docs.example.com/", "limit": 500, "include_paths": ["/docs/*"]},
    timeout=30,
)
print(resp.json()["job_id"])
Node.js
const res = await fetch('https://webreaderapi.com/api/v1/crawl', {
  method: 'POST',
  headers: { Authorization: `Bearer ${process.env.WEBREADER_API_KEY}`, 'Content-Type': 'application/json' },
  body: JSON.stringify({ url: 'https://docs.example.com/', limit: 500, include_paths: ['/docs/*'] }),
});
console.log((await res.json()).job_id);

Get job status and results

GET/api/v1/jobs/{job_id}

Returns the job's status and the next results that are ready. Each item of results is either a full /read response or {url, error} for a page that could not be read. When next_cursor is present, call again with cursor set to it; when it is absent, no more results are ready yet. Values are illustrative.

Authentication: API key in the Authorization: Bearer API_KEY header.

Parameters

NameInTypeRequiredDescription
job_idpathstringYesThe ID returned by /batch or /crawl.
cursorquerystringNoThe next_cursor value from the previous call, to get the following results.

Response 200

{
  "job_id": "job_8b27f41c0e",
  "status": "running",
  "results": [
    {
      "url": "https://docs.example.com/docs/install",
      "final_url": "https://docs.example.com/docs/install",
      "status_code": 200,
      "rendered": false,
      "credits_used": 1,
      "title": "Install",
      "markdown": "# Install\n\nRun the installer...",
      "word_count": 418
    },
    {
      "url": "https://docs.example.com/docs/private",
      "error": {
        "code": "blocked_by_robots",
        "message": "robots.txt disallows this page for WebReaderAPI."
      }
    }
  ],
  "next_cursor": "c_000002",
  "credits_used": 1
}

Code examples

curl
curl -sG "https://webreaderapi.com/api/v1/jobs/job_8b27f41c0e" \
  -H "Authorization: Bearer $WEBREADER_API_KEY" \
  -d cursor=c_000002
Python
import os
import requests

resp = requests.get(
    "https://webreaderapi.com/api/v1/jobs/job_8b27f41c0e",
    headers={"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]},
    params={"cursor": "c_000002"},
    timeout=30,
)
job = resp.json()
print(job["status"], len(job["results"]), job.get("next_cursor"))
Node.js
const res = await fetch('https://webreaderapi.com/api/v1/jobs/job_8b27f41c0e?cursor=c_000002', {
  headers: { Authorization: `Bearer ${process.env.WEBREADER_API_KEY}` },
});
const job = await res.json();
console.log(job.status, job.results.length, job.next_cursor);

How do I wait for a job to finish?

Poll /jobs/{job_id} and keep the cursor. Process the results you get, then call again with cursor set to next_cursor. When next_cursor is absent and status is done, you have everything; when it is absent and the job is still running, wait and ask again with the same cursor.

Start polling every 2 seconds and double the wait up to 30 seconds while nothing new arrives. A crawl that respects a site's Crawl-delay can take a while, so a long wait is normal and costs nothing.

Python: poll a job and save every page as a .md file
import os
import pathlib
import re
import time

import requests

JOBS = "https://webreaderapi.com/api/v1/jobs"
HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}


def save_job(job_id, folder="pages"):
    out = pathlib.Path(folder)
    out.mkdir(exist_ok=True)
    cursor, wait, saved = None, 2, 0
    while True:
        params = {"cursor": cursor} if cursor else {}
        job = requests.get(f"{JOBS}/{job_id}", headers=HEADERS, params=params, timeout=30).json()
        for item in job["results"]:
            if "error" in item:
                print("skipped", item["url"], item["error"]["code"])
                continue
            name = re.sub(r"[^a-z0-9]+", "-", item["final_url"].lower()).strip("-")[:120]
            (out / f"{name}.md").write_text(item["markdown"], encoding="utf-8")
            saved += 1
        if job.get("next_cursor"):
            cursor, wait = job["next_cursor"], 2
            continue
        if job["status"] == "done":
            return saved
        time.sleep(wait)
        wait = min(wait * 2, 30)


if __name__ == "__main__":
    print(save_job("job_8b27f41c0e"), "pages saved")

Which pages does a crawl read?

A crawl starts from your URL and moves outward one link step at a time, breadth first, up to max_depth. It also reads the site's sitemap, from the address given in robots.txt or from /sitemap.xml, to find pages that few links point to.

  • Same domain only. Links to other domains are listed in the results when you ask for include_links, but they are not read.
  • Path filters. include_paths and exclude_paths take simple patterns such as /blog/*.
  • One copy per page. When a site declares a canonical URL, a page reached through several addresses is read once.
  • A hard stop. The job ends when it has read limit pages.

How does a crawl respect the site?

Every request carries the User-Agent WebReaderAPI/1.0 (+https://webreaderapi.com/bot/), and the crawler follows the robots.txt rules for WebReaderAPI, or for all robots when none name it. Disallowed pages are skipped and appear in results with the error blocked_by_robots; they cost nothing.

When robots.txt sets a Crawl-delay, the crawler waits that long between two requests to the site. If a site declares Content-Signal: ai-input=no in its robots.txt, we do not read its pages for you. If a site refuses our request, we return its answer: we never solve CAPTCHAs, log in to accounts, or try to get around a block or a paywall.

How much does a crawl cost?

Each page read costs what a /read call would: 1 credit, 2 if the page needed a browser, plus 10 with country. Skipped and failed pages are free. The job's credits_used adds up what has been billed so far.

A 500-page documentation site read without a browser uses about 500 credits, which the 10,000 pages pack ($3.75) covers many times over. Since a job cannot be cancelled in v1, limit is your spending cap: set it to the most pages you are ready to pay for.

Batch and crawl: common questions

Should I use /batch or many /read calls?

Use /batch when you already have the URLs and do not need each answer at once. It saves you from managing concurrency and retries. Use /read when an agent needs one page now.

How long are job results kept?

Collect them as soon as they are ready. Results wait until you fetch them; once delivered, no copy of the page content is kept, so store what you need on your side.

No. A crawl reads one domain. Other domains appear as links when you ask for them, and you can send those URLs to /batch if you want them read.

Can I stop a crawl that has started?

Not in v1. The job stops at limit pages, so choose a limit that matches your budget. Pages already read stay billed; pages never read cost nothing.

From $0.07/1K pages · no KYC

Get an API key