# Batch and crawl API: read many URLs or a whole site > Batch and crawl API reference: read up to 100 URLs per job or a whole site up to 10,000 pages, robots.txt and Crawl-delay respected. No KYC, pay in crypto. - URL: https://webreaderapi.com/docs/crawl/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Two endpoints start background jobs: `/batch` for a list of up to 100 URLs, `/crawl` for one website. Both return a job ID; you collect the pages from `/jobs/{job_id}`, a page of results at a time. **Quick answer — How do I crawl a website to Markdown with WebReaderAPI?** Send the start URL to `POST https://webreaderapi.com/api/v1/crawl` with a page `limit`. **WebReaderAPI** reads the site in the background, same domain only, following its sitemap, robots.txt and Crawl-delay, up to 10,000 pages per job. Collect the Markdown pages from the job, a page of results at a time. Each page read costs 1 credit (2 with a browser), from $0.07/1K pages as of October 2026. No KYC; pay in crypto. ## Jobs at a glance | Attribute | Value | | --- | --- | | Batch | Up to 100 URLs per batch job: `POST https://webreaderapi.com/api/v1/batch` | | Crawl | Crawl one website per job, up to 10,000 pages, following its sitemap, its robots.txt and its Crawl-delay: `POST https://webreaderapi.com/api/v1/crawl` | | Results | `GET https://webreaderapi.com/api/v1/jobs/{job_id}`, then `?cursor=…` for the next page of results | | Billing | Each page read is billed like a `/read` call; pages skipped by robots.txt and failed pages cost nothing | | Scope of a crawl | One domain; links to other domains are listed but not read | | Cancel a job | Not available in v1: set `limit` to the most pages you are ready to pay for | ## Start a batch job `POST /api/v1/batch` Reads up to 100 URLs in the background with the same options. Options use the names and values of [/read](https://webreaderapi.com/docs/read/). The answer is the job ID; values are illustrative. Authentication: API key in the `Authorization: Bearer API_KEY` header. **Request body** | Field | Type | Required | Description | | --- | --- | --- | --- | | `urls` | array of strings | Yes | 1 to 100 http or https URLs. Duplicates are read once. | | `options` | object | No | Any `/read` parameter except `url`: `format`, `main_content`, `render`, `include_links`, `include_images`, `country`, `max_chars`, `timeout`. | *Example: three articles as Markdown* ```json { "urls": [ "https://example.com/blog/post-1", "https://example.com/blog/post-2", "https://example.org/news/launch" ], "options": { "format": "markdown", "max_chars": 50000 } } ``` **Response** `202 Accepted` ```json { "job_id": "job_5c1e8a90d2" } ``` *curl* ```bash curl -s "https://webreaderapi.com/api/v1/batch" \ -H "Authorization: Bearer $WEBREADER_API_KEY" \ -H "Content-Type: application/json" \ -d '{"urls": ["https://example.com/blog/post-1", "https://example.com/blog/post-2"], "options": {"format": "markdown"}}' ``` *Python* ```python import os import requests resp = requests.post( "https://webreaderapi.com/api/v1/batch", headers={"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}, json={"urls": ["https://example.com/blog/post-1", "https://example.com/blog/post-2"], "options": {"format": "markdown"}}, timeout=30, ) print(resp.json()["job_id"]) ``` *Node.js* ```javascript const res = await fetch('https://webreaderapi.com/api/v1/batch', { method: 'POST', headers: { Authorization: `Bearer ${process.env.WEBREADER_API_KEY}`, 'Content-Type': 'application/json' }, body: JSON.stringify({ urls: ['https://example.com/blog/post-1', 'https://example.com/blog/post-2'], options: { format: 'markdown' } }), }); console.log((await res.json()).job_id); ``` ## Start a crawl job `POST /api/v1/crawl` Reads one website from a start URL, breadth first. The crawler stays on the start URL's domain, uses the sitemap named in robots.txt or found at `/sitemap.xml`, and respects robots.txt and Crawl-delay. Values are illustrative. Authentication: API key in the `Authorization: Bearer API_KEY` header. **Request body** | Field | Type | Required | Description | | --- | --- | --- | --- | | `url` | string | Yes | The start page, such as the home page of a documentation site. | | `limit` | integer | No | Most pages to read, up to 10,000. The job stops when it reaches this number. | | `max_depth` | integer | No | Most link steps away from the start URL. | | `include_paths` | array of strings | No | Simple path patterns to keep, such as `/docs/*`. | | `exclude_paths` | array of strings | No | Simple path patterns to skip, such as `/blog/tag/*`. | | `options` | object | No | Any `/read` parameter except `url`, applied to every page. | *Example: the docs section of a site, 500 pages at most* ```json { "url": "https://docs.example.com/", "limit": 500, "max_depth": 3, "include_paths": [ "/docs/*" ], "exclude_paths": [ "/docs/archive/*" ], "options": { "format": "markdown" } } ``` **Response** `202 Accepted` ```json { "job_id": "job_8b27f41c0e" } ``` *curl* ```bash curl -s "https://webreaderapi.com/api/v1/crawl" \ -H "Authorization: Bearer $WEBREADER_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://docs.example.com/", "limit": 500, "include_paths": ["/docs/*"]}' ``` *Python* ```python import os import requests resp = requests.post( "https://webreaderapi.com/api/v1/crawl", headers={"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}, json={"url": "https://docs.example.com/", "limit": 500, "include_paths": ["/docs/*"]}, timeout=30, ) print(resp.json()["job_id"]) ``` *Node.js* ```javascript const res = await fetch('https://webreaderapi.com/api/v1/crawl', { method: 'POST', headers: { Authorization: `Bearer ${process.env.WEBREADER_API_KEY}`, 'Content-Type': 'application/json' }, body: JSON.stringify({ url: 'https://docs.example.com/', limit: 500, include_paths: ['/docs/*'] }), }); console.log((await res.json()).job_id); ``` ## Get job status and results `GET /api/v1/jobs/{job_id}` Returns the job's status and the next results that are ready. Each item of `results` is either a full `/read` response or `{url, error}` for a page that could not be read. When `next_cursor` is present, call again with `cursor` set to it; when it is absent, no more results are ready yet. Values are illustrative. Authentication: API key in the `Authorization: Bearer API_KEY` header. **Parameters** | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `job_id` | path | string | Yes | The ID returned by `/batch` or `/crawl`. | | `cursor` | query | string | No | The `next_cursor` value from the previous call, to get the following results. | **Response** `200` ```json { "job_id": "job_8b27f41c0e", "status": "running", "results": [ { "url": "https://docs.example.com/docs/install", "final_url": "https://docs.example.com/docs/install", "status_code": 200, "rendered": false, "credits_used": 1, "title": "Install", "markdown": "# Install\n\nRun the installer...", "word_count": 418 }, { "url": "https://docs.example.com/docs/private", "error": { "code": "blocked_by_robots", "message": "robots.txt disallows this page for WebReaderAPI." } } ], "next_cursor": "c_000002", "credits_used": 1 } ``` *curl* ```bash curl -sG "https://webreaderapi.com/api/v1/jobs/job_8b27f41c0e" \ -H "Authorization: Bearer $WEBREADER_API_KEY" \ -d cursor=c_000002 ``` *Python* ```python import os import requests resp = requests.get( "https://webreaderapi.com/api/v1/jobs/job_8b27f41c0e", headers={"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}, params={"cursor": "c_000002"}, timeout=30, ) job = resp.json() print(job["status"], len(job["results"]), job.get("next_cursor")) ``` *Node.js* ```javascript const res = await fetch('https://webreaderapi.com/api/v1/jobs/job_8b27f41c0e?cursor=c_000002', { headers: { Authorization: `Bearer ${process.env.WEBREADER_API_KEY}` }, }); const job = await res.json(); console.log(job.status, job.results.length, job.next_cursor); ``` ## How do I wait for a job to finish? Poll `/jobs/{job_id}` and keep the cursor. Process the results you get, then call again with `cursor` set to `next_cursor`. When `next_cursor` is absent and `status` is `done`, you have everything; when it is absent and the job is still `running`, wait and ask again with the same cursor. Start polling every 2 seconds and double the wait up to 30 seconds while nothing new arrives. A crawl that respects a site's Crawl-delay can take a while, so a long wait is normal and costs nothing. *Python: poll a job and save every page as a .md file* ```python import os import pathlib import re import time import requests JOBS = "https://webreaderapi.com/api/v1/jobs" HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]} def save_job(job_id, folder="pages"): out = pathlib.Path(folder) out.mkdir(exist_ok=True) cursor, wait, saved = None, 2, 0 while True: params = {"cursor": cursor} if cursor else {} job = requests.get(f"{JOBS}/{job_id}", headers=HEADERS, params=params, timeout=30).json() for item in job["results"]: if "error" in item: print("skipped", item["url"], item["error"]["code"]) continue name = re.sub(r"[^a-z0-9]+", "-", item["final_url"].lower()).strip("-")[:120] (out / f"{name}.md").write_text(item["markdown"], encoding="utf-8") saved += 1 if job.get("next_cursor"): cursor, wait = job["next_cursor"], 2 continue if job["status"] == "done": return saved time.sleep(wait) wait = min(wait * 2, 30) if __name__ == "__main__": print(save_job("job_8b27f41c0e"), "pages saved") ``` ## Which pages does a crawl read? A crawl starts from your URL and moves outward one link step at a time, breadth first, up to `max_depth`. It also reads the site's sitemap, from the address given in robots.txt or from `/sitemap.xml`, to find pages that few links point to. - **Same domain only.** Links to other domains are listed in the results when you ask for `include_links`, but they are not read. - **Path filters.** `include_paths` and `exclude_paths` take simple patterns such as `/blog/*`. - **One copy per page.** When a site declares a canonical URL, a page reached through several addresses is read once. - **A hard stop.** The job ends when it has read `limit` pages. ## How does a crawl respect the site? Every request carries the User-Agent `WebReaderAPI/1.0 (+https://webreaderapi.com/bot/)`, and the crawler follows the robots.txt rules for `WebReaderAPI`, or for all robots when none name it. Disallowed pages are skipped and appear in `results` with the error `blocked_by_robots`; they cost nothing. When robots.txt sets a `Crawl-delay`, the crawler waits that long between two requests to the site. If a site declares Content-Signal: ai-input=no in its robots.txt, we do not read its pages for you. If a site refuses our request, we return its answer: we never solve CAPTCHAs, log in to accounts, or try to get around a block or a paywall. ## How much does a crawl cost? Each page read costs what a `/read` call would: 1 credit, 2 if the page needed a browser, plus 10 with `country`. Skipped and failed pages are free. The job's `credits_used` adds up what has been billed so far. A 500-page documentation site read without a browser uses about 500 credits, which the 10,000 pages pack ($3.75) covers many times over. Since a job cannot be cancelled in v1, `limit` is your spending cap: set it to the most pages you are ready to pay for. ## Batch and crawl: common questions ### Should I use /batch or many /read calls? Use `/batch` when you already have the URLs and do not need each answer at once. It saves you from managing concurrency and retries. Use `/read` when an agent needs one page now. ### How long are job results kept? Collect them as soon as they are ready. Results wait until you fetch them; once delivered, no copy of the page content is kept, so store what you need on your side. ### Can a crawl follow links to other websites? No. A crawl reads one domain. Other domains appear as links when you ask for them, and you can send those URLs to `/batch` if you want them read. ### Can I stop a crawl that has started? Not in v1. The job stops at `limit` pages, so choose a limit that matches your budget. Pages already read stay billed; pages never read cost nothing. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto.