Quick answer
What do WebReaderAPI error codes mean, and which ones should I retry?
WebReaderAPI errors come in three groups, and none is ever charged. Request problems (invalid_url, invalid_api_key, insufficient_credits) need a fix on your side. Site decisions (blocked_by_robots, blocked_by_site_policy, target_refused) are final. Temporary failures (fetch_failed, fetch_timeout, concurrency_limit) can be retried with backoff. Credits start at $0.07/1K pages, paid in crypto with no KYC.
Which error codes can the API return?
The body is always {"error": {"code", "message"}}, plus target_status when the refusal came from the site.
| HTTP | code | Cause | Retry? |
|---|---|---|---|
| 400 | invalid_url | Not http or https, too long, or a private, local or cloud metadata address | No: fix the URL |
| 400 | invalid_parameter | A parameter has a value outside its list or range | No: fix the request |
| 401 | invalid_api_key | Key missing, wrong or revoked | No: check the key |
| 402 | insufficient_credits | Your balance has no credits left | After buying a pack |
| 403 | blocked_by_robots | The site's robots.txt disallows the page for WebReaderAPI or for all robots | No: final |
| 403 | blocked_by_site_policy | The site declares Content-Signal: ai-input=no | No: final |
| 422 | unsupported_content | A type we do not read (video, archive) or a file too large | No |
| 429 | concurrency_limit | Too many of your requests are running at the same time | Yes, with backoff |
| 502 | target_refused | The site refused: 403, 429, a challenge page or a login wall; its status is in target_status | No: final |
| 502 | fetch_failed | DNS, TLS or connection error, or the site answered 404 or 410 | Once or twice, with backoff |
| 504 | fetch_timeout | The site did not answer within timeout | Yes, with backoff or a longer timeout |
Why are robots.txt and site refusals final?
Because they are the site owner's decision, and WebReaderAPI is built to follow it. We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged. If a site declares Content-Signal: ai-input=no in its robots.txt, we do not read its pages for you.
When a site turns a request away in another way, with a 403, a 429, a challenge page or a login form, you get target_refused and the site's own status in target_status. If a site refuses our request, we return its answer: we never solve CAPTCHAs, log in to accounts, or try to get around a block or a paywall. Retrying the same URL a minute later will usually get the same answer, so treat these codes as a result, not a failure: log them and move on.
How do I check why a page was refused?
Open the site's /robots.txt and look for a group that names WebReaderAPI, or the * group that applies to all robots. A Disallow line that matches the path explains a blocked_by_robots. A line such as Content-Signal: ai-input=no explains a blocked_by_site_policy.
We keep a copy of each robots.txt for at most 24 hours. If a site has just opened a section to robots, the change can take up to a day to apply. Site owners can read how our crawler behaves, and how to allow or block it, on our crawler page.
How should I retry temporary failures?
Retry only concurrency_limit, fetch_timeout and, once or twice, fetch_failed. Wait before each new attempt and double the wait each time, with a little random jitter so parallel workers do not retry together. Stop after 4 or 5 attempts.
For concurrency_limit, the better fix is to send fewer requests at once: Every account has a limit on how many pages it reads at once; your dashboard shows it, and support can raise it. For fetch_timeout, raise timeout up to 60 seconds before retrying a slow site. Retries cost nothing, because a failed attempt is never billed.
import os
import random
import time
import requests
API = "https://webreaderapi.com/api/v1/read"
HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}
RETRY = {"concurrency_limit", "fetch_timeout", "fetch_failed"}
FINAL = {"blocked_by_robots", "blocked_by_site_policy", "target_refused", "unsupported_content"}
def read_page(url, attempts=5, **options):
wait = 1.0
for attempt in range(1, attempts + 1):
resp = requests.get(API, headers=HEADERS, params={"url": url, **options}, timeout=75)
if resp.status_code == 200:
return resp.json()
error = resp.json().get("error", {})
code = error.get("code", "unknown")
if code in FINAL:
return {"url": url, "skipped": code, "target_status": error.get("target_status")}
if code not in RETRY or attempt == attempts or (code == "fetch_failed" and attempt >= 3):
raise RuntimeError(f"{code}: {error.get('message')}")
time.sleep(wait + random.uniform(0, wait / 2))
wait = min(wait * 2, 30)
raise RuntimeError("unreachable")
if __name__ == "__main__":
print(read_page("https://example.com/blog/first-post", max_chars=20000).get("title"))
const RETRY = new Set(['concurrency_limit', 'fetch_timeout', 'fetch_failed']);
const FINAL = new Set(['blocked_by_robots', 'blocked_by_site_policy', 'target_refused', 'unsupported_content']);
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
export async function readPage(url, options = {}, attempts = 5) {
let wait = 1000;
for (let attempt = 1; attempt <= attempts; attempt++) {
const params = new URLSearchParams({ url, ...options });
const res = await fetch(`https://webreaderapi.com/api/v1/read?${params}`, {
headers: { Authorization: `Bearer ${process.env.WEBREADER_API_KEY}` },
});
const body = await res.json();
if (res.ok) return body;
const code = body.error?.code ?? 'unknown';
if (FINAL.has(code)) return { url, skipped: code, target_status: body.error.target_status };
if (!RETRY.has(code) || attempt === attempts) throw new Error(`${code}: ${body.error?.message}`);
await sleep(wait + Math.random() * (wait / 2));
wait = Math.min(wait * 2, 30000);
}
}
What about errors inside a batch or crawl?
A job does not stop because one page fails. That page appears in results as {url, error} with the same codes as above, and it costs nothing. A crawl that meets blocked_by_robots on a section simply reads the rest of the site.
At the end of a job, count the error codes. Many target_refused on one site mean the site does not accept automated readers; many fetch_timeout mean it is slow, and a longer timeout in the job options may help.
Errors: common questions
Am I charged for a page that robots.txt blocks?
No. 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged. The same goes for every other error on this page.
Can you read a page that blocks your crawler if I ask?
No. We follow the site owner's choice, and the policy is the same for every customer. If you own the site, add a rule for WebReaderAPI in your robots.txt; the crawler page shows how.
Why do I get fetch_failed for a page that exists?
The site may have been down, slow to accept a TLS connection or returning 404 to automated clients. Retry once or twice with backoff. If it keeps failing, check the address in a browser.
What should my agent tell the user after a refusal?
Something simple, such as the site does not allow automated reading of that page. Then let the model use other sources. Returning the error code as the tool result is enough for most models.