# Errors, robots.txt refusals and retries

> Every WebReaderAPI error code: robots.txt refusals, site refusals, timeouts and credit errors, which ones to retry, and why none of them is ever charged.

- URL: https://webreaderapi.com/docs/errors/
- Last updated: 2026-10-22
- Publisher: WebReaderAPI (https://webreaderapi.com)

Each error has an HTTP status, a stable `code` and a readable `message`. None of them uses a credit. Some are worth a retry; the refusals that come from a site are final, and we never work around them.

**Quick answer — What do WebReaderAPI error codes mean, and which ones should I retry?**

**WebReaderAPI** errors come in three groups, and none is ever charged. Request problems (`invalid_url`, `invalid_api_key`, `insufficient_credits`) need a fix on your side. Site decisions (`blocked_by_robots`, `blocked_by_site_policy`, `target_refused`) are final. Temporary failures (`fetch_failed`, `fetch_timeout`, `concurrency_limit`) can be retried with backoff. Credits start at $0.07/1K pages, paid in crypto with no KYC.

## Which error codes can the API return?

The body is always `{"error": {"code", "message"}}`, plus `target_status` when the refusal came from the site.

| HTTP | `code` | Cause | Retry? |
| --- | --- | --- | --- |
| 400 | `invalid_url` | Not http or https, too long, or a private, local or cloud metadata address | No: fix the URL |
| 400 | `invalid_parameter` | A parameter has a value outside its list or range | No: fix the request |
| 401 | `invalid_api_key` | Key missing, wrong or revoked | No: check the key |
| 402 | `insufficient_credits` | Your balance has no credits left | After buying a pack |
| 403 | `blocked_by_robots` | The site's robots.txt disallows the page for WebReaderAPI or for all robots | No: final |
| 403 | `blocked_by_site_policy` | The site declares `Content-Signal: ai-input=no` | No: final |
| 422 | `unsupported_content` | A type we do not read (video, archive) or a file too large | No |
| 429 | `concurrency_limit` | Too many of your requests are running at the same time | Yes, with backoff |
| 502 | `target_refused` | The site refused: 403, 429, a challenge page or a login wall; its status is in `target_status` | No: final |
| 502 | `fetch_failed` | DNS, TLS or connection error, or the site answered 404 or 410 | Once or twice, with backoff |
| 504 | `fetch_timeout` | The site did not answer within `timeout` | Yes, with backoff or a longer timeout |

## Why are robots.txt and site refusals final?

Because they are the site owner's decision, and WebReaderAPI is built to follow it. We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged. If a site declares Content-Signal: ai-input=no in its robots.txt, we do not read its pages for you.

When a site turns a request away in another way, with a 403, a 429, a challenge page or a login form, you get `target_refused` and the site's own status in `target_status`. If a site refuses our request, we return its answer: we never solve CAPTCHAs, log in to accounts, or try to get around a block or a paywall. Retrying the same URL a minute later will usually get the same answer, so treat these codes as a result, not a failure: log them and move on.

## How do I check why a page was refused?

Open the site's `/robots.txt` and look for a group that names `WebReaderAPI`, or the `*` group that applies to all robots. A `Disallow` line that matches the path explains a `blocked_by_robots`. A line such as `Content-Signal: ai-input=no` explains a `blocked_by_site_policy`.

We keep a copy of each robots.txt for at most 24 hours. If a site has just opened a section to robots, the change can take up to a day to apply. Site owners can read how our crawler behaves, and how to allow or block it, on [our crawler page](https://webreaderapi.com/bot/).

## How should I retry temporary failures?

Retry only `concurrency_limit`, `fetch_timeout` and, once or twice, `fetch_failed`. Wait before each new attempt and double the wait each time, with a little random jitter so parallel workers do not retry together. Stop after 4 or 5 attempts.

For `concurrency_limit`, the better fix is to send fewer requests at once: Every account has a limit on how many pages it reads at once; your dashboard shows it, and support can raise it. For `fetch_timeout`, raise `timeout` up to 60 seconds before retrying a slow site. Retries cost nothing, because a failed attempt is never billed.

*Python: retry only the errors that can change*

```python
import os
import random
import time

import requests

API = "https://webreaderapi.com/api/v1/read"
HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}
RETRY = {"concurrency_limit", "fetch_timeout", "fetch_failed"}
FINAL = {"blocked_by_robots", "blocked_by_site_policy", "target_refused", "unsupported_content"}


def read_page(url, attempts=5, **options):
    wait = 1.0
    for attempt in range(1, attempts + 1):
        resp = requests.get(API, headers=HEADERS, params={"url": url, **options}, timeout=75)
        if resp.status_code == 200:
            return resp.json()
        error = resp.json().get("error", {})
        code = error.get("code", "unknown")
        if code in FINAL:
            return {"url": url, "skipped": code, "target_status": error.get("target_status")}
        if code not in RETRY or attempt == attempts or (code == "fetch_failed" and attempt >= 3):
            raise RuntimeError(f"{code}: {error.get('message')}")
        time.sleep(wait + random.uniform(0, wait / 2))
        wait = min(wait * 2, 30)
    raise RuntimeError("unreachable")


if __name__ == "__main__":
    print(read_page("https://example.com/blog/first-post", max_chars=20000).get("title"))

```

*Node.js: the same rule with exponential backoff*

```javascript
const RETRY = new Set(['concurrency_limit', 'fetch_timeout', 'fetch_failed']);
const FINAL = new Set(['blocked_by_robots', 'blocked_by_site_policy', 'target_refused', 'unsupported_content']);
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

export async function readPage(url, options = {}, attempts = 5) {
  let wait = 1000;
  for (let attempt = 1; attempt <= attempts; attempt++) {
    const params = new URLSearchParams({ url, ...options });
    const res = await fetch(`https://webreaderapi.com/api/v1/read?${params}`, {
      headers: { Authorization: `Bearer ${process.env.WEBREADER_API_KEY}` },
    });
    const body = await res.json();
    if (res.ok) return body;
    const code = body.error?.code ?? 'unknown';
    if (FINAL.has(code)) return { url, skipped: code, target_status: body.error.target_status };
    if (!RETRY.has(code) || attempt === attempts) throw new Error(`${code}: ${body.error?.message}`);
    await sleep(wait + Math.random() * (wait / 2));
    wait = Math.min(wait * 2, 30000);
  }
}

```

## What about errors inside a batch or crawl?

A job does not stop because one page fails. That page appears in `results` as `{url, error}` with the same codes as above, and it costs nothing. A crawl that meets `blocked_by_robots` on a section simply reads the rest of the site.

At the end of a job, count the error codes. Many `target_refused` on one site mean the site does not accept automated readers; many `fetch_timeout` mean it is slow, and a longer `timeout` in the job options may help.

## Errors: common questions

### Am I charged for a page that robots.txt blocks?

No. 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged. The same goes for every other error on this page.

### Can you read a page that blocks your crawler if I ask?

No. We follow the site owner's choice, and the policy is the same for every customer. If you own the site, add a rule for `WebReaderAPI` in your robots.txt; the [crawler page](https://webreaderapi.com/bot/) shows how.

### Why do I get fetch_failed for a page that exists?

The site may have been down, slow to accept a TLS connection or returning 404 to automated clients. Retry once or twice with backoff. If it keeps failing, check the address in a browser.

### What should my agent tell the user after a refusal?

Something simple, such as the site does not allow automated reading of that page. Then let the model use other sources. Returning the error code as the tool result is enough for most models.

---

WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto.