Quick answer
What is the WebReaderAPI bot, and how do I block it?
The WebReaderAPI bot fetches public pages when one of our customers asks the API to read them. It sends the user agent WebReaderAPI/1.0 (+https://webreaderapi.com/bot/) and obeys robots.txt rules written for the token WebReaderAPI or for all robots. To block it everywhere, add User-agent: WebReaderAPI followed by Disallow: / to your robots.txt.
The bot at a glance
| User-Agent header | WebReaderAPI/1.0 (+https://webreaderapi.com/bot/) |
|---|---|
| robots.txt token | WebReaderAPI |
| robots.txt | We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged |
| AI preference signals | If a site declares Content-Signal: ai-input=no in its robots.txt, we do not read its pages for you |
| When a site says no | If a site refuses our request, we return its answer: we never solve CAPTCHAs, log in to accounts, or try to get around a block or a paywall |
| Copies of your pages | None kept after the answer is sent to the customer |
| Crawl jobs | Crawl one website per job, up to 10,000 pages, following its sitemap, its robots.txt and its Crawl-delay |
| Contact | [email protected] |
Why did WebReaderAPI fetch my page?
Because a customer of the API asked for it. WebReaderAPI does not build its own index of the web and does not browse on its own initiative. Each fetch belongs to one of three request types:
- A single read: an application or an AI agent sent your URL to
/read, usually because a person asked a question that your page answers. - A batch job: a customer sent a list of up to 100 URLs, and yours was on it.
- A crawl job: a customer asked us to read a whole site, starting from one address and staying on that domain. Crawls follow your sitemap and your robots.txt.
The customer receives the main text of the page as Markdown. We keep no copy of it afterwards.
How do I block the bot on my whole site?
Add a group for our token to your robots.txt file. The bot reads /robots.txt before fetching a page and caches it for at most 24 hours, so a change applies within a day at the latest.
User-agent: WebReaderAPI
Disallow: /
Can I block only part of my site, or slow the bot down?
Yes. List the folders you want to keep out under Disallow, and set a Crawl-delay in seconds for crawl jobs. Rules written for all robots (User-agent: *) also apply to us when you have no group for our token. When you do have one, that group wins, as RFC 9309 describes.
User-agent: WebReaderAPI
Disallow: /account/
Disallow: /drafts/
Crawl-delay: 5
Does the bot honour AI preference signals?
Yes, the ones a site states in robots.txt. If your file declares Content-Signal: ai-input=no, we treat it as a refusal to have your pages used as input for AI answers, and we do not read them for our customers. The API then returns the error blocked_by_site_policy to the customer, who is not charged.
User-agent: *
Content-Signal: search=yes, ai-input=no
Allow: /
What does the bot do when my site refuses it?
It stops and reports your answer. A 403 or 429 status, a challenge page, a login form or a paywall all end the request. The customer gets the error target_refused with your status code, and is not charged for it.
The bot never solves challenges, never logs in, never changes its user agent to look like a browser and never retries in disguise. If you want us to read your pages, allow the token in robots.txt and let the requests through; if not, nothing more is needed on your side.
What does WebReaderAPI not promise yet?
We do not publish a list of IP addresses today, so please identify the bot by its user agent and its robots.txt behaviour, not by IP. Pages read with the country option leave from an address in the chosen country, still with the same user agent. We will add our address ranges to this page once they are fixed, with the date of the change.
How do I report a problem with the bot?
Write to [email protected] with the date and time, your domain, the URL fetched and a line of your access log showing our user agent. A person reads each report and checks it against our logs. If a customer uses the API against our Acceptable Use Policy, their account is suspended.
To check whether a request really came from us, compare the user agent and the robots.txt behaviour with this page. A client that ignores your robots.txt rules while using our name is not our bot.
Site owners' questions
Is the WebReaderAPI bot a search engine crawler?
No. It does not index the web or rank pages. It reads a page only when a customer of the API asks for that page, in a single request, a batch or a crawl of one site.
Will blocking the bot affect my Google or Bing rankings?
No. Rules written for WebReaderAPI concern only our bot. Search engine crawlers follow their own groups in your robots.txt.
How fast does a robots.txt change take effect?
Within 24 hours at the latest. The bot caches robots.txt for at most a day before reading it again.
Does the bot respect Crawl-delay?
Yes, for crawl jobs, which read many pages of one site. A single read is one request and is not spaced out.
Do you store or sell the content of my pages?
No. The page text goes to the customer who asked for it and is not kept by us. Our logs hold the host name, the time, the status and the credits used, for 30 days.