Test
10,000 pages
$3.75
$0.38/1K pages
- Try it on your real URLs
- Same API as the biggest pack
- Credits never expire
Quick answer
What is the best way to give an AI agent a web fetch tool?
Wrap a web reader API as a tool that takes a URL and returns the page's main content as Markdown, cut to a set length. WebReaderAPI does the reading, renders JavaScript when needed, respects robots.txt, and charges one credit per page from $0.07/1K pages as of October 2026. Ready tool code covers MCP, OpenAI function calling, LangChain, n8n and OpenClaw. No KYC; crypto.
WebReaderAPI as an agent tool
| Tool | read_url(url, max_chars) calling https://webreaderapi.com/api/v1/read |
|---|---|
| Returns | Markdown of the main content, plus title, author, date and word count |
| Length cap | max_chars, 1,000 to 500,000 characters per page |
| JavaScript pages | JavaScript rendering in a headless browser when the page needs it (render=auto), or always, or never |
| Refusals | Returned as clear errors, never charged |
| Integrations | MCP, OpenAI, LangChain and LlamaIndex, n8n, OpenClaw |
| Price | $3.75 for 10,000 pages; credits never expire |
Why does an agent need a reader rather than raw HTML?
Because raw HTML spends the context window on things the model does not need. A news article or a docs page arrives wrapped in navigation, scripts, consent banners and related links. Passed as is, it crowds out the conversation and makes the model quote menus.
A reader solves three problems at once. It keeps only the main content, it renders pages that are empty until JavaScript runs, and it turns PDFs into text. The agent gets one compact block of Markdown with a title and a date it can cite.
Why do built-in fetch tools sometimes fail?
Mostly because sites refuse automated traffic. Since 1 July 2025, Cloudflare blocks AI crawlers by default for new domains, and from 15 September 2026 its presets block "Agent" traffic on ad-supported pages of new domains. Built-in tools in coding agents and assistants meet those refusals like any other client.
Other failures are technical. OpenClaw's documentation, read on 22 October 2026, says its built-in web_fetch extracts content with Readability, does not run JavaScript and caps output at 20,000 characters. Pages built in the browser, long documents, slow servers and rate limits all trip such tools. A reader that renders pages fixes the JavaScript case; caching and patience fix the rest. Refusals are the site owner's call. Our guide to web fetch 403 errors explains each case.
Why is the tool called read_url and not web_fetch?
To avoid a clash with tools agents already have. Claude Code and OpenClaw both ship a built-in tool named web_fetch. A second tool with the same name confuses the model about which one it is calling. Naming ours read_url keeps both available: the agent can use its built-in fetcher for quick checks and read_url when it needs clean Markdown, PDFs or JavaScript pages.
The tool definitions on our docs pages all use this name, so you can switch between frameworks without changing your prompts.
import os
import requests
API = "https://webreaderapi.com/api/v1/read"
HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}
def read_url(url: str, max_chars: int = 12000) -> str:
"""Return a web page as Markdown for a model, or a short reason it was not read."""
resp = requests.get(API, headers=HEADERS, params={"url": url, "max_chars": max_chars}, timeout=90)
data = resp.json()
if resp.status_code != 200:
err = data.get("error", {})
return "Not read (" + err.get("code", "error") + "): " + err.get("message", "")
head = "# " + (data.get("title") or url) + "\nSource: " + data["final_url"]
if data.get("published_at"):
head += " | Published: " + data["published_at"]
return head + "\n\n" + data["markdown"]
if __name__ == "__main__":
print(read_url("https://example.com/blog/pour-over-coffee", max_chars=2000)[:500])
How should an agent decide what to read?
Give it a budget and a rule. Three habits keep costs and context in check:
- Read few, read well. Two to four sources per question are usually enough to answer with citations.
- Cap each page. Start with
max_charsaround 10,000 to 15,000 characters; raise it only when theword_countshows the answer is likely further down. - Report refusals. When a page is not read, let the model say so and move to another source instead of retrying.
The tool above returns refusals as one plain sentence, so the model can explain them to the user.
What does it cost to run an agent on WebReaderAPI?
One credit per page, two when the page needs a browser. An assistant that answers 100 questions a day and reads 4 pages for each reads about 12,000 pages a month. 50,000 pages ($14.25) cover that for four months, as of October 2026.
Credits never expire, so an agent that is busy one week and idle the next does not waste a monthly quota. Pages that fail or that robots.txt refuses cost nothing.
Which pack suits an agent?
Prototype on 10,000 pages; move to 50,000 pages or 100,000 pages when the agent serves real users. Credits never expire.
Starter
50,000 pages
$14.25
$0.28/1K pages
- An AI agent or a RAG prototype
- Crawl a documentation site
- Credits never expire
Build
100,000 pages
$21.75
$0.22/1K pages
- Production agents and RAG pipelines
- Batch and crawl jobs
- Credits never expire
In every plan: Main-content extraction · JavaScript rendering when needed · Failed and refused pages not charged · Credits never expire
AI agents: common questions
Can I add WebReaderAPI to Claude Desktop or Cursor?
Yes, through a small MCP server that exposes read_url. The MCP page has the code and the configuration snippet; your API key stays in an environment variable.
Does it work with OpenClaw?
Yes, as an MCP server next to OpenClaw's built-in web_fetch. The OpenClaw page shows the commands from OpenClaw's own documentation.
Can the agent read pages that block AI bots?
No. If robots.txt disallows the page, the API returns blocked_by_robots; if the site refuses, it returns target_refused with the site's status. Neither is charged, and we never try to get around the refusal.
How do I stop a long page from filling the context window?
Set max_chars. The Markdown is cut at that length, and word_count tells the agent how long the full page was, so it can ask for more only when needed.
Do I need a card or an ID?
No. WebReaderAPI asks for no ID, no selfie, no documents and no card. Payment is in BTC, USDT, USDC, ETH, SOL, LTC, with a minimum top-up of $25.
Sources (3)
- Press release: AI crawlers blocked by default — Cloudflare (2025-07-01)
- Blog: new defaults for AI crawlers and agents — Cloudflare (2026-09-15)
- Web fetch tool — OpenClaw (read 2026-10-22)