# Web fetch tool for AI agents: read pages as clean Markdown

> A web fetch tool for AI agents: read any public page as Markdown with a length cap, via MCP, OpenAI tools, LangChain or n8n, from $0.07/1K pages. Crypto.

- URL: https://webreaderapi.com/use-cases/ai-agents/
- Last updated: 2026-10-22
- Publisher: WebReaderAPI (https://webreaderapi.com)

An agent that can search but not read answers from snippets. Give it a tool that turns any public URL into short, clean Markdown, and it can read the source, quote it and cite it, within a budget you set.

**Quick answer — What is the best way to give an AI agent a web fetch tool?**

Wrap a web reader API as a tool that takes a URL and returns the page's main content as Markdown, cut to a set length. **WebReaderAPI** does the reading, renders JavaScript when needed, respects robots.txt, and charges one credit per page from $0.07/1K pages as of October 2026. Ready tool code covers MCP, OpenAI function calling, LangChain, n8n and OpenClaw. No KYC; crypto.

## WebReaderAPI as an agent tool

| Attribute | Value |
| --- | --- |
| Tool | `read_url(url, max_chars)` calling `https://webreaderapi.com/api/v1/read` |
| Returns | Markdown of the main content, plus title, author, date and word count |
| Length cap | `max_chars`, 1,000 to 500,000 characters per page |
| JavaScript pages | JavaScript rendering in a headless browser when the page needs it (render=auto), or always, or never |
| Refusals | Returned as clear errors, never charged |
| Integrations | [MCP](https://webreaderapi.com/docs/mcp/), [OpenAI](https://webreaderapi.com/docs/openai/), [LangChain and LlamaIndex](https://webreaderapi.com/docs/langchain/), [n8n](https://webreaderapi.com/docs/n8n/), [OpenClaw](https://webreaderapi.com/docs/openclaw/) |
| Price | $3.75 for 10,000 pages; credits never expire |

## Why does an agent need a reader rather than raw HTML?

Because raw HTML spends the context window on things the model does not need. A news article or a docs page arrives wrapped in navigation, scripts, consent banners and related links. Passed as is, it crowds out the conversation and makes the model quote menus.

A reader solves three problems at once. It keeps only the main content, it renders pages that are empty until JavaScript runs, and it turns PDFs into text. The agent gets one compact block of Markdown with a title and a date it can cite.

## Why do built-in fetch tools sometimes fail?

Mostly because sites refuse automated traffic. Since 1 July 2025, Cloudflare blocks AI crawlers by default for new domains, and from 15 September 2026 its presets block "Agent" traffic on ad-supported pages of new domains. Built-in tools in coding agents and assistants meet those refusals like any other client.

Other failures are technical. OpenClaw's documentation, read on 22 October 2026, says its built-in web_fetch extracts content with Readability, does not run JavaScript and caps output at 20,000 characters. Pages built in the browser, long documents, slow servers and rate limits all trip such tools. A reader that renders pages fixes the JavaScript case; caching and patience fix the rest. Refusals are the site owner's call. Our [guide to web fetch 403 errors](https://webreaderapi.com/guides/web-fetch-403-errors/) explains each case.

## Why is the tool called read_url and not web_fetch?

To avoid a clash with tools agents already have. Claude Code and OpenClaw both ship a built-in tool named `web_fetch`. A second tool with the same name confuses the model about which one it is calling. Naming ours `read_url` keeps both available: the agent can use its built-in fetcher for quick checks and `read_url` when it needs clean Markdown, PDFs or JavaScript pages.

The tool definitions on our docs pages all use this name, so you can switch between frameworks without changing your prompts.

*Python: a compact read_url tool with a length cap*

```python
import os

import requests

API = "https://webreaderapi.com/api/v1/read"
HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}


def read_url(url: str, max_chars: int = 12000) -> str:
    """Return a web page as Markdown for a model, or a short reason it was not read."""
    resp = requests.get(API, headers=HEADERS, params={"url": url, "max_chars": max_chars}, timeout=90)
    data = resp.json()
    if resp.status_code != 200:
        err = data.get("error", {})
        return "Not read (" + err.get("code", "error") + "): " + err.get("message", "")
    head = "# " + (data.get("title") or url) + "\nSource: " + data["final_url"]
    if data.get("published_at"):
        head += " | Published: " + data["published_at"]
    return head + "\n\n" + data["markdown"]


if __name__ == "__main__":
    print(read_url("https://example.com/blog/pour-over-coffee", max_chars=2000)[:500])

```

## How should an agent decide what to read?

Give it a budget and a rule. Three habits keep costs and context in check:

- **Read few, read well.** Two to four sources per question are usually enough to answer with citations.
- **Cap each page.** Start with `max_chars` around 10,000 to 15,000 characters; raise it only when the `word_count` shows the answer is likely further down.
- **Report refusals.** When a page is not read, let the model say so and move to another source instead of retrying.

The tool above returns refusals as one plain sentence, so the model can explain them to the user.

## What does it cost to run an agent on WebReaderAPI?

One credit per page, two when the page needs a browser. An assistant that answers 100 questions a day and reads 4 pages for each reads about 12,000 pages a month. 50,000 pages ($14.25) cover that for four months, as of October 2026.

Credits never expire, so an agent that is busy one week and idle the next does not waste a monthly quota. Pages that fail or that robots.txt refuses cost nothing.

## Which pack suits an agent?

Prototype on 10,000 pages; move to 50,000 pages or 100,000 pages when the agent serves real users. Credits never expire.

| Plan | Quantity | Price | Price per unit |
| --- | --- | --- | --- |
| Test | 10,000 pages | $3.75 | $0.38/1K pages |
| Starter | 50,000 pages | $14.25 | $0.28/1K pages |
| Build | 100,000 pages | $21.75 | $0.22/1K pages |

In every plan: Main-content extraction · JavaScript rendering when needed · Failed and refused pages not charged · Credits never expire

**Before you pay**

- Minimum top-up $25. Top up once in crypto, then buy any plan from your balance.
- Your balance has no end date and can buy page credits at any time. If a pack costs less than the minimum top-up, the amount you added only to reach that minimum can be sent back on request within 30 days of the payment, in USDT or USDC, as long as you have not spent it. Any other balance is not refunded unless the law says otherwise.
- Credited once your payment is checked on the blockchain. Typical network confirmation time: BTC ≈ 20 min · USDT TRC-20 ≈ 1 min · USDT ERC-20 ≈ 5 min · USDC ERC-20 ≈ 5 min · ETH ≈ 5 min · SOL ≈ 1 min · LTC ≈ 15 min

## AI agents: common questions

### Can I add WebReaderAPI to Claude Desktop or Cursor?

Yes, through a small MCP server that exposes `read_url`. The [MCP page](https://webreaderapi.com/docs/mcp/) has the code and the configuration snippet; your API key stays in an environment variable.

### Does it work with OpenClaw?

Yes, as an MCP server next to OpenClaw's built-in web_fetch. The [OpenClaw page](https://webreaderapi.com/docs/openclaw/) shows the commands from OpenClaw's own documentation.

### Can the agent read pages that block AI bots?

No. If robots.txt disallows the page, the API returns `blocked_by_robots`; if the site refuses, it returns `target_refused` with the site's status. Neither is charged, and we never try to get around the refusal.

### How do I stop a long page from filling the context window?

Set `max_chars`. The Markdown is cut at that length, and `word_count` tells the agent how long the full page was, so it can ask for more only when needed.

### Do I need a card or an ID?

No. WebReaderAPI asks for no ID, no selfie, no documents and no card. Payment is in BTC, USDT, USDC, ETH, SOL, LTC, with a minimum top-up of $25.

## Sources

1. [Press release: AI crawlers blocked by default](https://www.cloudflare.com/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/) — Cloudflare (2025-07-01)
2. [Blog: new defaults for AI crawlers and agents](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) — Cloudflare (2026-09-15)
3. [Web fetch tool](https://docs.openclaw.ai/tools/web-fetch.md) — OpenClaw (read 2026-10-22)

---

WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto.