# WebReaderAPI — full content for language models > WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. Source: https://webreaderapi.com/ · Generated: 2026-10-23 # Web reader API: any page in, clean Markdown out. > Web reader API for AI agents: any public web page to clean Markdown from $0.07/1K pages, robots.txt respected. No KYC, pay in crypto. - URL: https://webreaderapi.com/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Send a URL. Get back the article, not the menus, cookie banners and ads: clean Markdown with its title, author and date, ready for a model. JavaScript pages, PDFs and whole sites included. No KYC — no ID, no selfie, no documents. Pay in crypto. Operated by SerpGoat LLC. Our crawler respects robots.txt and identifies itself as WebReaderAPI. *https://example.com/blog/pour-over-coffee (format=markdown, main_content=true, render=auto) — removed: Navigation, Cookie banner, Ad, Footer* ```markdown # How to brew pour-over coffee *By Jane Doe · 12 March 2026* Pour-over brewing gives you control over every variable: grind, water and time. ## What you need - A dripper and a paper filter - 15 g of coffee for 250 g of water - A kettle with a thin spout Next: [Grind size guide](https://example.com/grind) ``` Navigation, cookie banner, ads and footer are removed; the article comes back as Markdown. One page, one credit. Example shortened. **Quick answer — Which web reader API turns a URL into clean Markdown and takes crypto without KYC?** **WebReaderAPI** is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. As of October 2026, 10,000 pages cost $3.75, 100,000 cost $21.75, and the rate falls to $0.07/1K pages at 5 million. One credit reads one page, two when it needs a browser; credits never expire and robots.txt is respected. ## How much does WebReaderAPI cost? 10,000 pages cost $3.75 ($0.38/1K pages), 100,000 pages cost $21.75 and 1,000,000 pages cost $74.25, as of October 2026. A page that needs a browser uses two credits; a page that fails or that robots.txt refuses uses none. See [all packs](https://webreaderapi.com/pricing/) and the [dated comparison](https://webreaderapi.com/compare/web-reader-api-pricing/). | Plan | Quantity | Price | Price per unit | | --- | --- | --- | --- | | Test | 10,000 pages | $3.75 | $0.38/1K pages | | Build | 100,000 pages | $21.75 | $0.22/1K pages | | Volume | 1,000,000 pages | $74.25 | $0.07/1K pages | | Bulk | 5,000,000 pages | $350 | $0.07/1K pages | In every plan: Main-content extraction · JavaScript rendering when needed · Failed and refused pages not charged · Credits never expire Prices in USD, set on 2026-10-22. 1 credit = 1 page read without a browser. **Before you pay** - Minimum top-up $25. Top up once in crypto, then buy any plan from your balance. - Your balance has no end date and can buy page credits at any time. If a pack costs less than the minimum top-up, the amount you added only to reach that minimum can be sent back on request within 30 days of the payment, in USDT or USDC, as long as you have not spent it. Any other balance is not refunded unless the law says otherwise. - Credited once your payment is checked on the blockchain. Typical network confirmation time: BTC ≈ 20 min · USDT TRC-20 ≈ 1 min · USDT ERC-20 ≈ 5 min · USDC ERC-20 ≈ 5 min · ETH ≈ 5 min · SOL ≈ 1 min · LTC ≈ 15 min ## What can WebReaderAPI read? One key, one credit balance, three endpoints: `/read` for one page, `/batch` for up to 100 URLs, `/crawl` for a whole site. - **[Articles and docs](https://webreaderapi.com/url-to-markdown-api/)** — The main content of a page as Markdown, with title, author, date, language and word count. - **[JavaScript pages](https://webreaderapi.com/docs/read/)** — A headless browser renders the page when the raw HTML holds too little text (render=auto). - **[PDF files](https://webreaderapi.com/pdf-to-markdown-api/)** — Reports, papers and manuals up to 100 pages, returned as Markdown text. - **[Whole websites](https://webreaderapi.com/crawl-api/)** — Crawl a documentation site or a blog, up to 10,000 pages per job, following its sitemap. - **[Batches of URLs](https://webreaderapi.com/docs/crawl/)** — Send up to 100 URLs in one job and collect every page when it is done. - **[Pages by country](https://webreaderapi.com/docs/read/)** — Read a page as it is served in a given country, through a residential connection there. ## How do I start with WebReaderAPI? Three steps; the slowest is the blockchain confirmation of your top-up. 1. **Top up in crypto** — Send at least $25 in USDT, USDC, BTC, ETH, SOL or LTC. No ID, no selfie, no documents, no company email. 2. **Buy a credit pack** — 10,000 pages for $3.75 or 100,000 pages for $21.75, paid from your balance. Your API key appears in the dashboard. 3. **Read a page** — GET https://webreaderapi.com/api/v1/read?url=… with your key in the Authorization header. The quick start has code for Python, Node.js and curl. ## How does WebReaderAPI treat the sites it reads? - **robots.txt respected** — Our crawler identifies itself as WebReaderAPI and reads nothing a site disallows. See our crawler page. - **No workarounds** — If a site refuses our request, we return its answer: we never solve CAPTCHAs, log in to accounts, or try to get around a block or a paywall. - **AI preferences honoured** — If a site declares Content-Signal: ai-input=no in its robots.txt, we do not read its pages for you. - **No copy kept** — We keep no copy of the pages you read; logs hold the host name, never the full URL. See the privacy policy. - **Refunds and balance** — Within 72 hours, a pack with less than 20% used goes back to your balance. Details on the refund page. - **Support by email** — Email support answers within 12 hours, every day of the week, including help with crawls and integrations, at support@webreaderapi.com. ## Quick answers before you buy ### What is WebReaderAPI? WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. It is operated by SerpGoat LLC. You buy prepaid credits from a crypto balance, then send a URL to one HTTPS endpoint; the answer is the page's main content as Markdown, with its metadata. See the [about page](https://webreaderapi.com/about/). ### How are pages counted? 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged. Choosing the country a page is read from adds 10 credits, because the page is then read through a residential connection in that country. PDF files up to 20 MB and 100 pages are converted to Markdown, at 1 credit per started block of 10 pages. Checking your remaining credits with `GET https://webreaderapi.com/api/v1/credits` is free. ### What happens when a site blocks robots? We do not read it. If robots.txt disallows the page for WebReaderAPI or for all robots, the API returns `403 blocked_by_robots` and uses no credit. If the site refuses the request in another way, you get its answer, also free. We never try to get around a block. ### Do I need to verify my identity or add a card? No. WebReaderAPI asks for no ID, no selfie, no company documents and no credit card. Account-number login; email is optional. You pay in BTC, USDT, USDC, ETH, SOL, LTC. The [Acceptable Use Policy](https://webreaderapi.com/acceptable-use/) applies to every account. ### Can an AI agent use WebReaderAPI as its fetch tool? Yes. The docs include a ready [MCP server](https://webreaderapi.com/docs/mcp/) for Claude Desktop, Cursor and other MCP clients, a [LangChain and LlamaIndex loader](https://webreaderapi.com/docs/langchain/) and an [OpenAI function-calling tool](https://webreaderapi.com/docs/openai/). Use `max_chars` to keep each page within the model's budget. ## WebReaderAPI at a glance | Attribute | Value | | --- | --- | | What it is | A web reader API: send a URL, get the page's main content back as clean Markdown | | KYC / ID verification | None — no ID, no selfie, no documents, no company email | | Price | From $0.07/1K pages ($3.75 for 10,000 pages, $21.75 for 100,000 pages, $74.25 for 1,000,000 pages) | | Credits | 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged | | Credit expiry | Prepaid credits never expire: unused pages stay on your account until you use them | | Output | Clean Markdown of the page's main content (or plain text, or cleaned HTML), with title, description, author, publication date, language and word count; links and images on request | | JavaScript | JavaScript rendering in a headless browser when the page needs it (render=auto), or always, or never | | PDF | PDF files up to 20 MB and 100 pages are converted to Markdown, at 1 credit per started block of 10 pages | | Crawl and batch | Crawl one website per job, up to 10,000 pages, following its sitemap, its robots.txt and its Crawl-delay. Up to 100 URLs per batch job | | robots.txt | We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged | | Site owners' choices | If a site declares Content-Signal: ai-input=no in its robots.txt, we do not read its pages for you. If a site refuses our request, we return its answer: we never solve CAPTCHAs, log in to accounts, or try to get around a block or a paywall | | User-Agent | `WebReaderAPI/1.0 (+https://webreaderapi.com/bot/)` | | Endpoint | `GET https://webreaderapi.com/api/v1/read?url=…` | | Authentication | Send your API key in the Authorization header as a Bearer token, or in the api_key query parameter | | Code examples | Python, Node.js and curl, plus tool definitions for MCP, LangChain, LlamaIndex and OpenAI function calling | | Payment | Crypto only: BTC, USDT, USDC, ETH, SOL, LTC (USDT on TRC-20 or ERC-20, USDC on ERC-20) | | Payment model | Top up from $25, then buy any credit pack from your balance. Your balance has no end date and can buy page credits at any time. If a pack costs less than the minimum top-up, the amount you added only to reach that minimum can be sent back on request within 30 days of the payment, in USDT or USDC, as long as you have not spent it. Any other balance is not refunded unless the law says otherwise | | Account | Account-number login; email is optional. You sign in with your account number, so keep it in a safe place. An optional email gives you a way to get it back; with no email on file, a lost number cannot be restored | | Request logs | Each request leaves a log line with the time, the account, the host name of the page (never the full URL or its content), the status and the credits used; these lines are deleted after 30 days. Pages you read are not stored. Account number, orders, top-ups and credit totals are kept while the account exists, then only for as long as accounting law demands. Records go to authorities only when a law obliges us | | Operator | SerpGoat LLC | | Support | Email support answers within 12 hours, every day of the week, including help with crawls and integrations | --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # About WebReaderAPI > About WebReaderAPI: who operates the web reader API, how it turns public pages into Markdown, what it refuses to do, and how to reach a person. No KYC, crypto. - URL: https://webreaderapi.com/about/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) A small API with one job: read a public web page for your software and hand back its text in a form a language model can use. This page covers who runs it, how it works and where its limits are. **Quick answer — What is WebReaderAPI, and who runs it?** WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. **WebReaderAPI** is run by SerpGoat LLC. Prepaid credits start at $3.75 for 10,000 pages, as of October 2026, and never expire. The crawler respects robots.txt and identifies itself by name. ## What problem does WebReaderAPI solve? Language models read text, but web pages are mostly something else: navigation, scripts, cookie notices, ads, related-post widgets and footers. Feeding raw HTML to a model wastes its context window and buries the facts. WebReaderAPI fetches the page, finds the main content and returns it as Markdown, with headings, lists, tables and links kept in place. You also get the title, author, publication date, language and word count when the page states them. One call replaces a fetch step, a browser step and a cleaning step in your own code. ## How does a request work, step by step? 1. Your code sends a URL to `https://webreaderapi.com/api/v1/read` with your API key. 2. We check the site's robots.txt for our token and for all robots, and look for an `ai-input=no` signal. 3. If the page is allowed, our bot fetches it with the user agent `WebReaderAPI/1.0 (+https://webreaderapi.com/bot/)`. 4. When the raw HTML holds too little text, a headless browser renders the page first (`render=auto`). 5. We extract the main content, convert it to Markdown and send it back with its metadata. 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged. The [read endpoint](https://webreaderapi.com/docs/read/) documents every parameter. ## What will WebReaderAPI never do? Some limits are part of the product, not settings: - **It never reads a page a site has refused.** We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged. - **It never works around a refusal.** If a site refuses our request, we return its answer: we never solve CAPTCHAs, log in to accounts, or try to get around a block or a paywall. - **It never reads behind a login or a paywall**, even with credentials you provide. - **It never keeps your pages.** The text goes back to you, and our logs hold only the host name. These rules cost us some pages we could technically fetch. We accept that trade, because site owners, customers and AI assistants all need to trust the tool. ## Who are the customers? Mostly developers who give AI agents a reliable way to read the web, and teams that load documentation, help centres or blogs into retrieval systems. Researchers use it to collect public articles in a clean form, and some tools use it to build reading views or summaries. The [Acceptable Use Policy](https://webreaderapi.com/acceptable-use/) lists what is not allowed, such as collecting personal data about private people. A person looks at every report, and accounts that break the rules are suspended. ## Why crypto and no KYC? Because many of our customers are individual developers, small teams and autonomous projects that prefer not to share identity documents or card details with every API they try. Account-number login; email is optional. You top up from $25 in BTC, USDT, USDC, ETH, SOL, LTC, then buy credits from your balance. No KYC does not mean no rules. The acceptable use policy applies to every account, robots.txt applies to every request, and we act on documented abuse reports. ## What do we not claim? We publish no response-time, uptime or success-rate figures today. Success depends on the sites you read, and an honest figure needs a measured public sample with a method and a date. When we have one, it will appear on the [status page](https://webreaderapi.com/status/). We also do not claim to read every page on the web. Sites that refuse robots, require a login or block automated requests stay out of reach by design. ## How can I reach a person? Write to support@webreaderapi.com for questions about the API, billing or your account; a person answers, every day of the week. Site owners and anyone reporting misuse can write to abuse@webreaderapi.com; the [bot page](https://webreaderapi.com/bot/) explains how to block or slow our crawler. Nobody from our team will ever ask you for your account number or an API key. ## About us: common questions ### Which company operates WebReaderAPI? SerpGoat LLC. The company name appears on invoices in the client area and in the [terms of service](https://webreaderapi.com/terms/). ### Does WebReaderAPI build its own search index? No. It reads the pages its customers send, one URL, a batch or one site at a time. It does not crawl the web for itself and does not rank pages. ### Is the code that extracts the main content public? The API is a hosted service and its internal code is not published. The behaviour you can rely on is documented in the [read endpoint reference](https://webreaderapi.com/docs/read/) and the [response format](https://webreaderapi.com/docs/response/). ### Can I try WebReaderAPI for free? No, there is no free tier. The smallest pack, 10,000 pages for $3.75, gives the same API as the largest one, paid from a $25 top-up. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Acceptable use policy > WebReaderAPI acceptable use policy: what you may read with the web reader API, what is banned, how robots.txt applies and how abuse is handled. No KYC. - URL: https://webreaderapi.com/acceptable-use/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) No identity check does not mean no rules. This policy says what the API may be used for, what it may never be used for, and what happens when someone breaks it. **Quick answer — What may I use WebReaderAPI for?** You may use **WebReaderAPI** to read public web pages for AI agents, retrieval systems, research, monitoring of public information and reading tools. You may not use it for Reading pages to collect personal data about private people, reading content behind a login or a paywall, child sexual abuse material, spam, fraud, and any use that breaks the law where you or we operate. Every request respects robots.txt, and we never help anyone get around a site's refusal. A person reads every abuse report, and accounts found misusing the service are suspended. ## Which uses are welcome? These uses match what the service was built for: - **AI agents and assistants** that read a page to answer a user's question. - **Retrieval and search over public content**, such as documentation, help centres, blogs and public reports. - **Research** on public articles, policies, prices or announcements. - **Monitoring public pages** for changes: terms of service, release notes, product pages, news. - **Accessibility and reading tools** that present a page as clean text. - **Archiving and migrating your own sites** into another format. ## Which uses are forbidden? The following are not allowed on any account, whatever the purpose given: - Reading pages to collect personal data about private people, reading content behind a login or a paywall, child sexual abuse material, spam, fraud, and any use that breaks the law where you or we operate. - Building profiles of private individuals, or harvesting contact details, from any website. - Using the API, its output or several accounts to get around a site's refusal, a rate limit, a challenge page or a robots.txt rule. - Reading content that requires a login, a subscription or a paywall, even with credentials you own. - Reselling raw access to the API as your own reading service without our written agreement. - Feeding the output into spam, phishing, impersonation or fraud. - Sending requests meant to overload or harm a website. ## How do robots.txt and site owners' choices apply to me? They apply to every request automatically, and you cannot switch them off. We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged. If a site declares Content-Signal: ai-input=no in its robots.txt, we do not read its pages for you. If a site refuses our request, we return its answer: we never solve CAPTCHAs, log in to accounts, or try to get around a block or a paywall. If a site you need is out of reach, ask its owner for access or use an official feed or API that the site offers. The [bot page](https://webreaderapi.com/bot/) shows site owners how to allow or refuse our crawler. ## Who is responsible for how the content is used? You are. The pages you read keep their owners' rights: copyright, database rights and any licence the site states. WebReaderAPI delivers the text you asked for; it does not grant you a right to republish it. Check the terms and licence of the sites you read before you store, publish or train on their content. This policy is not legal advice; for advice, consult a lawyer where you operate. ## What happens if someone breaks this policy? We look at the facts first: a person reviews each report before any action. Depending on how serious the case is, we may warn the account, revoke its API keys, suspend it or close it. Credits on an account closed for a breach are lost, as the [terms](https://webreaderapi.com/terms/) state. Content involving the sexual abuse of children is reported to the competent authorities. ## How do I report misuse? Write to abuse@webreaderapi.com with the URL concerned, the date and time, and a line of your server log showing our user agent `WebReaderAPI/1.0 (+https://webreaderapi.com/bot/)`. A short, factual report is easiest to check. We do not reveal to the reporter which customer sent the request, but we act on what we find. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Firecrawl alternative: per-page Markdown credits, paid in crypto > Firecrawl alternative for 2026: web pages to Markdown from $0.07/1K pages, credits that never expire, robots.txt respected, no KYC, crypto. - URL: https://webreaderapi.com/alternatives/firecrawl/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Firecrawl is the reader API most comparison articles start with. This page sets its public plans next to WebReaderAPI's packs, lists what Firecrawl does that we do not, and maps its options to our endpoints for a move. **Quick answer — What is a good Firecrawl alternative that I can pay for in crypto?** **WebReaderAPI** turns public web pages, PDFs and whole sites into Markdown, sold as prepaid credits that never expire, paid in crypto with no KYC. At the prices on this page, set on 2026-10-22, 100,000 static pages cost $21.75; Firecrawl's Standard plan costs $99 a month for 100,000 credits. Firecrawl renders every page for one credit and adds JSON extraction, browser actions and a free tier that we do not offer. ## Firecrawl and WebReaderAPI at a glance | | WebReaderAPI (ours) | Firecrawl | | --- | --- | --- | | Billing | Prepaid packs; credits never expire | Monthly plans; plan credits reset each month; credits bought on top stay until cancellation | | Smallest paid option | 10,000 pages for $3.75 | Hobby: 5,000 credits for $19 a month | | Credits per page | 1 for a static page, 2 when a browser is needed | 1 per page, rendering included | | Free tier | None | 1,000 credits a month, no card; keyless MCP within daily limits | | Structured JSON extraction | Not offered | JSON format with a schema, +4 credits per page | | Browser actions and screenshots | Not offered | Up to 50 actions per request; screenshot format | | Site map endpoint | Not in version 1 | Map: 1 credit per call | | robots.txt | Always respected; no override | Respected by default; ignoreRobotsTxt is Enterprise-only | | Payment | BTC, USDT, USDC, ETH, SOL, LTC | Cards, debit cards and PayPal through Stripe | | Identity checks | None: no ID, no selfie, no documents | None stated in its terms | | Self-hosting | No | Core open source under AGPL-3.0 | *Prices checked on October 22, 2026.* Firecrawl facts from firecrawl.dev and docs.firecrawl.dev, read on 2026-10-22. ## How do the prices compare per 1,000 pages? It depends on whether your pages need a browser. Firecrawl bills one credit per page in every case. WebReaderAPI bills one credit for a page whose HTML holds its text and two when it had to run a browser; with `render=auto`, the browser starts only when needed. | Pages a month | Firecrawl (any page) | WebReaderAPI, static pages | WebReaderAPI, pages needing a browser | |---|---|---|---| | 10,000 | $44 (Hobby $19 + 5,000 extra credits at $5 per 1,000) | $3.75 | 2 × $3.75 | | 100,000 | $99 (Standard) | $21.75 | 2 × $21.75 | | 1,000,000 | $749 (Scale; $599 a month billed yearly) | $74.25 | 2 × $74.25 | At the prices on this page, set on 2026-10-22, both of our columns cost less than Firecrawl's monthly price at these three volumes. Two differences matter more than the totals: Firecrawl's plan credits reset each month while our packs keep their value, and Firecrawl's JSON extraction costs 5 credits per page in all, a feature we do not sell. ## When is Firecrawl the better choice? Choose Firecrawl when you need more than clean Markdown from public pages: - **Structured extraction.** Its JSON format returns fields that match a schema you send, for 4 extra credits per page. We return Markdown and metadata only. - **Browser actions.** Click, wait, scroll and type before reading, up to 50 actions per request. We read pages as they load. - **Screenshots, site maps and change tracking.** All are formats or endpoints on Firecrawl's side. - **A free start.** 1,000 credits a month without a card, and keyless access through its MCP server, capped per IP address each day. - **An ecosystem.** Its integrations page lists LangChain, LlamaIndex, CrewAI, n8n, Zapier and coding agents; OpenClaw ships Firecrawl as its bundled web_fetch provider. - **Self-hosting.** The core is open source under AGPL-3.0. Firecrawl also announced a $75M Series B on 22 September 2026, which funds a fast product roadmap. ## When is WebReaderAPI the better choice? Choose WebReaderAPI if you want a plain reader with predictable costs and no card or identity checks. - **Crypto, no KYC.** Top up from $25 in BTC, USDT, USDC, ETH, SOL, LTC. Account-number login; email is optional. - **Credits that keep.** Prepaid credits never expire: unused pages stay on your account until you use them. A crawl you run twice a year does not pay for eleven idle months. - **Errors cost nothing.** 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged. Firecrawl charges nothing when it returns no document, but a page that answers 403 or 404 still costs it one credit. - **One rule for site owners.** Our crawler checks robots.txt on every read and honours `Content-Signal: ai-input=no`, with no setting to turn that off. Its User-Agent is published on [our crawler page](https://webreaderapi.com/bot/). - **Price per page** at the volumes above, as of 2026-10-22. ## How do Firecrawl options map to the WebReaderAPI endpoints? Single-page reads and crawls move over directly; extraction, actions and maps have no equivalent. | Firecrawl | WebReaderAPI | Note | |---|---|---| | Scrape one page | `GET` or `POST /read` | Same idea: one URL in, one document out | | `formats: ["markdown"]` (default) | `format=markdown` (default) | Main content only by default; `main_content=false` for the full page | | `formats: ["html"]` | `format=html` | Cleaned HTML of the content; no raw HTML in version 1 | | `formats: ["links"]` | `include_links=true` | Links returned as a list | | `formats: ["json"]` with a schema | — | Run your own model on our Markdown | | Actions, screenshot | — | Not offered | | `timeout` in ms (default 60,000) | `timeout` in seconds (5 to 60, default 30) | | | `maxAge` cache (default 2 days) | — | We keep no copy; each read fetches the page | | Crawl with `limit` and `maxDiscoveryDepth` | `POST /crawl` with `limit` (up to 10,000) and `max_depth` | Results paged with a cursor | | Map | — | Not in version 1 | | `ignoreRobotsTxt` (Enterprise) | — | robots.txt is always applied | *Python: the read you used to send as a scrape* ```python import os import requests resp = requests.get( "https://webreaderapi.com/api/v1/read", headers={"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}, params={"url": "https://example.com/changelog", "format": "markdown", "include_links": "true"}, timeout=90, ) if resp.status_code == 403: print("refused by robots.txt or by the site:", resp.json()["error"]["code"]) else: resp.raise_for_status() page = resp.json() print(page["title"], "-", page["credits_used"], "credit(s)") print(len(page.get("links", [])), "links") print(page["markdown"][:400]) ``` ## Will pages that Firecrawl could read also work here? Most public pages will, with one difference in policy. Firecrawl's enhanced mode retries a request through what it calls enhanced proxies after a 401, 403 or 429 answer. WebReaderAPI does not retry around a refusal: when a site says no, you get `502 target_refused` with the site's own status, and no credit is used. Run your 50 most important URLs through both services for a day before moving production traffic. Compare the Markdown, the `rendered` flag and the credits used; that tells you both the quality and the real price for your pages. ## Which pack replaces my Firecrawl plan? Count your pages per month and how many of them need a browser. A Standard-sized workload of 100,000 pages fits 100,000 pages if the pages are static, or two of those packs if every page needs a browser. Credits never expire. | Plan | Quantity | Price | Price per unit | | --- | --- | --- | --- | | Test | 10,000 pages | $3.75 | $0.38/1K pages | | Build | 100,000 pages | $21.75 | $0.22/1K pages | | Volume | 1,000,000 pages | $74.25 | $0.07/1K pages | | Bulk | 5,000,000 pages | $350 | $0.07/1K pages | In every plan: Main-content extraction · JavaScript rendering when needed · Failed and refused pages not charged · Credits never expire **Before you pay** - Minimum top-up $25. Top up once in crypto, then buy any plan from your balance. - Your balance has no end date and can buy page credits at any time. If a pack costs less than the minimum top-up, the amount you added only to reach that minimum can be sent back on request within 30 days of the payment, in USDT or USDC, as long as you have not spent it. Any other balance is not refunded unless the law says otherwise. - Credited once your payment is checked on the blockchain. Typical network confirmation time: BTC ≈ 20 min · USDT TRC-20 ≈ 1 min · USDT ERC-20 ≈ 5 min · USDC ERC-20 ≈ 5 min · ETH ≈ 5 min · SOL ≈ 1 min · LTC ≈ 15 min ## Firecrawl alternative: common questions ### Is there a free Firecrawl alternative? Firecrawl itself has a free tier of 1,000 credits a month. Open-source tools such as Crawl4AI, MarkItDown and Trafilatura are free to run on your own servers; see [Crawl4AI or a hosted API](https://webreaderapi.com/compare/crawl4ai-vs-api/). WebReaderAPI has no free tier: its smallest pack is 10,000 pages for $3.75. ### Can I pay Firecrawl in crypto? Its pricing page names Stripe, with cards, debit cards and PayPal, and no crypto option. WebReaderAPI takes BTC, USDT, USDC, ETH, SOL, LTC only, with no KYC. ### Does WebReaderAPI extract JSON with a schema like Firecrawl? No. WebReaderAPI returns Markdown, text or cleaned HTML with metadata. If you need fields, pass our Markdown to the model of your choice with your schema; Firecrawl's JSON format does this for 4 extra credits per page. ### Do Firecrawl credits expire? According to Firecrawl's billing documentation, plan credits reset each month and do not roll over by default; credits bought on top stay until used but expire if you cancel. WebReaderAPI credits never expire. ### Can I self-host Firecrawl instead? Yes. Its core is open source under AGPL-3.0, and the README says it respects robots.txt by default. The hosted service adds features that the open-source version does not include. ## Sources 1. [Firecrawl pricing](https://www.firecrawl.dev/pricing) — Firecrawl (read 2026-10-22) 2. [Billing](https://docs.firecrawl.dev/billing.md) — Firecrawl docs (read 2026-10-22) 3. [Advanced scraping guide](https://docs.firecrawl.dev/advanced-scraping-guide.md) — Firecrawl docs (read 2026-10-22) 4. [Crawl](https://docs.firecrawl.dev/features/crawl.md) — Firecrawl docs (read 2026-10-22) 5. [Enhanced mode](https://docs.firecrawl.dev/features/enhanced-mode.md) — Firecrawl docs (read 2026-10-22) 6. [Integrations](https://docs.firecrawl.dev/integrations.md) — Firecrawl docs (read 2026-10-22) 7. [Firecrawl keyless launch](https://www.firecrawl.dev/blog/firecrawl-keyless-launch) — Firecrawl (read 2026-10-22) 8. [Introducing Alexandria and our Series B](https://www.firecrawl.dev/blog/introducing-alexandria-series-b) — Firecrawl (read 2026-10-22) 9. [firecrawl/firecrawl repository](https://github.com/firecrawl/firecrawl) — GitHub (read 2026-10-22) 10. [Web tools](https://docs.openclaw.ai/tools/web) — OpenClaw docs (read 2026-10-22) --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Jina Reader alternative: per-page credits instead of tokens > Jina Reader alternative: per-page credits instead of tokens, from $0.07/1K pages, robots.txt always checked, no KYC, crypto. Prices dated 2026-10-22. - URL: https://webreaderapi.com/alternatives/jina-reader/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Jina Reader made the URL-prefix trick famous: put r.jina.ai/ in front of a link and get Markdown back. This page explains how Jina bills, when it is the right tool, and when a per-page price is easier to plan with. **Quick answer — What is a good Jina Reader alternative?** Jina Reader charges for output tokens: $50 for 1 billion, $500 for 11 billion, with 10 million free tokens per new key for non-commercial use (read 22 October 2026). **WebReaderAPI** charges per page instead, from $0.07/1K pages, so a long page costs the same as a short one. It checks robots.txt on every read, sells credits that never expire, and takes crypto with no KYC. ## Jina Reader and WebReaderAPI at a glance | | WebReaderAPI (ours) | Jina Reader | | --- | --- | --- | | How you call it | `GET https://webreaderapi.com/api/v1/read?url=…` with an API key | Prefix the link: `https://r.jina.ai/` + URL; key optional | | Billing unit | Pages: 1 credit, 2 with a browser | Output tokens | | Price | $3.75 for 10,000 pages; from $0.07/1K pages | $50 per 1B tokens; $500 per 11B tokens | | Free use | None | Keyless at 20 requests a minute; 10M tokens per new key, non-commercial only | | Rate limits | Concurrency limit shown in the dashboard | 500 requests a minute with a key; 5,000 on premium | | robots.txt | Checked on every read | Opt-in `X-Robots-Txt` header; default not stated | | Crawl a whole site | `POST /crawl`, up to 10,000 pages | No crawl endpoint stated | | PDF | Yes, up to 100 pages | Yes | | Payment | BTC, USDT, USDC, ETH, SOL, LTC | Cards, Google Pay and PayPal through Stripe | | Owner | SerpGoat LLC | Jina AI, part of Elastic since October 2025 | *Prices checked on October 22, 2026.* Jina facts from jina.ai/reader and its pricing dashboard, read on 2026-10-22. ## How does Jina Reader charge, and what does one page cost? Jina counts the tokens in the response. Its packages are $50 for 1 billion tokens, about $0.05 per million, and $500 for 11 billion, about $0.045 per million. Failed requests are not deducted, and a page requested twice within 5 minutes comes from a cache. The cost of a page therefore depends on its length, and Jina publishes no average. As an **estimate, assuming 5,000 tokens per page**, 1,000 pages cost about $0.25 on the first package and $0.225 on the second. A 2,000-token news brief costs less than that; a 20,000-token documentation page costs four times more. OCR through jina-ocr-v1 costs 40 times more tokens. ## When are per-page credits easier to plan with? When you know how many pages you will read but not how long they are. A crawl of a documentation site, a list of competitor pages to watch, or an agent that reads five links per task all have a page count, not a token count. With WebReaderAPI, 100,000 pages cost $21.75, whatever the length of each page, as of 22 October 2026. Pages that need a browser use two credits, and the `rendered` field says when that happened. Use `max_chars` to cap the size of what comes back; it protects your model's context window but does not change the price. ## When is Jina Reader the better choice? Choose Jina Reader when setup time matters more than anything else, or when your pages are short: - **No setup.** A URL prefix works without a key at 20 requests a minute, which is enough for tests and small scripts. - **Short pages.** With token billing, a short page costs less than a per-page price; at the 5,000-token estimate above, Jina costs less per page than our 10,000 pages pack. - **Fine control through headers.** `X-Target-Selector`, `X-Remove-Selector`, `X-Token-Budget`, `X-Engine` and others shape each read. - **Models and search.** ReaderLM-v2 output and `s.jina.ai` search come from the same account. - **An official MCP server** whose `read_url` tool reads up to five URLs per call. - **Open source.** The Reader branch on GitHub is Apache-2.0, without the hosted storage layer. Jina AI is now part of Elastic, which completed the acquisition on 9 October 2025. ## How do Jina Reader headers map to WebReaderAPI parameters? Most reads move over by changing the URL and adding a key. | Jina Reader | WebReaderAPI | Note | |---|---|---| | `https://r.jina.ai/https://example.com` | `GET https://webreaderapi.com/api/v1/read?url=https://example.com` | Key in the `Authorization` header | | Default Markdown output | `format=markdown` (default) | Main content by default | | `X-Engine: direct` | `render=false` | Plain HTTP read | | Default headless browser | `render=auto` or `render=true` | 2 credits when the browser runs | | `X-Token-Budget` | `max_chars` | Characters, not tokens | | `X-Target-Selector`, `X-Remove-Selector` | — | Main-content detection instead | | `X-Proxy` (country) | `country` | +10 credits, residential connection | | `X-No-Cache` | — | We keep no cache | | `X-Robots-Txt` (opt-in) | — | robots.txt is always checked | | `X-User-Agent` override | — | Our User-Agent is fixed and published | *Python: from the r.jina.ai prefix to the read endpoint* ```python import os import requests # Before: requests.get("https://r.jina.ai/" + url) url = "https://example.com/guide/install" resp = requests.get( "https://webreaderapi.com/api/v1/read", headers={"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}, params={"url": url, "render": "auto", "max_chars": 30000}, timeout=90, ) resp.raise_for_status() page = resp.json() print(page["word_count"], "words, rendered:", page["rendered"], "credits:", page["credits_used"]) print(page["markdown"][:400]) ``` ## What changes for the sites you read? With Jina, robots.txt is checked only when you send the `X-Robots-Txt` header, and you can override the browser User-Agent; its pages do not state the default for either. With WebReaderAPI, both are fixed: every read checks robots.txt for `WebReaderAPI` and for all robots, and every request carries `WebReaderAPI/1.0 (+https://webreaderapi.com/bot/)`. That means some pages Jina can read will be refused here, at no cost to you. In exchange, site owners can see who reads their pages and block or allow us in one line, as our [crawler page](https://webreaderapi.com/bot/) explains. ## What does WebReaderAPI cost instead of tokens? A flat price per page: 10,000 pages for $3.75 up to 5,000,000 pages for $350, as of 22 October 2026. Credits never expire. | Plan | Quantity | Price | Price per unit | | --- | --- | --- | --- | | Test | 10,000 pages | $3.75 | $0.38/1K pages | | Build | 100,000 pages | $21.75 | $0.22/1K pages | | Volume | 1,000,000 pages | $74.25 | $0.07/1K pages | | Bulk | 5,000,000 pages | $350 | $0.07/1K pages | In every plan: Main-content extraction · JavaScript rendering when needed · Failed and refused pages not charged · Credits never expire **Before you pay** - Minimum top-up $25. Top up once in crypto, then buy any plan from your balance. - Your balance has no end date and can buy page credits at any time. If a pack costs less than the minimum top-up, the amount you added only to reach that minimum can be sent back on request within 30 days of the payment, in USDT or USDC, as long as you have not spent it. Any other balance is not refunded unless the law says otherwise. - Credited once your payment is checked on the blockchain. Typical network confirmation time: BTC ≈ 20 min · USDT TRC-20 ≈ 1 min · USDT ERC-20 ≈ 5 min · USDC ERC-20 ≈ 5 min · ETH ≈ 5 min · SOL ≈ 1 min · LTC ≈ 15 min ## Jina Reader alternative: common questions ### Is Jina Reader free? Partly. Without a key, r.jina.ai is free for basic use at 20 requests a minute. Each new key also gets 10 million free tokens, which Jina's pricing page limits to non-commercial use. Paid packages start at $50 for 1 billion tokens. ### Why does a Jina Reader page cost more when the page is long? Because Jina bills the tokens in its response. A long article returns more tokens than a short one. WebReaderAPI bills one credit per page, two with a browser, whatever the length. ### Can I pay Jina Reader in crypto? Jina's pages name Stripe with cards, Google Pay and PayPal, and no crypto option. WebReaderAPI takes BTC, USDT, USDC, ETH, SOL, LTC, with no KYC. ### Does WebReaderAPI have a free keyless mode like r.jina.ai? No. Every read needs a key, and keys come with a paid pack. The smallest pack is 10,000 pages for $3.75. ### Who owns Jina Reader now? Elastic. It completed its acquisition of Jina AI on 9 October 2025, and Jina's legal page says Elastic's data processing terms now apply. ## Sources 1. [Reader](https://jina.ai/reader/) — Jina AI (read 2026-10-22) 2. [API dashboard: pricing](https://jina.ai/api-dashboard/pricing/) — Jina AI (read 2026-10-22) 3. [Legal](https://jina.ai/legal/) — Jina AI (read 2026-10-22) 4. [jina-ai/reader repository](https://github.com/jina-ai/reader) — GitHub (read 2026-10-22) 5. [jina-ai/MCP repository](https://github.com/jina-ai/MCP) — GitHub (read 2026-10-22) 6. [Elastic completes acquisition of Jina AI](https://ir.elastic.co/News--Events/news/news-details/2025/Elastic-Completes-Acquisition-of-Jina-AI-a-Leader-in-Frontier-Models-for-Multimodal-and-Multilingual-Search/default.aspx) — Elastic (2025-10-09, read 2026-10-22) --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Best web reader APIs in 2026: 10 ways to turn URLs into Markdown > Best web reader and URL to Markdown APIs in 2026: 10 options compared on price per 1,000 pages, JavaScript, robots.txt, crypto and KYC. Checked 2026-10-22. - URL: https://webreaderapi.com/best/web-reader-apis/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Ten hosted APIs and open-source tools that turn web pages into Markdown for models, compared on the same criteria from their own pages. We publish WebReaderAPI, which is first on this list; every entry says what that option does better. **Quick answer — What is the best web reader or URL to Markdown API in 2026?** It depends on the job. On 22 October 2026: **WebReaderAPI** for crypto payment, no KYC, robots.txt respected and credits that never expire, from $0.07/1K pages; Firecrawl for the most complete feature set at $0.99 per 1,000 pages on its $99 plan; Jina Reader for zero-setup reads billed by tokens; Crawl4AI if you prefer to self-host under the Apache-2.0 licence. ## The 10 best web reader APIs in 2026 **Methodology.** Published by WebReaderAPI (SerpGoat LLC). Every provider is compared on the same criteria, taken from its own public website: list price per 1,000 pages for plain and browser-rendered pages, free tier and credit expiry, Markdown quality features (main content, PDFs, crawl), robots.txt policy, accepted payment methods, including crypto, identity or card requirements, the date the prices were checked. *Prices checked on October 22, 2026.* ### 1. WebReaderAPI — Our product A web reader API for AI agents with prepaid credits that never expire, paid in crypto with no KYC; its crawler respects robots.txt and never tries to get around a refusal. - Best for: Paying in crypto without documents; agents and RAG pipelines that want polite crawling - Price: $0.38/1K pages on 10,000 pages, $0.07/1K pages on 1,000,000 pages; browser pages use 2 credits - KYC: None - Crypto payment: BTC, USDT, USDC, ETH, SOL, LTC - Pros: Credits never expire - Pros: Failed and robots-refused pages free - Pros: Tool code for MCP, LangChain, LlamaIndex, OpenAI, n8n, OpenClaw - Cons: No free tier - Cons: No structured-JSON extraction or page actions - Cons: Pages a site refuses stay refused ### 2. Firecrawl The best-known API for turning sites into LLM-ready data: scrape, crawl, map and extract, with a large open-source and MCP ecosystem. - Best for: Teams that need structured extraction, page actions and a mature ecosystem - Price: $3.80 per 1,000 on Hobby ($19 for 5,000); $0.99 on Standard ($99 for 100,000); $0.749 on Scale ($749 for 1 million); JavaScript included - KYC: None stated - Crypto payment: No (cards, PayPal) - Pros: 1,000 free credits a month, no card - Pros: Respects robots.txt by default - Pros: Self-hostable core (AGPL-3.0) - Cons: Plan credits reset monthly - Cons: JSON extraction adds 4 credits per page ### 3. Jina Reader Prefix a URL with r.jina.ai and get Markdown back; billed by output tokens and now part of Elastic. - Best for: Quick reads with no setup; short pages - Price: $50 per 1 billion output tokens; $500 per 11 billion (cost per page depends on page length) - KYC: None stated - Crypto payment: Not stated - Pros: Keyless use at 20 requests a minute - Pros: 10 million free tokens per new key (non-commercial) - Cons: Token billing makes long pages costly to predict - Cons: robots.txt only through an opt-in header ### 4. Crawl4AI The most popular open-source crawler for LLMs, with a hosted cloud in soft launch. - Best for: Teams that want to self-host and control everything - Price: Free to self-host; Cloud from $10 (Starter), plain page 0.5 credit, browser page 1 credit - KYC: No card (Cloud) - Crypto payment: Not stated - Pros: Apache-2.0 licence - Pros: Self-hosted MCP server - Cons: Self-hosting means servers and maintenance - Cons: check_robots_txt is off by default ### 5. Olostep An API that renders every page in a browser, with batches, crawls and maps, and per-request crypto for agents. - Best for: JavaScript-heavy sites at volume - Price: $1.80 per 1,000 on Starter ($9 for 5,000); $0.399 on Scale ($399 for 1 million), every page rendered - KYC: Card for top-ups - Crypto payment: Per request, USDC (x402) - Pros: Lowest verified browser-page price at 1 million a month - Pros: Batches, crawls and maps - Cons: Plan credits reset monthly ### 6. Exa contents Page contents from Exa's search index at a flat price, with per-request crypto for agents. - Best for: Agents that already use Exa search - Price: $1.00 per 1,000 pages, flat - KYC: Card for pay as you go - Crypto payment: Per request, USDC (x402) - Pros: $10 free credit a month - Pros: x402 at $0.001 per page - Cons: Cache-first; no documented browser option ### 7. Tavily Extract Extraction from the Tavily search stack, priced per batch of URLs, with an advanced mode. - Best for: Agents built around Tavily search - Price: $1.50 per 1,000 basic URLs on Project ($30); $1.00 on Growth ($500); advanced mode doubles it - KYC: None stated - Crypto payment: Only at its discretion (terms) - Pros: 1,000 credits a month free (5,000 basic URLs) - Cons: Credits reset monthly - Cons: robots.txt policy not stated ### 8. ScrapingBee A general scraping API with a Markdown output option and low plain-page prices at volume. - Best for: High volumes of plain pages - Price: $0.099 per 1,000 plain pages at 1 million a month ($99 Startup); browser pages cost 5 credits - KYC: Card for paid plans - Crypto payment: No - Pros: Low plain-page price at volume - Pros: Markdown and text output - Cons: No credit rollover - Cons: robots.txt policy not stated ### 9. ZenRows A scraping API with Markdown output included and an agent storefront that takes USDC. - Best for: Small agent workloads paid per use - Price: $0.50 per 1,000 plain pages through its agent storefront; $0.199 at 1 million on plans; browser pages cost 5 credits - KYC: Email validation - Crypto payment: Agents only, USDC (x402) - Pros: Markdown at no extra cost - Pros: 5,000 free credits a month - Cons: Accounts cannot pay in crypto - Cons: Plan credits expire at renewal ### 10. Cloudflare Browser Run Cloudflare's browser service with /markdown and /crawl endpoints, billed by browser time. - Best for: Teams already on Cloudflare Workers - Price: $0.09 per browser-hour after 10 included hours, $5 monthly minimum - KYC: Cloudflare account - Crypto payment: USDC through Stripe for Cloudflare billing (one billing page omits it) - Pros: /crawl respects robots.txt and always identifies itself as a bot - Pros: Pay for browser time, not per page - Cons: Cost per page depends on how long each page takes *Disclosure: this page is published by WebReaderAPI. Competitor details come from their public websites at the date shown above and may have changed since.* ## How do I choose a web reader API? Start with your pages, then your volume. 1. **Do your pages need a browser?** News, blogs and documentation usually do not; web apps and some shops do. Credit-based scraping APIs charge 5 to 10 times more for browser pages, while Firecrawl and Olostep price every page the same. 2. **How many pages a month, and how steady?** Monthly plans with large allowances suit steady volume; prepaid credits that do not expire suit agents and one-off crawls. 3. **Do you need extraction beyond Markdown?** Structured JSON, page actions and screenshots are where Firecrawl and the large scraping APIs lead. 4. **How do you want to treat site owners?** Firecrawl, Cloudflare's /crawl and WebReaderAPI respect robots.txt by default; several APIs leave it to you or do not say. 5. **How will you pay?** Most take cards only. Scrape.do, Decodo and Scrapeless state crypto on accounts; Exa, Olostep and others take per-request USDC; WebReaderAPI takes BTC, USDT, USDC, ETH, SOL, LTC with no KYC. ## Should I self-host instead of paying for an API? Self-hosting is free in licence terms and costly in time. Crawl4AI (Apache-2.0), Microsoft MarkItDown (MIT), Trafilatura (Apache-2.0) and Mozilla Readability (Apache-2.0) cover the conversion; you still run the servers, the headless browsers, the retries and the robots.txt handling. Firecrawl's core can also be self-hosted under the AGPL-3.0 licence. For a few thousand pages a month, a hosted API usually costs less than the time spent maintaining a crawler. Our [Crawl4AI or hosted API](https://webreaderapi.com/compare/crawl4ai-vs-api/) page sets out the trade-off. ## What does WebReaderAPI cost? Prepaid credits, one per plain page and two per browser page; credits never expire. Prices set on 22 October 2026. | Plan | Quantity | Price | Price per unit | | --- | --- | --- | --- | | Test | 10,000 pages | $3.75 | $0.38/1K pages | | Build | 100,000 pages | $21.75 | $0.22/1K pages | | Volume | 1,000,000 pages | $74.25 | $0.07/1K pages | | Bulk | 5,000,000 pages | $350 | $0.07/1K pages | In every plan: Main-content extraction · JavaScript rendering when needed · Failed and refused pages not charged · Credits never expire **Before you pay** - Minimum top-up $25. Top up once in crypto, then buy any plan from your balance. - Your balance has no end date and can buy page credits at any time. If a pack costs less than the minimum top-up, the amount you added only to reach that minimum can be sent back on request within 30 days of the payment, in USDT or USDC, as long as you have not spent it. Any other balance is not refunded unless the law says otherwise. - Credited once your payment is checked on the blockchain. Typical network confirmation time: BTC ≈ 20 min · USDT TRC-20 ≈ 1 min · USDT ERC-20 ≈ 5 min · USDC ERC-20 ≈ 5 min · ETH ≈ 5 min · SOL ≈ 1 min · LTC ≈ 15 min ## Best web reader API: common questions ### Which is better, Firecrawl or Jina Reader? Firecrawl suits teams that need crawling, structured extraction and a predictable price per page; Jina Reader suits quick reads with no setup, billed by tokens. Our [Firecrawl vs Jina Reader](https://webreaderapi.com/compare/firecrawl-vs-jina-reader/) page compares them line by line. ### Which web reader API respects robots.txt? Firecrawl respects it by default, with an override only on Enterprise; Cloudflare's /crawl respects it with no documented opt-out; WebReaderAPI respects it on every request. Jina makes it an opt-in header, and many scraping APIs do not state a policy. ### Is there a free URL to Markdown API? Several have free allowances: Firecrawl 1,000 credits a month, Tavily 1,000 credits a month, Exa $10 a month, ZenRows 5,000 credits a month, and Jina 10 million tokens for non-commercial use. WebReaderAPI has no free tier. ### Is this ranking independent? No. It is published by WebReaderAPI, which is ranked first. Every option is described from its own public pages, read on 22 October 2026, against the same criteria, and each entry lists what it does better. ## Sources 1. [Firecrawl pricing](https://www.firecrawl.dev/pricing) — Firecrawl (read 2026-10-22) 2. [Jina API pricing](https://jina.ai/api-dashboard/pricing/) — Jina AI (read 2026-10-22) 3. [Crawl4AI](https://crawl4ai.com/) — Crawl4AI (read 2026-10-22) 4. [Crawl4AI repository](https://github.com/unclecode/crawl4ai) — GitHub (read 2026-10-22) 5. [Olostep pricing](https://www.olostep.com/pricing) — Olostep (read 2026-10-22) 6. [Exa pricing](https://exa.ai/pricing) — Exa (read 2026-10-22) 7. [Tavily API credits](https://docs.tavily.com/documentation/api-credits) — Tavily (read 2026-10-22) 8. [ScrapingBee pricing](https://www.scrapingbee.com/pricing/) — ScrapingBee (read 2026-10-22) 9. [ZenRows pricing](https://docs.zenrows.com/first-steps/pricing.md) — ZenRows (read 2026-10-22) 10. [Browser Run pricing](https://developers.cloudflare.com/browser-run/pricing/) — Cloudflare (read 2026-10-22) --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # WebReaderAPI bot: user agent, robots.txt and how to block it > The WebReaderAPI bot: its exact user agent, its robots.txt token, how to block or slow it, and what it never does. Written for site owners and AI tools. - URL: https://webreaderapi.com/bot/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) If you saw our name in your server logs, this page explains who we are, why we fetched your page and how to tell us not to. Every rule here is a commitment the service is built around. **Quick answer — What is the WebReaderAPI bot, and how do I block it?** The **WebReaderAPI** bot fetches public pages when one of our customers asks the API to read them. It sends the user agent `WebReaderAPI/1.0 (+https://webreaderapi.com/bot/)` and obeys robots.txt rules written for the token `WebReaderAPI` or for all robots. To block it everywhere, add `User-agent: WebReaderAPI` followed by `Disallow: /` to your robots.txt. ## The bot at a glance | Attribute | Value | | --- | --- | | User-Agent header | `WebReaderAPI/1.0 (+https://webreaderapi.com/bot/)` | | robots.txt token | `WebReaderAPI` | | robots.txt | We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged | | AI preference signals | If a site declares Content-Signal: ai-input=no in its robots.txt, we do not read its pages for you | | When a site says no | If a site refuses our request, we return its answer: we never solve CAPTCHAs, log in to accounts, or try to get around a block or a paywall | | Copies of your pages | None kept after the answer is sent to the customer | | Crawl jobs | Crawl one website per job, up to 10,000 pages, following its sitemap, its robots.txt and its Crawl-delay | | Contact | abuse@webreaderapi.com | ## Why did WebReaderAPI fetch my page? Because a customer of the API asked for it. WebReaderAPI does not build its own index of the web and does not browse on its own initiative. Each fetch belongs to one of three request types: - **A single read:** an application or an AI agent sent your URL to `/read`, usually because a person asked a question that your page answers. - **A batch job:** a customer sent a list of up to 100 URLs, and yours was on it. - **A crawl job:** a customer asked us to read a whole site, starting from one address and staying on that domain. Crawls follow your sitemap and your robots.txt. The customer receives the main text of the page as Markdown. We keep no copy of it afterwards. ## How do I block the bot on my whole site? Add a group for our token to your robots.txt file. The bot reads `/robots.txt` before fetching a page and caches it for at most 24 hours, so a change applies within a day at the latest. *robots.txt: refuse every page to WebReaderAPI* ```text User-agent: WebReaderAPI Disallow: / ``` ## Can I block only part of my site, or slow the bot down? Yes. List the folders you want to keep out under `Disallow`, and set a `Crawl-delay` in seconds for crawl jobs. Rules written for all robots (`User-agent: *`) also apply to us when you have no group for our token. When you do have one, that group wins, as RFC 9309 describes. *robots.txt: keep two folders out and space out crawl requests* ```text User-agent: WebReaderAPI Disallow: /account/ Disallow: /drafts/ Crawl-delay: 5 ``` ## Does the bot honour AI preference signals? Yes, the ones a site states in robots.txt. If your file declares `Content-Signal: ai-input=no`, we treat it as a refusal to have your pages used as input for AI answers, and we do not read them for our customers. The API then returns the error `blocked_by_site_policy` to the customer, who is not charged. *robots.txt: allow search engines, refuse AI input* ```text User-agent: * Content-Signal: search=yes, ai-input=no Allow: / ``` ## What does the bot do when my site refuses it? It stops and reports your answer. A 403 or 429 status, a challenge page, a login form or a paywall all end the request. The customer gets the error `target_refused` with your status code, and is not charged for it. The bot never solves challenges, never logs in, never changes its user agent to look like a browser and never retries in disguise. If you want us to read your pages, allow the token in robots.txt and let the requests through; if not, nothing more is needed on your side. ## What does WebReaderAPI not promise yet? We do not publish a list of IP addresses today, so please identify the bot by its user agent and its robots.txt behaviour, not by IP. Pages read with the `country` option leave from an address in the chosen country, still with the same user agent. We will add our address ranges to this page once they are fixed, with the date of the change. ## How do I report a problem with the bot? Write to abuse@webreaderapi.com with the date and time, your domain, the URL fetched and a line of your access log showing our user agent. A person reads each report and checks it against our logs. If a customer uses the API against our [Acceptable Use Policy](https://webreaderapi.com/acceptable-use/), their account is suspended. To check whether a request really came from us, compare the user agent and the robots.txt behaviour with this page. A client that ignores your robots.txt rules while using our name is not our bot. ## Site owners' questions ### Is the WebReaderAPI bot a search engine crawler? No. It does not index the web or rank pages. It reads a page only when a customer of the API asks for that page, in a single request, a batch or a crawl of one site. ### Will blocking the bot affect my Google or Bing rankings? No. Rules written for `WebReaderAPI` concern only our bot. Search engine crawlers follow their own groups in your robots.txt. ### How fast does a robots.txt change take effect? Within 24 hours at the latest. The bot caches robots.txt for at most a day before reading it again. ### Does the bot respect Crawl-delay? Yes, for crawl jobs, which read many pages of one site. A single read is one request and is not spaced out. ### Do you store or sell the content of my pages? No. The page text goes to the customer who asked for it and is not kept by us. Our logs hold the host name, the time, the status and the credits used, for 30 days. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Changelog > WebReaderAPI changelog: dated changes to the web reader API, its prices, its crawler rules and its policies. First entry: site and documentation published. - URL: https://webreaderapi.com/changelog/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Changes to the API, prices, crawler behaviour and policies are listed here, newest first. A change to a promise made elsewhere on the site appears here too. **Quick answer — What has changed at WebReaderAPI?** On 22 October 2026, **WebReaderAPI** published its website, documentation, prices and policies. The API itself is not open yet; its launch will be the next entry on this page. Prices start at $3.75 for 10,000 pages and fall to $0.07/1K pages on the largest pack. ## 22 October 2026: site and documentation published - **Website, prices and policies** went online: [pricing](https://webreaderapi.com/pricing/), [terms](https://webreaderapi.com/terms/), [privacy](https://webreaderapi.com/privacy/), [acceptable use](https://webreaderapi.com/acceptable-use/) and [refunds](https://webreaderapi.com/refunds/). - **Documentation** for the `/read`, `/batch`, `/crawl` and `/credits` endpoints, with code for Python, Node.js and curl. - **Crawler rules** published on the [bot page](https://webreaderapi.com/bot/): user agent `WebReaderAPI/1.0 (+https://webreaderapi.com/bot/)`, robots.txt token `WebReaderAPI`, respect for `ai-input=no`. - **API status:** launch pending. Until the launch entry appears here, no API key can be bought. ## What counts as a change worth a dated entry? Anything a customer or a site owner could rely on: - a new endpoint, parameter or response field; - a field renamed or removed (which only happens in a new version of the API, such as `/api/v2/`); - a price, a credit rule or a refund rule; - a change to how the crawler identifies itself or treats robots.txt; - an incident that affected requests, with its time window. ## How are price changes handled? A new price applies to packs bought after it is published here. Credits you already hold keep their value: they are counted in pages, not in dollars, and Prepaid credits never expire: unused pages stay on your account until you use them. ## How stable is version 1 of the API? Within version 1, we may add fields and parameters, never rename or remove them. Code that reads the fields documented in the [response format](https://webreaderapi.com/docs/response/) keeps working. A breaking change would go to a new version, announced here before it goes live. ## How do I follow the changes? Check this page, or read its Markdown version, which [llms.txt](https://webreaderapi.com/llms.txt) lists with the other pages of the site. Each entry keeps its date, so you can see what changed since your last visit. Questions go to support@webreaderapi.com. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Crawl4AI or a hosted API? When to run your own web reader > Crawl4AI or a hosted API for web pages to Markdown? Licences, robots.txt defaults, what self-hosting costs, and hosted prices checked on 2026-10-22. - URL: https://webreaderapi.com/compare/crawl4ai-vs-api/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Open-source tools can turn web pages into Markdown for free, on your own servers. Hosted APIs charge per page and run the servers for you. This page sets out what each choice really costs, from licences and robots.txt defaults to browsers and maintenance. **Quick answer — Should I self-host Crawl4AI or use a hosted web reader API?** Self-host Crawl4AI (Apache-2.0, with an attribution requirement) when you have steady volume and people to run browsers and servers; note that its robots.txt check is off by default. Use a hosted API when volume is uneven or you want no operations work: Crawl4AI Cloud starts at $10 for 10,000 credits, and **WebReaderAPI** at $3.75 for 10,000 pages, paid in crypto with no KYC (22 October 2026). ## Which open-source tools turn web pages into Markdown? The main options in October 2026, with the licence and defaults their own repositories and docs state. | Tool | Licence | Renders JavaScript | robots.txt by default | MCP server | | --- | --- | --- | --- | --- | | Crawl4AI | Apache-2.0, plus a README attribution requirement | Yes (Playwright) | Off: `check_robots_txt` defaults to False | Yes, on your own server | | Firecrawl (self-hosted) | AGPL-3.0 core; SDKs and some UI parts MIT | Yes (Playwright service) | On | Yes (firecrawl-mcp-server, MIT) | | Microsoft MarkItDown | MIT | No | Not documented | Yes (markitdown-mcp) | | Trafilatura | Apache-2.0 (GPLv3+ before v1.8.0) | No | Its spider follows the Robots Exclusion Standard | None mentioned | | Mozilla Readability | Apache-2.0 | No; needs a DOM and does not fetch | Not applicable | None found | | Jina Reader (open-source branch) | Apache-2.0 | Yes (Puppeteer) | Not mentioned | Not in the repository | *Prices checked on October 22, 2026.* From the GitHub repositories and documentation listed below, read on 2026-10-22. ## What does self-hosting a web reader really cost? The software is free; running it is not. Before you compare a hosted price with zero, count these: - **Servers and browsers.** Static pages need little CPU, but JavaScript pages need a headless browser per request, with memory to match. Peaks need spare capacity that sits idle the rest of the time. - **Maintenance.** Browser versions, extraction rules and dependencies change. Crawl4AI shipped v0.9.5 on 14 October 2026; MarkItDown 0.1.9 on 21 October 2026. Someone has to test and deploy each update. - **Network.** Outbound bandwidth, DNS, timeouts and retries are yours to tune. Sites can refuse traffic from your servers' addresses, and the right answer then is to respect the refusal. - **Policy.** Crawl4AI leaves `check_robots_txt` off unless you turn it on, and lets you set or randomise the user agent. Deciding and enforcing your own robots.txt and identification rules is part of the job. - **Security.** MarkItDown's README warns that it performs I/O with the privileges of the current process and that inputs should be sanitised in untrusted environments. A service that fetches arbitrary URLs must also refuse private and local addresses. ## What do hosted readers charge in comparison? Hosted APIs turn those costs into a price per page. Two examples from 22 October 2026: | | Crawl4AI Cloud | WebReaderAPI (ours) | |---|---|---| | Entry | Starter $10 a month, 10,000 credits | $3.75 for 10,000 pages | | Static page | 0.5 credit | 1 credit | | Page with a browser | 1 credit ("real browser") | 2 credits | | Larger plans | Builder $99 for 125,000 credits; Scale $499 for 750,000 | $74.25 for 1,000,000 pages; $350 for 5,000,000 pages | | Unused credits | Monthly credits do not roll over; extra credits never expire | Never expire | | Payment | Card through Stripe | BTC, USDT, USDC, ETH, SOL, LTC | | robots.txt | Customers must honour sites' robots directives (terms) | Checked by the service on every read | Crawl4AI Cloud is run by the Crawl4AI maintainers and marks its prices as a "soft launch" that may change. Its credit is worth $0.001, so a static page costs $0.50 per 1,000 on Starter. ## When should I self-host? Self-hosting pays off when three things are true: your volume is high and steady, you already run servers and browsers, and you need control that an API cannot give. - **Custom extraction.** Crawl4AI's CSS, XPath and LLM schemas, or your own rules on top of Readability or Trafilatura. - **Data residency.** Pages never leave your infrastructure. - **Inputs beyond web pages.** MarkItDown converts Word, Excel, PowerPoint, EPub and more, which a web reader API does not. - **Licence fit.** Apache-2.0 and MIT tools allow closed products; Firecrawl's AGPL-3.0 core asks you to share changes if you offer it as a network service. ## When is a hosted API the better choice? Buy pages instead of running servers when your workload is uneven, your team is small, or the reader is a tool inside an agent rather than a product of its own. - **Spiky or seasonal volume.** A crawl twice a year does not justify a browser fleet. With WebReaderAPI, Prepaid credits never expire: unused pages stay on your account until you use them. - **Agents.** A hosted endpoint plugs into MCP, LangChain or OpenAI tools in minutes; see [our MCP server](https://webreaderapi.com/docs/mcp/). - **A policy you do not have to build.** We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged. If a site declares Content-Signal: ai-input=no in its robots.txt, we do not read its pages for you. - **Payment without a card.** WebReaderAPI takes BTC, USDT, USDC, ETH, SOL, LTC with no KYC. ## Can I combine both? Yes, and many teams do. Run an open-source converter for files you already hold, such as PDFs and Office documents with MarkItDown, and call a hosted reader for live web pages, especially JavaScript pages that would otherwise need your own browsers. Keep the output format the same on both sides, Markdown with a source URL and a title, so your chunking and indexing code does not care where a document came from. Our [RAG guide](https://webreaderapi.com/use-cases/rag/) shows that format in practice. ## What does a hosted reader cost with WebReaderAPI? Per-page credits paid in crypto, as of 22 October 2026: 1 credit per static page, 2 with a browser, nothing for failed or refused pages. Credits never expire. | Plan | Quantity | Price | Price per unit | | --- | --- | --- | --- | | Test | 10,000 pages | $3.75 | $0.38/1K pages | | Build | 100,000 pages | $21.75 | $0.22/1K pages | | Volume | 1,000,000 pages | $74.25 | $0.07/1K pages | | Bulk | 5,000,000 pages | $350 | $0.07/1K pages | In every plan: Main-content extraction · JavaScript rendering when needed · Failed and refused pages not charged · Credits never expire **Before you pay** - Minimum top-up $25. Top up once in crypto, then buy any plan from your balance. - Your balance has no end date and can buy page credits at any time. If a pack costs less than the minimum top-up, the amount you added only to reach that minimum can be sent back on request within 30 days of the payment, in USDT or USDC, as long as you have not spent it. Any other balance is not refunded unless the law says otherwise. - Credited once your payment is checked on the blockchain. Typical network confirmation time: BTC ≈ 20 min · USDT TRC-20 ≈ 1 min · USDT ERC-20 ≈ 5 min · USDC ERC-20 ≈ 5 min · ETH ≈ 5 min · SOL ≈ 1 min · LTC ≈ 15 min ## Self-hosting a web reader: common questions ### Is Crawl4AI free for commercial use? Its README says it is licensed under Apache-2.0, which allows commercial use, and adds that users must include an attribution, either a badge or a text line. Read the licence and README yourself before you ship. ### Does Crawl4AI respect robots.txt? Only if you ask. Its `check_robots_txt` parameter defaults to False; set it to True to check and cache robots.txt rules before crawling. You can also set or randomise the user agent, so your own policy decides how it behaves. ### Can I self-host Firecrawl instead of Crawl4AI? Yes. Firecrawl's core is open source under AGPL-3.0 and respects robots.txt by default. The AGPL asks you to publish changes when you offer the software as a network service, which matters for commercial products. ### Which tool converts PDFs and Office files? Microsoft MarkItDown, under the MIT licence, converts PDF, Word, PowerPoint, Excel, HTML, EPub and more into Markdown. It does not render JavaScript pages. WebReaderAPI reads PDFs from a URL, up to 100 pages. ### How much does Crawl4AI Cloud cost? On 22 October 2026: Starter $10 a month for 10,000 credits, Builder $99 for 125,000 and Scale $499 for 750,000, with a static page at 0.5 credit and a browser page at 1. The site marks these prices as a soft launch that may change. ## Sources 1. [unclecode/crawl4ai repository](https://github.com/unclecode/crawl4ai) — GitHub (read 2026-10-22) 2. [Crawl4AI parameters](https://docs.crawl4ai.com/api/parameters/) — Crawl4AI docs (read 2026-10-22) 3. [Crawl4AI self-hosting](https://docs.crawl4ai.com/core/self-hosting/) — Crawl4AI docs (read 2026-10-22) 4. [crawl4ai on PyPI](https://pypi.org/project/crawl4ai/) — PyPI (read 2026-10-22) 5. [Crawl4AI Cloud](https://crawl4ai.com/) — Crawl4AI (read 2026-10-22) 6. [microsoft/markitdown repository](https://github.com/microsoft/markitdown) — GitHub (read 2026-10-22) 7. [markitdown on PyPI](https://pypi.org/project/markitdown/) — PyPI (read 2026-10-22) 8. [adbar/trafilatura repository](https://github.com/adbar/trafilatura) — GitHub (read 2026-10-22) 9. [mozilla/readability repository](https://github.com/mozilla/readability) — GitHub (read 2026-10-22) 10. [firecrawl/firecrawl repository](https://github.com/firecrawl/firecrawl) — GitHub (read 2026-10-22) 11. [jina-ai/reader repository](https://github.com/jina-ai/reader) — GitHub (read 2026-10-22) --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Firecrawl vs Jina Reader (2026): which web reader fits your project? > Firecrawl vs Jina Reader in 2026: per-page credits vs token billing, free tiers, crawling, JSON extraction, robots.txt and payment. Checked 2026-10-22. - URL: https://webreaderapi.com/compare/firecrawl-vs-jina-reader/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Firecrawl and Jina Reader are the two names that come up first when developers ask how to turn web pages into Markdown for a model. They bill, crawl and treat robots.txt differently. This page compares them on the same facts, with our own service in a separate, labelled column. **Quick answer — Firecrawl or Jina Reader: which should I use?** Use Firecrawl for full pipelines: it bills 1 credit per page with rendering included, crawls and maps sites, and extracts JSON to a schema, on plans from $19 to $749 a month (22 October 2026). Use Jina Reader for quick reads: prefix a URL with r.jina.ai, pay $50 per billion output tokens, or nothing at 20 requests a minute. **WebReaderAPI**, which publishes this page, sells per-page credits in crypto with no KYC. ## Firecrawl vs Jina Reader, with WebReaderAPI for reference Each row comes from the provider's own pages, read on 22 October 2026. The third column is our service, published by a competitor of both. | | Firecrawl | Jina Reader | WebReaderAPI (ours) | | --- | --- | --- | --- | | Billing unit | Credits: 1 per page, rendering included | Output tokens | Credits: 1 per page, 2 with a browser | | Entry price | Hobby: $19 a month for 5,000 credits | $50 for 1B tokens | $3.75 for 10,000 pages | | Larger volumes | Standard $99 for 100K; Scale $749 for 1M a month | $500 for 11B tokens | $74.25 for 1,000,000 pages | | Free use | 1,000 credits a month, no card; keyless MCP, capped per IP each day | Keyless at 20 requests a minute; 10M tokens per new key, non-commercial | None | | Unused credits | Plan credits reset monthly | Validity not shown in the text we could read | Never expire | | Crawl and map | Crawl (default limit 10,000 pages) and map | No crawl endpoint stated | Crawl up to 10,000 pages; no map | | Structured extraction | JSON with a schema, +4 credits per page | JSON schema or instruction headers | Not offered | | robots.txt | Respected by default; override Enterprise-only | Opt-in `X-Robots-Txt` header | Always respected | | Payment | Cards, debit cards, PayPal (Stripe) | Cards, Google Pay, PayPal (Stripe) | BTC, USDT, USDC, ETH, SOL, LTC | | Open source | Core under AGPL-3.0 | Reader branch under Apache-2.0 | No | | Company | Firecrawl; $75M Series B on 22 September 2026 | Jina AI, part of Elastic since October 2025 | SerpGoat LLC | *Prices checked on October 22, 2026.* Sources at the bottom of the page. ## Which one costs less per page? It depends on page length and volume, because the two bill in different units. Firecrawl charges per page: $0.99 per 1,000 pages on Standard, $0.749 on Scale with monthly billing, rendering included. Jina charges per output token, and publishes no average per page. As an **estimate, assuming 5,000 tokens per page**, Jina costs about $0.25 per 1,000 pages on its $50 package and $0.225 on the $500 package. On that assumption Jina costs less per page than Firecrawl at every volume. For pages far longer than 5,000 tokens, such as long documentation pages, the gap closes or reverses; for short pages it widens. Firecrawl's JSON extraction adds 4 credits per page, which changes the comparison for extraction work. ## Which one is better for crawling a whole site? Firecrawl. Its crawl endpoint follows the sitemap by default, takes a page limit (10,000 by default) and a discovery depth, and returns 402 if your credits cannot cover the limit. A separate map endpoint lists a site's URLs for one credit per call. Jina Reader is a single-URL reader plus a search endpoint; its pages state no crawl feature. To read a whole site with Jina, you collect the URLs yourself, for example from the sitemap, and call the reader once per page. ## How do they treat robots.txt and site owners? Firecrawl respects robots.txt by default and lets only Enterprise customers turn that off; site owners allow or refuse it with the `FirecrawlAgent` token. Its documentation warns that this token may not be the User-Agent header of the fetch itself. Jina checks robots.txt when the request carries the `X-Robots-Txt` header, which names the bot to check as. It also lets the caller override the browser User-Agent. Its pages do not state the default for either. Both approaches leave much of the choice to the customer; ours does not, as the next section explains. ## Which one fits AI agents better? Both serve agents well, in different ways. Firecrawl offers a hosted MCP server, keyless access with a daily cap per IP, and a long list of integrations, including LangChain, LlamaIndex, CrewAI and n8n; OpenClaw bundles Firecrawl as its web_fetch provider. Jina offers an official remote MCP server whose `read_url` tool reads up to five URLs per call, an n8n credential, and the URL prefix that any tool able to make a GET request can use. For an agent that reads a few pages per task, Jina's prefix is the simplest start; for an agent that crawls, maps and extracts, Firecrawl covers more. ## Where does WebReaderAPI fit between the two? WebReaderAPI sits closer to Jina in scope, a reader, and closer to Firecrawl in billing, per page, with a crawl endpoint for whole sites. What sets it apart is the payment and the policy rather than the feature list: - **Crypto with no KYC.** Top up from $25 in BTC, USDT, USDC, ETH, SOL, LTC. - **Credits that never expire**, sold in packs from $3.75. - **A fixed policy for sites.** We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged. If a site refuses our request, we return its answer: we never solve CAPTCHAs, log in to accounts, or try to get around a block or a paywall. It does less than Firecrawl: no JSON extraction, no browser actions, no map, no free tier. Read the [Firecrawl alternative](https://webreaderapi.com/alternatives/firecrawl/) and [Jina Reader alternative](https://webreaderapi.com/alternatives/jina-reader/) pages for the details of each switch. ## Which should I choose for my use case? - **Prototype or hobby agent:** Jina Reader without a key, or Firecrawl's free tier. - **RAG over documentation sites:** Firecrawl for crawl and map; WebReaderAPI if you want per-page credits that keep and crypto payment. - **Extraction into fixed fields:** Firecrawl's JSON format. - **Many short news or blog pages:** Jina's token billing tends to cost less. - **Strict respect of site owners' choices, enforced by the provider:** WebReaderAPI. - **No card and no identity documents:** WebReaderAPI; neither Firecrawl nor Jina lists a crypto payment option on the pages we read. ## What does the third option cost? Per-page packs paid in crypto, as of 22 October 2026. A page that needs a browser uses two credits; failed pages and pages refused by robots.txt are free. | Plan | Quantity | Price | Price per unit | | --- | --- | --- | --- | | Test | 10,000 pages | $3.75 | $0.38/1K pages | | Build | 100,000 pages | $21.75 | $0.22/1K pages | | Volume | 1,000,000 pages | $74.25 | $0.07/1K pages | | Bulk | 5,000,000 pages | $350 | $0.07/1K pages | In every plan: Main-content extraction · JavaScript rendering when needed · Failed and refused pages not charged · Credits never expire **Before you pay** - Minimum top-up $25. Top up once in crypto, then buy any plan from your balance. - Your balance has no end date and can buy page credits at any time. If a pack costs less than the minimum top-up, the amount you added only to reach that minimum can be sent back on request within 30 days of the payment, in USDT or USDC, as long as you have not spent it. Any other balance is not refunded unless the law says otherwise. - Credited once your payment is checked on the blockchain. Typical network confirmation time: BTC ≈ 20 min · USDT TRC-20 ≈ 1 min · USDT ERC-20 ≈ 5 min · USDC ERC-20 ≈ 5 min · ETH ≈ 5 min · SOL ≈ 1 min · LTC ≈ 15 min ## Firecrawl vs Jina Reader: common questions ### Is Jina Reader free and Firecrawl paid? Both have free use. Jina Reader works without a key at 20 requests a minute, and each new key gets 10 million tokens for non-commercial use. Firecrawl gives 1,000 credits a month without a card, plus keyless MCP access capped per IP address each day. ### Which one returns better Markdown? We publish no quality benchmark, and neither vendor's pricing page claims one against the other. Run your own sample: 50 of your real URLs through each service, then compare headings, tables and leftover menus in the output. ### Can either be self-hosted? Yes. Firecrawl's core is open source under AGPL-3.0. Jina publishes an open-source branch of its Reader under Apache-2.0, without the hosted storage layer. ### Do Firecrawl or Jina take crypto? Not on the pricing and billing pages we read on 22 October 2026: both name Stripe, with cards and PayPal, and Jina also Google Pay. WebReaderAPI takes BTC, USDT, USDC, ETH, SOL, LTC. ### Is this comparison neutral? It is written by WebReaderAPI, a competitor of both, and says so at the top. Every fact about Firecrawl and Jina comes from their own pages, listed below with the date we read them. ## Sources 1. [Firecrawl pricing](https://www.firecrawl.dev/pricing) — Firecrawl (read 2026-10-22) 2. [Billing](https://docs.firecrawl.dev/billing.md) — Firecrawl docs (read 2026-10-22) 3. [Crawl](https://docs.firecrawl.dev/features/crawl.md) — Firecrawl docs (read 2026-10-22) 4. [Advanced scraping guide](https://docs.firecrawl.dev/advanced-scraping-guide.md) — Firecrawl docs (read 2026-10-22) 5. [MCP server](https://docs.firecrawl.dev/mcp-server.md) — Firecrawl docs (read 2026-10-22) 6. [Introducing Alexandria and our Series B](https://www.firecrawl.dev/blog/introducing-alexandria-series-b) — Firecrawl (read 2026-10-22) 7. [Reader](https://jina.ai/reader/) — Jina AI (read 2026-10-22) 8. [API dashboard: pricing](https://jina.ai/api-dashboard/pricing/) — Jina AI (read 2026-10-22) 9. [jina-ai/MCP repository](https://github.com/jina-ai/MCP) — GitHub (read 2026-10-22) 10. [Elastic completes acquisition of Jina AI](https://ir.elastic.co/News--Events/news/news-details/2025/Elastic-Completes-Acquisition-of-Jina-AI-a-Leader-in-Frontier-Models-for-Multimodal-and-Multilingual-Search/default.aspx) — Elastic (2025-10-09, read 2026-10-22) 11. [Web tools](https://docs.openclaw.ai/tools/web) — OpenClaw docs (read 2026-10-22) --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Web reader API pricing compared: what 10K, 100K and 1M pages cost > Web reader and URL-to-Markdown API pricing compared on 2026-10-22: the monthly bill for 10K, 100K and 1M pages at 13 APIs, plain and JS pages. - URL: https://webreaderapi.com/compare/web-reader-api-pricing/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Credit prices hide plan sizes, browser multipliers and expiry. This page turns public list prices into the monthly bill for three volumes, for plain pages and for pages that need a browser, then lists the costs that pricing pages leave out. **Quick answer — How do web reader API prices compare in 2026?** On 2026-10-22, 100,000 browser-rendered pages a month cost $99 at eight APIs, including Firecrawl, Olostep and Crawl4AI Cloud; plain pages cost from $29 (Scrape.do). At 1 million pages, plain pages start at about $99 and browser pages at $399 (Olostep). **WebReaderAPI** charges $21.75 for 100,000 plain pages and $43.50 if all of them need a browser, paid in crypto with no KYC. ## What does a month of pages cost at each API? Monthly bill for 10,000, 100,000 and 1,000,000 pages, using the least expensive published plan, overage or pay-as-you-go rate. Where a provider prices browser pages differently, both figures are shown (plain / browser). | API | 10,000 pages | 100,000 pages | 1,000,000 pages | | --- | --- | --- | --- | | WebReaderAPI, plain pages | $3.75 | $21.75 | $74.25 | | WebReaderAPI, browser pages (2 credits) | $7.50 | $43.50 | $148.50 | | Firecrawl | $44.00 | $99.00 | $749.00 | | Crawl4AI Cloud, plain / browser | $10.00 | $50.00 / $99.00 | $395.00 / $667.00 | | Olostep | $29.00 | $99.00 | $399.00 | | Tavily Extract, plain / browser | $16.00 / $30.00 | $140.00 / $236.00 | $1,300 / $2,900 | | Exa contents | $10.00 | $100.00 | $1,000 | | Linkup fetch, plain / browser | $10.00 / $50.00 | $100.00 / $500.00 | $1,000 / $5,000 | | ScrapingBee, plain / browser | $19.00 | $49.00 / $99.00 | $99.00 / $599.00 | | Scrape.do, plain / browser | $29.00 | $29.00 / $99.00 | $99.00 / $699.00 | | ZenRows, plain / browser | $5.00 / $25.00 | $39.00 / $99.00 | $199.00 / $549.00 | | Scrapfly, plain / browser | $30.00 | $30.00 / $100.00 | $100.00 / $560.00 | | Bright Data Web Unlocker | $15.00 | $150.00 | $1,300 | | Diffbot | $299.00 | $299.00 | $899.00 | *Prices checked on October 22, 2026.* List prices in USD, read on 2026-10-22. "—" = not published or not self-serve at that volume. Exa and Parallel document no browser option. Full method and 21 APIs: [URL to Markdown API price index 2026](https://webreaderapi.com/data/url-to-markdown-price-index-2026/). ## Which costs do pricing pages leave out? 1. **Browser multipliers.** A JavaScript page costs 5 credits at ScrapingBee, ZenRows and Scrape.do, 6 at Scrapfly and 10 at ScraperAPI; Firecrawl and Olostep render every page at one price. 2. **Plan steps.** A $99 plan bought for 20,000 pages still costs $99. 3. **Expiry.** Most monthly plans reset unused credits; a quiet month is lost money. 4. **Extraction add-ons.** Structured JSON extraction costs extra at several APIs: 4 more credits per page at Firecrawl. 5. **Premium connections.** Residential or "premium" proxies multiply the credit cost at most scraping APIs; WebReaderAPI adds 10 credits only when you choose a country. 6. **Failed pages.** Check whether errors are billed. WebReaderAPI charges nothing for failed pages or pages refused by robots.txt. ## Where does WebReaderAPI sit? At the prices on this page, set on 2026-10-22, WebReaderAPI charges $0.38/1K pages per plain page at 10,000 pages, $0.22/1K pages at 100,000 and $0.07/1K pages at 1 million; a browser page uses two credits. Credits are prepaid, never expire and are paid in BTC, USDT, USDC, ETH, SOL, LTC with no KYC. We do not claim to be the cheapest at every volume or for every page type. Our crawler respects robots.txt and never tries to get around a refusal, so some pages that other tools fetch will come back refused, and free. ## What does WebReaderAPI cost? Prepaid packs; one credit per plain page, two with a browser; credits never expire. Prices set on 2026-10-22. | Plan | Quantity | Price | Price per unit | | --- | --- | --- | --- | | Test | 10,000 pages | $3.75 | $0.38/1K pages | | Build | 100,000 pages | $21.75 | $0.22/1K pages | | Volume | 1,000,000 pages | $74.25 | $0.07/1K pages | | Bulk | 5,000,000 pages | $350 | $0.07/1K pages | In every plan: Main-content extraction · JavaScript rendering when needed · Failed and refused pages not charged · Credits never expire **Before you pay** - Minimum top-up $25. Top up once in crypto, then buy any plan from your balance. - Your balance has no end date and can buy page credits at any time. If a pack costs less than the minimum top-up, the amount you added only to reach that minimum can be sent back on request within 30 days of the payment, in USDT or USDC, as long as you have not spent it. Any other balance is not refunded unless the law says otherwise. - Credited once your payment is checked on the blockchain. Typical network confirmation time: BTC ≈ 20 min · USDT TRC-20 ≈ 1 min · USDT ERC-20 ≈ 5 min · USDC ERC-20 ≈ 5 min · ETH ≈ 5 min · SOL ≈ 1 min · LTC ≈ 15 min ## Web reader API pricing: common questions ### What is the cheapest URL to Markdown API in 2026? For plain pages at 1 million a month, ScrapingBee ($0.099 per 1,000) and Scrape.do ($0.099 per 1,000) listed the lowest prices on 2026-10-22. For browser pages at that volume, Olostep listed $0.399. The answer changes with volume, so compare at your own. ### Is Firecrawl expensive? At small volumes, yes: $19 for 5,000 pages on Hobby is $3.80 per 1,000. At 100,000 pages ($99) and 1 million ($749) it sits close to the market for browser-rendered pages, because every Firecrawl page is rendered. ### Do I pay for pages that fail? It depends on the provider; read its billing page. Crawlbase states that failed requests and retries are free. WebReaderAPI does not charge for failed pages or for pages refused by robots.txt. ### Can I pay for a web reader API with crypto? Yes, at a few. Scrape.do, Decodo Web Data API, Scrapeless state crypto payment on an account, and Exa contents, Parallel Extract, Linkup fetch, Olostep and others take per-request USDC through x402. WebReaderAPI takes BTC, USDT, USDC, ETH, SOL, LTC with no KYC. ## Sources 1. [Firecrawl: pricing](https://www.firecrawl.dev/pricing) — Firecrawl (read 2026-10-22) 2. [Crawl4AI Cloud: pricing](https://crawl4ai.com/) — Crawl4AI Cloud (read 2026-10-22) 3. [Olostep: pricing](https://www.olostep.com/pricing) — Olostep (read 2026-10-22) 4. [Tavily Extract: pricing](https://docs.tavily.com/documentation/api-credits) — Tavily Extract (read 2026-10-22) 5. [Exa contents: pricing](https://exa.ai/pricing) — Exa contents (read 2026-10-22) 6. [Linkup fetch: pricing](https://docs.linkup.so/pages/documentation/development/pricing) — Linkup fetch (read 2026-10-22) 7. [ScrapingBee: pricing](https://www.scrapingbee.com/pricing/) — ScrapingBee (read 2026-10-22) 8. [Scrape.do: pricing](https://scrape.do/pricing.md) — Scrape.do (read 2026-10-22) 9. [ZenRows: pricing](https://docs.zenrows.com/first-steps/pricing.md) — ZenRows (read 2026-10-22) 10. [Scrapfly: pricing](https://scrapfly.io/pricing) — Scrapfly (read 2026-10-22) 11. [Bright Data Web Unlocker: pricing](https://www.bright.cn/pricing/web-unlocker) — Bright Data Web Unlocker (read 2026-10-22) 12. [Diffbot: pricing](https://www.diffbot.com/pricing/) — Diffbot (read 2026-10-22) --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Contact WebReaderAPI > Contact WebReaderAPI: support email for API and billing questions, abuse email for site owners and misuse reports, and what to include to get a fast answer. - URL: https://webreaderapi.com/contact/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Two addresses: one for customers, one for site owners and misuse reports. Both are read by a person. **Quick answer — How do I contact WebReaderAPI?** Write to support@webreaderapi.com for help with the API, your account or a payment; Email support answers within 12 hours, every day of the week, including help with crawls and integrations. Site owners and anyone reporting misuse of the **WebReaderAPI** crawler can write to abuse@webreaderapi.com. Our team will never ask you for your account number or an API key by email or chat. ## What makes a support request quick to answer? A short message with the facts gets the quickest answer: - the order ID from your order history, for billing questions; - the time of the request, the endpoint and the error code, for API problems; - the URL you tried to read, if a page did not come back as expected. Never send your API key or your account number by email. ## I run a website. How do I reach you? Write to abuse@webreaderapi.com with your domain, the date and time, and a line of your server log showing the user agent `WebReaderAPI/1.0 (+https://webreaderapi.com/bot/)`. To stop our crawler without writing to us, add a rule for `WebReaderAPI` to your robots.txt; the [bot page](https://webreaderapi.com/bot/) has ready-made examples. A person reads every abuse report, and accounts found misusing the service are suspended. ## What can support help with? Anything about using the API or your account: - a page that comes back empty, cut short or with the wrong section kept as main content; - an error code you do not understand, or a crawl job that stops early; - a top-up that has not appeared on your balance after its confirmations; - a refund request under the [refund policy](https://webreaderapi.com/refunds/); - a higher concurrency limit for production use. Support cannot make a site accept our crawler. When a site refuses us, the only route is the site owner's permission. ## How do I report a payment problem? Give the coin, the network, the amount sent and the transaction ID, together with the order ID from the client area. These are enough to find any transfer on the blockchain. Do not include private keys, seed phrases, your account number or your API key: we never need them, and anyone asking for them is not us. ## Is there a chat or a phone line? No. Email keeps a written record for both sides and lets us answer precisely, with logs at hand. For the state of the API, see the [status page](https://webreaderapi.com/status/); for common questions, the [FAQ](https://webreaderapi.com/faq/). --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Crawl API: turn a whole website into Markdown files > Crawl API that turns a whole website into Markdown for RAG: up to 10,000 pages per job, sitemap and robots.txt followed, from $0.07/1K pages. Crypto. - URL: https://webreaderapi.com/crawl-api/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Give a start URL and a page limit. The crawler follows the site's sitemap and links within the same domain, reads each page as Markdown and hands you the results page by page. One credit per page. **Quick answer — How can I crawl a whole website and get every page as Markdown?** Use a crawl API: you send a start URL and a limit, and it returns each page of the site as Markdown. **WebReaderAPI** crawls up to 10,000 pages per job within one domain, follows the sitemap, robots.txt and Crawl-delay, and charges one credit per page read, from $0.07/1K pages as of October 2026. Pages refused by robots.txt are free. No KYC; pay in crypto. ## Crawl API at a glance | Attribute | Value | | --- | --- | | Start a crawl | `POST https://webreaderapi.com/api/v1/crawl` with `url`, `limit`, `max_depth`, `include_paths`, `exclude_paths` | | Get the results | `GET https://webreaderapi.com/api/v1/jobs/{job_id}`, paginated with a cursor | | Scope | Crawl one website per job, up to 10,000 pages, following its sitemap, its robots.txt and its Crawl-delay | | Many separate URLs | `POST https://webreaderapi.com/api/v1/batch`: Up to 100 URLs per batch job | | Billing | 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged | | Site owners | We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged | | Price | $14.25 for 50,000 pages, $50 for 500,000 pages | ## How does a crawl job work? A crawl is a background job, because reading hundreds of pages takes longer than one HTTP request should. 1. **Start:** `POST https://webreaderapi.com/api/v1/crawl` with the start URL and your limits. You get a `job_id` at once. 2. **Discover:** the crawler reads the site's robots.txt and sitemap, then follows links within the same domain, breadth first, up to `max_depth`. 3. **Read:** each allowed page is read with the same options as `/read`: Markdown, main content, rendering when needed. 4. **Collect:** poll `GET https://webreaderapi.com/api/v1/jobs/{job_id}` and page through the results with the cursor while the job runs, or once it is `done`. The job stops at your `limit`, so you never pay for more pages than you asked for. *Python: crawl a documentation site into Markdown files* ```python import os import pathlib import re import time import requests API = "https://webreaderapi.com/api/v1" HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]} job = requests.post( API + "/crawl", headers=HEADERS, json={"url": "https://docs.example.com/", "limit": 500, "max_depth": 4, "include_paths": ["/guide/*"]}, timeout=60, ).json() out = pathlib.Path("docs_md") out.mkdir(exist_ok=True) cursor, delay = None, 5 while True: params = {"cursor": cursor} if cursor else {} page = requests.get(API + "/jobs/" + job["job_id"], headers=HEADERS, params=params, timeout=60).json() for item in page.get("results", []): if "markdown" in item: name = re.sub(r"[^a-z0-9]+", "-", item["url"].lower()).strip("-")[:120] (out / (name + ".md")).write_text(item["markdown"], encoding="utf-8") cursor = page.get("next_cursor") if page["status"] == "done" and not cursor: break if not cursor: time.sleep(delay) delay = min(delay * 2, 60) print("credits used:", page.get("credits_used")) ``` ## Which pages does the crawler visit, and which does it skip? It stays inside the domain of the start URL and skips anything the site asks robots to leave alone. In detail: - **Same domain only.** Links to other domains are listed in the results but not read. - **robots.txt first.** Pages disallowed for `WebReaderAPI` or for all robots are skipped and not charged. - **Crawl-delay honoured.** If the site sets one, the crawler waits that long between two pages, so a large site may take longer. - **Your filters.** `include_paths` and `exclude_paths` take simple patterns such as `/blog/*`; use them to avoid tag pages, search pages and archives. - **Duplicates removed.** The same page reached by two URLs, for example with tracking parameters, is read once when the site declares a canonical URL. ## Should I use a crawl or a batch? Use a crawl when you know the site but not the pages; use a batch when you already have the list of URLs. A crawl discovers pages from the sitemap and links. A batch reads exactly the URLs you send, up to 100 per job, across any number of sites. For a weekly refresh of a knowledge base, a common pattern is one crawl to build the index, then batches of the pages whose `lastmod` changed in the sitemap. That keeps the weekly cost to the pages that actually moved. ## How much does it cost to crawl a site? One credit per page read, two for pages that needed a browser. A documentation site of 1,500 pages that renders on the server costs 1,500 credits; the same site built as a single-page app may cost up to 3,000. Three typical jobs, as of October 2026: a 300-page product manual fits in the 10,000 pages pack ($3.75); ten documentation sites of 2,000 pages each fit in 50,000 pages ($14.25); a 400,000-page archive fits in 500,000 pages ($50). Skipped and failed pages are free. ## Which pack fits a crawl? Size the pack on the number of pages, then add a margin for pages that need a browser. Credits never expire, so the rest serves the next crawl. | Plan | Quantity | Price | Price per unit | | --- | --- | --- | --- | | Test | 10,000 pages | $3.75 | $0.38/1K pages | | Starter | 50,000 pages | $14.25 | $0.28/1K pages | | Build | 100,000 pages | $21.75 | $0.22/1K pages | | Scale | 500,000 pages | $50 | $0.10/1K pages | In every plan: Main-content extraction · JavaScript rendering when needed · Failed and refused pages not charged · Credits never expire **Before you pay** - Minimum top-up $25. Top up once in crypto, then buy any plan from your balance. - Your balance has no end date and can buy page credits at any time. If a pack costs less than the minimum top-up, the amount you added only to reach that minimum can be sent back on request within 30 days of the payment, in USDT or USDC, as long as you have not spent it. Any other balance is not refunded unless the law says otherwise. - Credited once your payment is checked on the blockchain. Typical network confirmation time: BTC ≈ 20 min · USDT TRC-20 ≈ 1 min · USDT ERC-20 ≈ 5 min · USDC ERC-20 ≈ 5 min · ETH ≈ 5 min · SOL ≈ 1 min · LTC ≈ 15 min ## Crawl API: common questions ### How many pages can one crawl read? Up to 10,000 pages per job, within the domain of the start URL. For a larger site, split it into several jobs with `include_paths`, for example one per section. ### Does the crawler use the sitemap? Yes. It reads the sitemap listed in robots.txt or at the usual location, then follows links on the pages it reads. Pages that only the sitemap lists are still read if robots.txt allows them. ### Can I crawl a site that blocks AI crawlers? Not with WebReaderAPI. If robots.txt disallows our crawler or all robots, those pages are skipped and not charged. If the site declares `Content-Signal: ai-input=no`, we do not read it for you. We never try to get around such choices. ### Can I get the results while the crawl is still running? Yes. `GET https://webreaderapi.com/api/v1/jobs/{job_id}` returns the pages read so far with a cursor; keep paging until the job is `done` and no cursor is left. ### Can a crawl read more pages than my limit? No. Credits are counted page by page, and the job stops when it reaches the `limit` you set. Pages it skips because of robots.txt, and pages that fail, use no credit. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Pay for a web reader API in crypto > How to pay WebReaderAPI in crypto: BTC, USDT, USDC, ETH, SOL or LTC, the $25 minimum top-up, which network to choose, and what to do after sending. - URL: https://webreaderapi.com/crypto-payments/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) A step-by-step guide to topping up your WebReaderAPI balance in crypto and turning it into page credits, with the mistakes to avoid. **Quick answer — How do I pay for WebReaderAPI with crypto?** Top up your **WebReaderAPI** balance with at least $25 in BTC, USDT, USDC, ETH, SOL, LTC: send the exact amount shown, on the network shown, then click "I've sent the payment". The top-up is matched by its exact amount and credited once confirmed. You then buy page credits from the balance, from $3.75 for 10,000 pages. No KYC. ## Which coins can I pay with? Six coins, one balance. USDT works on TRC-20 or ERC-20, USDC on ERC-20; the payment page shows the address and network for the coin you pick. - Bitcoin (BTC) - Tether (USDT) — TRC-20 or ERC-20 - USD Coin (USDC) — ERC-20 - Ethereum (ETH) - Solana (SOL) - Litecoin (LTC) ## How does a crypto top-up work, step by step? Five steps from an empty account to a working API key. The only wait is the blockchain confirmation. 1. **Choose an amount and a coin** — Enter at least $25 and pick BTC, USDT, USDC, ETH, SOL or LTC. The payment page converts the dollar amount into that coin. 2. **Send the exact amount** — Copy the address and the amount shown, and send them on the network the page names: one transfer, the exact figure, no rounding. 3. **Click "I've sent the payment"** — Tell the page the transfer is on its way. If it asks for the transaction ID, paste it so the transfer is easier to find. 4. **Let the network confirm it** — The top-up is matched by its exact amount and credited once the network has confirmed it. Bitcoin needs more time than USDT on Tron or SOL on Solana. 5. **Buy a credit pack** — Pay any pack from the balance. If you started from a pack, it is bought once the top-up is credited, and your API key appears in the dashboard. ## Which coin and network should I use? Use the coin you already hold, on the network the payment page names. A stablecoin (USDT or USDC) keeps the amount in dollars, so the price you see is the price you pay. Network fees and confirmation times differ by chain; the table in the steps above gives the usual confirmation time for each network. If you hold USDT on several chains, pick the one whose fee is smaller for your amount. The fee is paid by the sender and is not part of your top-up. ## Why is there a minimum top-up? Because each payment costs us checking and bookkeeping, whatever its size. The minimum is $25; any pack can then be bought from the balance, including the 10,000 pages pack at $3.75. What you do not spend stays on the balance with no end date. If the pack you bought costs less than the minimum, the extra part can be asked back within 30 days, in USDT or USDC, as long as it is unspent. ## What happens after I send the payment? Click "I've sent the payment" on the payment page. We match the transfer by its exact amount, wait for the confirmations the network needs, and credit your balance. If you started from a pack, it is bought from the balance once the top-up is credited, and your API key is ready in the dashboard. If you closed the tab, open the client area: pending and credited top-ups appear in your order history. ## What if I send the wrong amount or use the wrong network? A different amount cannot be matched by itself, and a transfer on another network may not reach the address at all. Before you send, check three things: the coin, the network and the exact amount. If a mistake happened, write to support@webreaderapi.com with the transaction ID and the order ID. Do not send your account number or API key. Funds sent on a network the payment page did not show may not be recoverable. ## Can a crypto payment be refunded? Refunds go to your balance rather than back to your wallet, which avoids a second network fee. A pack used at less than 20% within 72 hours, or a pack that never worked, is credited back. The [refund policy](https://webreaderapi.com/refunds/) gives the exact conditions and three examples. ## What can I buy once my balance is credited? Any pack: 10,000 pages for $3.75, 50,000 pages for $14.25 or 100,000 pages for $21.75, and larger ones on the [pricing page](https://webreaderapi.com/pricing/). Credits never expire. | Plan | Quantity | Price | Price per unit | | --- | --- | --- | --- | | Test | 10,000 pages | $3.75 | $0.38/1K pages | | Starter | 50,000 pages | $14.25 | $0.28/1K pages | | Build | 100,000 pages | $21.75 | $0.22/1K pages | In every plan: Main-content extraction · JavaScript rendering when needed · Failed and refused pages not charged · Credits never expire Prices in USD, set on 2026-10-22. Every pack is paid from your crypto balance. ## Paying in crypto: common questions ### Do I need a card or an ID to pay? No. WebReaderAPI takes crypto only and asks for no identity documents. Account-number login; email is optional. ### Is the price fixed when I start the payment? The payment page shows the amount in the coin you chose for that payment. Send that exact amount; a stablecoin keeps it equal to the dollar price. ### Can I pay for credits per request instead of topping up? No. Payments go through a prepaid balance, and each read uses credits from it. The balance and the credits never expire. ### Where do I see my payments? In the client area's order history, with each top-up's status, coin, network and transaction ID. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # URL to Markdown API price index 2026: what 1,000 pages cost at 21 APIs > URL-to-Markdown and web reader API price index 2026: 21 APIs, cost per 1,000 pages at 3 volumes, static and JavaScript pages, crypto, KYC. Checked 2026-10-22. - URL: https://webreaderapi.com/data/url-to-markdown-price-index-2026/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Public list prices of 21 APIs that turn web pages into Markdown or text for models, normalised to the cost of 1,000 pages at three monthly volumes, for plain pages and for pages rendered in a browser. Published by WebReaderAPI, which is one of the APIs in the table. **Quick answer — How much does a URL-to-Markdown API cost per 1,000 pages in 2026?** Across 21 APIs whose list prices we read on 2026-10-22, reading 100,000 plain pages a month costs a median $0.99 per 1,000, and $1.00 when pages need a browser. At 1 million pages a month the medians fall to $0.774 and $0.899. **WebReaderAPI**, which publishes this index, charges $0.22/1K pages for plain pages at 100,000. ## The index in six numbers - **$2.00** median per 1,000 plain pages at 10,000 a month (21 APIs) - **$0.99** median per 1,000 plain pages at 100,000 a month (21 APIs) - **$0.774** median per 1,000 plain pages at 1 million a month (20 APIs) - **$1.00** median per 1,000 browser-rendered pages at 100,000 a month (18 APIs) - **$0.899** median per 1,000 browser-rendered pages at 1 million a month (17 APIs) - **3 + 7** APIs that state crypto payment on an account, plus APIs taking per-request USDC (x402) ## How was the index built? We read each provider's public pricing page on 2026-10-22 and kept only figures shown there; every source is listed at the bottom. A "page" is one URL read and returned as Markdown or text, with the provider's default proxy and no paid add-on. - **Plain page:** no JavaScript rendering. **Browser page:** rendered in a headless browser first; providers that always render (Firecrawl, Olostep, Diffbot) have the same price in both tables. - **Price at a volume:** the least expensive published way to buy that many pages every month on one provider (plan, plan plus published overage, or pay as you go), divided by the pages used. A $99 plan used for 100,000 pages counts as $0.99 per 1,000 even if it includes more. - **Credit multipliers are applied**, for example 5 credits per browser page at ScrapingBee, ZenRows and Scrape.do. - **Free tiers are listed, not priced.** APIs billed by tokens, browser time, data volume or compute, and one API whose Markdown output we could not confirm, are listed separately. WebReaderAPI is shown for reference and is not counted in the medians. ## Price per 1,000 plain pages, by monthly volume Static pages, read without a browser. "—" means not published or not self-serve at that volume. | API | 10K pages a month | 100K pages a month | 1M pages a month | | --- | --- | --- | --- | | WebReaderAPI (prepaid, never expires) | $0.38/1K pages | $0.22/1K pages | $0.07/1K pages | | Firecrawl | $4.40 | $0.99 | $0.749 | | Tavily Extract | $1.60 | $1.40 | $1.30 | | Exa contents | $1.00 | $1.00 | $1.00 | | Parallel Extract | $1.00 | $1.00 | $1.00 | | Linkup fetch | $1.00 | $1.00 | $1.00 | | Olostep | $2.90 | $0.99 | $0.399 | | ScrapeGraphAI | $2.00 | $1.00 | $1.25 | | Diffbot | $29.90 | $2.99 | $0.899 | | ScrapingBee | $1.90 | $0.49 | $0.099 | | ScraperAPI | $4.90 | $0.49 | $0.149 | | ZenRows | $0.50 | $0.39 | $0.199 | | Scrapfly | $3.00 | $0.30 | $0.10 | | Scrape.do | $2.90 | $0.29 | $0.099 | | Crawlbase | $2.10 | $0.764 | $0.349 | | Bright Data Web Unlocker | $1.50 | $1.50 | $1.30 | | Oxylabs Web Scraper API | $3.00 | $1.00 | $2.50 | | Decodo Web Data API | $1.90 | $0.49 | $0.249 | | Scrapeless | $1.00 | $0.90 | $0.80 | | Hyperbrowser Fetch | $3.00 | $1.00 | — | | Steel scrape | $5.00 | $5.00 | $5.00 | | Crawl4AI Cloud | $1.00 | $0.50 | $0.395 | *Prices checked on October 22, 2026.* List prices in USD, read on 2026-10-22. Our prices: the WebReaderAPI pricing page on the same date. ## Price per 1,000 browser-rendered pages, by monthly volume Pages that need JavaScript, rendered in a headless browser. APIs that document no browser option are left out of this table. | API | 10K pages a month | 100K pages a month | 1M pages a month | | --- | --- | --- | --- | | WebReaderAPI (2 credits per page) | $0.75/1K pages | $0.435/1K pages | $0.148/1K pages | | Firecrawl | $4.40 | $0.99 | $0.749 | | Tavily Extract | $3.00 | $2.36 | $2.90 | | Linkup fetch | $5.00 | $5.00 | $5.00 | | Olostep | $2.90 | $0.99 | $0.399 | | ScrapeGraphAI | $2.00 | $1.00 | $1.25 | | Diffbot | $29.90 | $2.99 | $0.899 | | ScrapingBee | $1.90 | $0.99 | $0.599 | | ScraperAPI | $4.90 | $1.49 | $0.975 | | ZenRows | $2.50 | $0.99 | $0.549 | | Scrapfly | $3.00 | $1.00 | $0.56 | | Scrape.do | $2.90 | $0.99 | $0.699 | | Crawlbase | — | $0.99 | $0.599 | | Bright Data Web Unlocker | $1.50 | $1.50 | $1.30 | | Oxylabs Web Scraper API | $3.00 | $5.00 | $2.50 | | Decodo Web Data API | $1.90 | $0.99 | $0.999 | | Hyperbrowser Fetch | $3.00 | $1.00 | — | | Steel scrape | $5.00 | $5.00 | $5.00 | | Crawl4AI Cloud | $1.00 | $0.99 | $0.667 | *Prices checked on October 22, 2026.* ## Why do prices differ so much between APIs? Four things explain most of the spread. 1. **Product type.** General scraping APIs (ScrapingBee, Scrape.do, ZenRows, Scrapfly) are priced for volume and reach about $0.10 per 1,000 plain pages at 1 million a month. AI-focused readers (Exa, Parallel, Linkup) charge a flat $1.00 and add retrieval features. 2. **Browser multipliers.** A browser page costs 2 to 10 times a plain page at credit-based APIs; flat-priced APIs charge the same for both. 3. **Plan steps.** Many providers sell monthly plans with a large allowance, so small users pay for unused pages: the median at 10,000 plain pages a month is $2.00. 4. **Expiry.** Most plans reset unused credits monthly; ScrapeGraphAI, Crawlbase, Crawl4AI Cloud keep bought credits without a monthly reset. ## Free tiers, expiry, crypto, identity checks and robots.txt What each provider states on the pages we read. "Not stated" means the pages did not say. | API | Free tier | Credit expiry | Crypto payment | Identity checks | robots.txt | | --- | --- | --- | --- | --- | --- | | WebReaderAPI | None | Never | BTC, USDT, USDC, ETH, SOL, LTC | None: no ID, no selfie, no documents | Respected; never overridden | | Firecrawl | 1,000 credits a month, no card | Plan credits reset monthly | No (cards, PayPal) | None stated | Respected by default; override on Enterprise only | | Tavily Extract | 1,000 credits a month (5,000 basic URLs) | Reset monthly | Only at its discretion (terms) | None stated | Not stated | | Exa contents | $10 credit a month | Free credit resets monthly | Per request, USDC (x402) | Card for pay as you go | Stated for its index crawler, not for user fetches | | Parallel Extract | 5,000 requests a month | Free credit expires monthly | Per request, USDC (x402) | Card for the monthly credit | Stated for its index crawler, not for user fetches | | Linkup fetch | $20 a month (professional email) | Not stated | Per request, USDC (x402) | Professional email for free credit | Crawling respects it; fetch not stated | | Olostep | 500-request trial | Plan monthly; top-ups 6 months | Per request, USDC (x402) | Card for top-ups | Tools to respect it; default not stated | | ScrapeGraphAI | 500 credits | Plan credits don't carry over; top-ups never expire | Agents only (MPP) | Not stated | Customer must comply (terms) | | Diffbot | 10,000 credits a month | Quota resets each period | Not stated | No card for the free plan | Crawl obeys by default; Extract not stated | | ScrapingBee | 1,000 credits | No rollover | No | Card for paid plans | Not stated | | ScraperAPI | 5,000 credits for 7 days | Expire each billing period | No | None stated | Not stated | | ZenRows | 5,000 credits a month | At renewal | Accounts: no; agents: USDC (x402) | Email validation | Not stated | | Scrapfly | 1,000 credits once | No rollover | Not stated | KYC for every customer | Crawler API respects it by default; can be disabled | | Scrape.do | 1,000 credits a month | Not stated | Yes (coins not stated) | None stated | Not stated | | Crawlbase | 1,000 credits | Pay as you go: nothing expires | Not stated | None stated | Not stated | | Bright Data Web Unlocker | 5,000 credits a month | Free credits reset monthly | No | Card for personal-email accounts; company KYC for robots-disallowed pages | Honoured by default | | Oxylabs Web Scraper API | $5 credit once | End of cycle | No | KYC for every customer | Not mentioned | | Decodo Web Data API | Up to 10,000 requests a month | Unused amounts expire monthly | Yes (via CoinGate) | Automatic check, then KYC | Not stated | | Scrapeless | Trial, size not stated | Prepaid minimum not kept | Yes (coins not stated) | None for basic packages | Not stated | | Hyperbrowser Fetch | 5,000 credits | Bought credits: 12 months | Per request, USDC (x402) | Not stated | Not stated | | Steel scrape | $30 once (90 days) | Launch credit 90 days | Per request, USDC (x402) | No card | Terms forbid violating it | | Crawl4AI Cloud | 1,000 credits a month | Bought credits never expire | Not stated | No card | Terms require honouring it | *Prices checked on October 22, 2026.* ## What are the main findings? - **Plain pages are cheap at volume.** At 1 million pages a month, ScrapingBee ($0.099), Scrape.do ($0.099), Scrapfly ($0.10) list the lowest plain-page prices. - **Browser pages cost more.** At 1 million a month, Olostep ($0.399), ZenRows ($0.549), Scrapfly ($0.56) list the lowest browser-page prices. - **The best-known reader is not the least expensive.** Firecrawl lists $0.99 per 1,000 at 100,000 pages and $0.749 at 1 million, with every page rendered. - **Crypto is rare on accounts.** Scrape.do, Decodo Web Data API, Scrapeless state crypto payment; Exa contents, Parallel Extract, Linkup fetch, Olostep, ZenRows, Hyperbrowser Fetch, Steel scrape take per-request USDC through x402. - **robots.txt is often left to the customer.** Providers stating a robots.txt-respecting default, at least on their crawl products: Firecrawl, Diffbot, Scrapfly, Bright Data Web Unlocker. Many say nothing. - **KYC is the exception.** Oxylabs and Scrapfly ask every customer for it; Decodo runs an automatic check first. ## APIs billed another way (not in the medians) These bill by tokens, browser time, data volume or compute, or did not confirm Markdown output, so a price per page depends on your pages. | API | Price | Note | | --- | --- | --- | | Jina Reader | $50 per 1 billion output tokens; $500 per 11 billion | Billed per token: the cost per page depends on page length. 10 million free tokens per new key, non-commercial only. | | Cloudflare Browser Run | $0.09 per browser-hour after 10 included hours, $5 monthly minimum | Billed by browser time; /markdown, /content and /crawl endpoints. | | Spider | $1 per GB of content plus compute | Billed by data volume; the per-page figures it publishes are its own estimates. | | Apify Website Content Crawler | Compute units ($0.13 per unit on Business) | Billed by compute; the per-page figures it publishes are its own estimates. | | Zyte API | $0.13 per 1,000 static requests (website tier 1, pay as you go); $1.01 with a browser | Price depends on a tier Zyte assigns per website; Markdown output not stated on the pages we read. | *Prices checked on October 22, 2026.* ## Can I reuse this data? Yes. The tables on this page are published under the Creative Commons Attribution 4.0 licence (CC BY 4.0): copy and adapt them with a link to this page and the date 2026-10-22. We read the prices again at least once a month. If a figure is wrong, write to support@webreaderapi.com with the page that shows the right one; we correct it and note the change in the [changelog](https://webreaderapi.com/changelog/). ## Price index: common questions ### Which URL-to-Markdown API costs least? It depends on volume and on JavaScript. On 2026-10-22, the lowest plain-page prices at 1 million pages a month were ScrapingBee at $0.099 and Scrape.do at $0.099 per 1,000; the lowest browser-page price at that volume was Olostep at $0.399. ### How much does Firecrawl cost per page? Firecrawl charges 1 credit per page with JavaScript included: $19 for 5,000 credits on Hobby, $99 for 100,000 on Standard and $749 for 1 million on Scale, read on its pricing page on 2026-10-22. JSON extraction adds 4 credits per page. ### Why is Jina Reader not in the main tables? Jina bills by output tokens ($50 per 1 billion tokens on its smallest package), so its price per page depends on how long your pages are. It is listed in the table of APIs billed another way. ### Does WebReaderAPI count itself in the medians? WebReaderAPI publishes this index and appears in the first rows for reference only. The medians use the other providers. ### How often is the index updated? At least once a month, and when a listed provider changes its public prices. The date at the top shows the last full check. ## Sources 1. [Firecrawl: pricing](https://www.firecrawl.dev/pricing) — Firecrawl (read 2026-10-22) 2. [Tavily Extract: pricing](https://docs.tavily.com/documentation/api-credits) — Tavily (read 2026-10-22) 3. [Exa contents: pricing](https://exa.ai/pricing) — Exa (read 2026-10-22) 4. [Parallel Extract: pricing](https://parallel.ai/pricing) — Parallel (read 2026-10-22) 5. [Linkup fetch: pricing](https://docs.linkup.so/pages/documentation/development/pricing) — Linkup (read 2026-10-22) 6. [Olostep: pricing](https://www.olostep.com/pricing) — Olostep (read 2026-10-22) 7. [ScrapeGraphAI: pricing](https://scrapegraphai.com/pricing) — ScrapeGraphAI (read 2026-10-22) 8. [Diffbot: pricing](https://www.diffbot.com/pricing/) — Diffbot (read 2026-10-22) 9. [ScrapingBee: pricing](https://www.scrapingbee.com/pricing/) — ScrapingBee (read 2026-10-22) 10. [ScraperAPI: pricing](https://www.scraperapi.com/pricing/) — ScraperAPI (read 2026-10-22) 11. [ZenRows: pricing](https://docs.zenrows.com/first-steps/pricing.md) — ZenRows (read 2026-10-22) 12. [Scrapfly: pricing](https://scrapfly.io/pricing) — Scrapfly (read 2026-10-22) 13. [Scrape.do: pricing](https://scrape.do/pricing.md) — Scrape.do (read 2026-10-22) 14. [Crawlbase: pricing](https://crawlbase.com/pricing) — Crawlbase (read 2026-10-22) 15. [Bright Data Web Unlocker: pricing](https://www.bright.cn/pricing/web-unlocker) — Bright Data (read 2026-10-22) 16. [Oxylabs Web Scraper API: pricing](https://oxylabs.io/pricing/web-scraper-api) — Oxylabs (read 2026-10-22) 17. [Decodo Web Data API: pricing](https://help.decodo.com/docs/web-data-api-billing) — Decodo (read 2026-10-22) 18. [Scrapeless: pricing](https://www.scrapeless.com/en/pricing) — Scrapeless (read 2026-10-22) 19. [Hyperbrowser Fetch: pricing](https://hyperbrowser.ai/docs/pricing.md) — Hyperbrowser (read 2026-10-22) 20. [Steel scrape: pricing](https://docs.steel.dev/overview/pricinglimits) — Steel (read 2026-10-22) 21. [Crawl4AI Cloud: pricing](https://crawl4ai.com/) — Crawl4AI (read 2026-10-22) 22. [Jina Reader: pricing](https://jina.ai/api-dashboard/pricing/) — Jina Reader (read 2026-10-22) 23. [Cloudflare Browser Run: pricing](https://developers.cloudflare.com/browser-run/pricing/) — Cloudflare Browser Run (read 2026-10-22) 24. [Spider: pricing](https://spider.cloud/pricing) — Spider (read 2026-10-22) 25. [Apify Website Content Crawler: pricing](https://apify.com/apify/website-content-crawler) — Apify Website Content Crawler (read 2026-10-22) 26. [Zyte API: pricing](https://www.zyte.com/pricing/) — Zyte API (read 2026-10-22) --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # URL to Markdown API quick start: your first page in two minutes > URL to Markdown API quick start: get a key, read your first page with curl, Python or Node.js, and check your credits. From $0.07/1K pages, no KYC. - URL: https://webreaderapi.com/docs/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) One endpoint, one parameter: send a URL to `/read` and get the page's main content back as Markdown. This page takes you from an empty balance to a working call, then points to everything else. **Quick answer — How do I convert a URL to Markdown with an API?** Send the URL to `GET https://webreaderapi.com/api/v1/read` with your API key in the `Authorization` header. **WebReaderAPI** fetches the page, keeps the main content and returns it as Markdown with its title, author, date and word count. One page costs one credit (two if it needs a browser), from $0.07/1K pages as of October 2026. No KYC; pay in crypto. ## The API in one table | Attribute | Value | | --- | --- | | Base URL | `https://webreaderapi.com/api/v1` | | Read one page | `GET` or `POST https://webreaderapi.com/api/v1/read` | | Many URLs or a site | `POST https://webreaderapi.com/api/v1/batch` and `POST https://webreaderapi.com/api/v1/crawl`, results from `GET https://webreaderapi.com/api/v1/jobs/{job_id}` | | Credits left | `GET https://webreaderapi.com/api/v1/credits` (free) | | Authentication | Send your API key in the Authorization header as a Bearer token, or in the api_key query parameter | | Billing | 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged | | Our crawler | `WebReaderAPI/1.0 (+https://webreaderapi.com/bot/)`, robots.txt token `WebReaderAPI` | | Smallest pack | 10,000 pages for $3.75; credits never expire | ## How do I get an API key? Three steps, no form to fill in. The slowest one is the blockchain confirmation of your top-up. 1. **Top up in crypto** — Send at least $25 in BTC, USDT, USDC, ETH, SOL, LTC. Account-number login; email is optional. 2. **Buy a credit pack** — 10,000 pages for $3.75 is enough to test every option on your own URLs. 3. **Copy your key** — Your API key is ready in the dashboard once your crypto top-up is confirmed and you have bought a pack. Store it in an environment variable named WEBREADER_API_KEY. ## What does the first request look like? It is a plain HTTPS GET with the page address in the `url` parameter. Encode the address, because it contains its own `:` and `/` characters; curl does it for you with `--data-urlencode`. The default settings already suit most articles and documentation pages: Markdown output, main content only, and a browser only when the page needs one. *curl: read one page* ```bash curl -sG "https://webreaderapi.com/api/v1/read" \ -H "Authorization: Bearer $WEBREADER_API_KEY" \ --data-urlencode "url=https://example.com/blog/first-post" ``` *Python: read one page and save it* ```python import os import requests resp = requests.get( "https://webreaderapi.com/api/v1/read", headers={"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}, params={"url": "https://example.com/blog/first-post"}, timeout=70, ) resp.raise_for_status() page = resp.json() print(page["title"], "-", page["word_count"], "words,", page["credits_used"], "credit(s)") with open("first-post.md", "w", encoding="utf-8") as f: f.write(page["markdown"]) ``` *Node.js 18+: read one page* ```javascript const params = new URLSearchParams({ url: 'https://example.com/blog/first-post' }); const res = await fetch(`https://webreaderapi.com/api/v1/read?${params}`, { headers: { Authorization: `Bearer ${process.env.WEBREADER_API_KEY}` }, }); const page = await res.json(); if (!res.ok) throw new Error(`${page.error.code}: ${page.error.message}`); console.log(page.title, page.word_count, 'words'); console.log(page.markdown.slice(0, 500)); ``` ## What comes back? A JSON object with the Markdown in the `markdown` field and the page's metadata next to it: `title`, `description`, `author`, `published_at`, `language`, `site_name` and `word_count`. Two fields help you check what happened: `final_url` is the address after redirects, and `rendered` tells you whether a browser was needed. `credits_used` shows what the call cost: 1 for a plain page, 2 when `rendered` is `true`, one more if you chose a `country`. The [response format](https://webreaderapi.com/docs/response/) page lists every field with an example. ## Which options will I use most? Five parameters cover most needs. All of them are optional. | Parameter | Use it to | |---|---| | `format` | Get `text` or cleaned `html` instead of `markdown` | | `main_content` | Set `false` to convert the whole page, menus included | | `render` | Force (`true`) or forbid (`false`) the browser; `auto` decides | | `include_links` | Get the page's links as a separate list | | `max_chars` | Cut long pages to fit a model's context budget | The [read reference](https://webreaderapi.com/docs/read/) describes every parameter, including `country`, `include_images` and `timeout`. ## How do I check my remaining credits? Call `GET https://webreaderapi.com/api/v1/credits` with the same key. It returns `credits_remaining`, `credits_used_today` and your `concurrency_limit`, and it costs nothing. Run it at the start of a long job so the job stops cleanly instead of failing on `402 insufficient_credits` halfway through. Prepaid credits never expire: unused pages stay on your account until you use them, so a pack bought for one project keeps its value for the next. ## What happens if a site does not allow robots? We do not read the page, and you are not charged. Our crawler announces itself as `WebReaderAPI/1.0 (+https://webreaderapi.com/bot/)` and follows the site's robots.txt. A disallowed page returns `403 blocked_by_robots`; a site that refuses the request another way returns `502 target_refused` with the site's own status code. If a site refuses our request, we return its answer: we never solve CAPTCHAs, log in to accounts, or try to get around a block or a paywall. The [errors page](https://webreaderapi.com/docs/errors/) explains which errors are worth a retry and which are final. ## Where do I go next? - [Read endpoint reference](https://webreaderapi.com/docs/read/) — Every parameter of /read, with examples - [Batch and crawl](https://webreaderapi.com/docs/crawl/) — Up to 100 URLs per batch, or a whole site - [Response format](https://webreaderapi.com/docs/response/) — Every field of the answer - [Errors and robots.txt](https://webreaderapi.com/docs/errors/) — Codes, causes and retry rules - [Python](https://webreaderapi.com/docs/python/) — requests, async httpx, RAG chunking - [Node.js](https://webreaderapi.com/docs/node/) — fetch, a small pool, crawl polling - [curl and jq](https://webreaderapi.com/docs/curl/) — Recipes for the terminal - [MCP server](https://webreaderapi.com/docs/mcp/) — A read_url tool for Claude and Cursor - [LangChain and LlamaIndex](https://webreaderapi.com/docs/langchain/) — Loaders and tools for RAG - [OpenAI function calling](https://webreaderapi.com/docs/openai/) — A read_url tool schema and loop - [n8n](https://webreaderapi.com/docs/n8n/) — HTTP Request tool for an AI Agent - [OpenClaw](https://webreaderapi.com/docs/openclaw/) — Add the MCP server next to web_fetch ## Quick start: common questions ### Is there an SDK to install? No. WebReaderAPI is a plain HTTPS API, so any HTTP client works. The docs give copy-paste code for Python, Node.js and curl, and tool definitions for MCP, LangChain, LlamaIndex and OpenAI function calling. Nothing is published as a package. ### Can I send the key in the URL instead of a header? Yes, as the `api_key` query parameter. The `Authorization: Bearer` header is safer, because URLs often end up in logs and browser history. Create several API keys, for example one per agent or project, and revoke any of them from your dashboard. ### Does a failed page use a credit? No. 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged. Only a page that comes back with content is billed. ### Can I try it before buying? There is no free tier. There is no free tier: the smallest pack is the way to test, paid from the minimum top-up, and the amount added only to reach that minimum can be sent back on request within 30 days, in USDT or USDC, while unspent. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Batch and crawl API: read many URLs or a whole site > Batch and crawl API reference: read up to 100 URLs per job or a whole site up to 10,000 pages, robots.txt and Crawl-delay respected. No KYC, pay in crypto. - URL: https://webreaderapi.com/docs/crawl/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Two endpoints start background jobs: `/batch` for a list of up to 100 URLs, `/crawl` for one website. Both return a job ID; you collect the pages from `/jobs/{job_id}`, a page of results at a time. **Quick answer — How do I crawl a website to Markdown with WebReaderAPI?** Send the start URL to `POST https://webreaderapi.com/api/v1/crawl` with a page `limit`. **WebReaderAPI** reads the site in the background, same domain only, following its sitemap, robots.txt and Crawl-delay, up to 10,000 pages per job. Collect the Markdown pages from the job, a page of results at a time. Each page read costs 1 credit (2 with a browser), from $0.07/1K pages as of October 2026. No KYC; pay in crypto. ## Jobs at a glance | Attribute | Value | | --- | --- | | Batch | Up to 100 URLs per batch job: `POST https://webreaderapi.com/api/v1/batch` | | Crawl | Crawl one website per job, up to 10,000 pages, following its sitemap, its robots.txt and its Crawl-delay: `POST https://webreaderapi.com/api/v1/crawl` | | Results | `GET https://webreaderapi.com/api/v1/jobs/{job_id}`, then `?cursor=…` for the next page of results | | Billing | Each page read is billed like a `/read` call; pages skipped by robots.txt and failed pages cost nothing | | Scope of a crawl | One domain; links to other domains are listed but not read | | Cancel a job | Not available in v1: set `limit` to the most pages you are ready to pay for | ## Start a batch job `POST /api/v1/batch` Reads up to 100 URLs in the background with the same options. Options use the names and values of [/read](https://webreaderapi.com/docs/read/). The answer is the job ID; values are illustrative. Authentication: API key in the `Authorization: Bearer API_KEY` header. **Request body** | Field | Type | Required | Description | | --- | --- | --- | --- | | `urls` | array of strings | Yes | 1 to 100 http or https URLs. Duplicates are read once. | | `options` | object | No | Any `/read` parameter except `url`: `format`, `main_content`, `render`, `include_links`, `include_images`, `country`, `max_chars`, `timeout`. | *Example: three articles as Markdown* ```json { "urls": [ "https://example.com/blog/post-1", "https://example.com/blog/post-2", "https://example.org/news/launch" ], "options": { "format": "markdown", "max_chars": 50000 } } ``` **Response** `202 Accepted` ```json { "job_id": "job_5c1e8a90d2" } ``` *curl* ```bash curl -s "https://webreaderapi.com/api/v1/batch" \ -H "Authorization: Bearer $WEBREADER_API_KEY" \ -H "Content-Type: application/json" \ -d '{"urls": ["https://example.com/blog/post-1", "https://example.com/blog/post-2"], "options": {"format": "markdown"}}' ``` *Python* ```python import os import requests resp = requests.post( "https://webreaderapi.com/api/v1/batch", headers={"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}, json={"urls": ["https://example.com/blog/post-1", "https://example.com/blog/post-2"], "options": {"format": "markdown"}}, timeout=30, ) print(resp.json()["job_id"]) ``` *Node.js* ```javascript const res = await fetch('https://webreaderapi.com/api/v1/batch', { method: 'POST', headers: { Authorization: `Bearer ${process.env.WEBREADER_API_KEY}`, 'Content-Type': 'application/json' }, body: JSON.stringify({ urls: ['https://example.com/blog/post-1', 'https://example.com/blog/post-2'], options: { format: 'markdown' } }), }); console.log((await res.json()).job_id); ``` ## Start a crawl job `POST /api/v1/crawl` Reads one website from a start URL, breadth first. The crawler stays on the start URL's domain, uses the sitemap named in robots.txt or found at `/sitemap.xml`, and respects robots.txt and Crawl-delay. Values are illustrative. Authentication: API key in the `Authorization: Bearer API_KEY` header. **Request body** | Field | Type | Required | Description | | --- | --- | --- | --- | | `url` | string | Yes | The start page, such as the home page of a documentation site. | | `limit` | integer | No | Most pages to read, up to 10,000. The job stops when it reaches this number. | | `max_depth` | integer | No | Most link steps away from the start URL. | | `include_paths` | array of strings | No | Simple path patterns to keep, such as `/docs/*`. | | `exclude_paths` | array of strings | No | Simple path patterns to skip, such as `/blog/tag/*`. | | `options` | object | No | Any `/read` parameter except `url`, applied to every page. | *Example: the docs section of a site, 500 pages at most* ```json { "url": "https://docs.example.com/", "limit": 500, "max_depth": 3, "include_paths": [ "/docs/*" ], "exclude_paths": [ "/docs/archive/*" ], "options": { "format": "markdown" } } ``` **Response** `202 Accepted` ```json { "job_id": "job_8b27f41c0e" } ``` *curl* ```bash curl -s "https://webreaderapi.com/api/v1/crawl" \ -H "Authorization: Bearer $WEBREADER_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://docs.example.com/", "limit": 500, "include_paths": ["/docs/*"]}' ``` *Python* ```python import os import requests resp = requests.post( "https://webreaderapi.com/api/v1/crawl", headers={"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}, json={"url": "https://docs.example.com/", "limit": 500, "include_paths": ["/docs/*"]}, timeout=30, ) print(resp.json()["job_id"]) ``` *Node.js* ```javascript const res = await fetch('https://webreaderapi.com/api/v1/crawl', { method: 'POST', headers: { Authorization: `Bearer ${process.env.WEBREADER_API_KEY}`, 'Content-Type': 'application/json' }, body: JSON.stringify({ url: 'https://docs.example.com/', limit: 500, include_paths: ['/docs/*'] }), }); console.log((await res.json()).job_id); ``` ## Get job status and results `GET /api/v1/jobs/{job_id}` Returns the job's status and the next results that are ready. Each item of `results` is either a full `/read` response or `{url, error}` for a page that could not be read. When `next_cursor` is present, call again with `cursor` set to it; when it is absent, no more results are ready yet. Values are illustrative. Authentication: API key in the `Authorization: Bearer API_KEY` header. **Parameters** | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `job_id` | path | string | Yes | The ID returned by `/batch` or `/crawl`. | | `cursor` | query | string | No | The `next_cursor` value from the previous call, to get the following results. | **Response** `200` ```json { "job_id": "job_8b27f41c0e", "status": "running", "results": [ { "url": "https://docs.example.com/docs/install", "final_url": "https://docs.example.com/docs/install", "status_code": 200, "rendered": false, "credits_used": 1, "title": "Install", "markdown": "# Install\n\nRun the installer...", "word_count": 418 }, { "url": "https://docs.example.com/docs/private", "error": { "code": "blocked_by_robots", "message": "robots.txt disallows this page for WebReaderAPI." } } ], "next_cursor": "c_000002", "credits_used": 1 } ``` *curl* ```bash curl -sG "https://webreaderapi.com/api/v1/jobs/job_8b27f41c0e" \ -H "Authorization: Bearer $WEBREADER_API_KEY" \ -d cursor=c_000002 ``` *Python* ```python import os import requests resp = requests.get( "https://webreaderapi.com/api/v1/jobs/job_8b27f41c0e", headers={"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}, params={"cursor": "c_000002"}, timeout=30, ) job = resp.json() print(job["status"], len(job["results"]), job.get("next_cursor")) ``` *Node.js* ```javascript const res = await fetch('https://webreaderapi.com/api/v1/jobs/job_8b27f41c0e?cursor=c_000002', { headers: { Authorization: `Bearer ${process.env.WEBREADER_API_KEY}` }, }); const job = await res.json(); console.log(job.status, job.results.length, job.next_cursor); ``` ## How do I wait for a job to finish? Poll `/jobs/{job_id}` and keep the cursor. Process the results you get, then call again with `cursor` set to `next_cursor`. When `next_cursor` is absent and `status` is `done`, you have everything; when it is absent and the job is still `running`, wait and ask again with the same cursor. Start polling every 2 seconds and double the wait up to 30 seconds while nothing new arrives. A crawl that respects a site's Crawl-delay can take a while, so a long wait is normal and costs nothing. *Python: poll a job and save every page as a .md file* ```python import os import pathlib import re import time import requests JOBS = "https://webreaderapi.com/api/v1/jobs" HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]} def save_job(job_id, folder="pages"): out = pathlib.Path(folder) out.mkdir(exist_ok=True) cursor, wait, saved = None, 2, 0 while True: params = {"cursor": cursor} if cursor else {} job = requests.get(f"{JOBS}/{job_id}", headers=HEADERS, params=params, timeout=30).json() for item in job["results"]: if "error" in item: print("skipped", item["url"], item["error"]["code"]) continue name = re.sub(r"[^a-z0-9]+", "-", item["final_url"].lower()).strip("-")[:120] (out / f"{name}.md").write_text(item["markdown"], encoding="utf-8") saved += 1 if job.get("next_cursor"): cursor, wait = job["next_cursor"], 2 continue if job["status"] == "done": return saved time.sleep(wait) wait = min(wait * 2, 30) if __name__ == "__main__": print(save_job("job_8b27f41c0e"), "pages saved") ``` ## Which pages does a crawl read? A crawl starts from your URL and moves outward one link step at a time, breadth first, up to `max_depth`. It also reads the site's sitemap, from the address given in robots.txt or from `/sitemap.xml`, to find pages that few links point to. - **Same domain only.** Links to other domains are listed in the results when you ask for `include_links`, but they are not read. - **Path filters.** `include_paths` and `exclude_paths` take simple patterns such as `/blog/*`. - **One copy per page.** When a site declares a canonical URL, a page reached through several addresses is read once. - **A hard stop.** The job ends when it has read `limit` pages. ## How does a crawl respect the site? Every request carries the User-Agent `WebReaderAPI/1.0 (+https://webreaderapi.com/bot/)`, and the crawler follows the robots.txt rules for `WebReaderAPI`, or for all robots when none name it. Disallowed pages are skipped and appear in `results` with the error `blocked_by_robots`; they cost nothing. When robots.txt sets a `Crawl-delay`, the crawler waits that long between two requests to the site. If a site declares Content-Signal: ai-input=no in its robots.txt, we do not read its pages for you. If a site refuses our request, we return its answer: we never solve CAPTCHAs, log in to accounts, or try to get around a block or a paywall. ## How much does a crawl cost? Each page read costs what a `/read` call would: 1 credit, 2 if the page needed a browser, plus 10 with `country`. Skipped and failed pages are free. The job's `credits_used` adds up what has been billed so far. A 500-page documentation site read without a browser uses about 500 credits, which the 10,000 pages pack ($3.75) covers many times over. Since a job cannot be cancelled in v1, `limit` is your spending cap: set it to the most pages you are ready to pay for. ## Batch and crawl: common questions ### Should I use /batch or many /read calls? Use `/batch` when you already have the URLs and do not need each answer at once. It saves you from managing concurrency and retries. Use `/read` when an agent needs one page now. ### How long are job results kept? Collect them as soon as they are ready. Results wait until you fetch them; once delivered, no copy of the page content is kept, so store what you need on your side. ### Can a crawl follow links to other websites? No. A crawl reads one domain. Other domains appear as links when you ask for them, and you can send those URLs to `/batch` if you want them read. ### Can I stop a crawl that has started? Not in v1. The job stops at `limit` pages, so choose a limit that matches your budget. Pages already read stay billed; pages never read cost nothing. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Web page to Markdown with curl and jq > Turn web pages into Markdown from the terminal with curl and jq: save a page, list its links, run a batch and check credits. From $0.07/1K pages, no KYC. - URL: https://webreaderapi.com/docs/curl/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Everything the API does fits in a shell one-liner. These recipes use curl to call the API and jq to pick fields out of the JSON, with a PowerShell version for Windows at the end. **Quick answer — How do I convert a web page to Markdown from the command line?** Run `curl -sG https://webreaderapi.com/api/v1/read --data-urlencode "url=…"` with your key in an `Authorization: Bearer` header, then pipe the answer to `jq -r .markdown`. **WebReaderAPI** returns the page's main content as Markdown with its metadata. One plain page uses 1 credit, from $0.07/1K pages as of October 2026. No KYC; pay in crypto. ## How do I set up the terminal? Install curl and jq, which most Linux and macOS systems already have, and export your key once per session. Every recipe below reads it from `$WEBREADER_API_KEY`, so the key never appears in your shell history as part of a command. Use `-G` with `--data-urlencode` for GET requests: curl then encodes the page address for you, which matters because addresses contain `?`, `&` and `=` of their own. *Set the key and save one page as a .md file* ```bash export WEBREADER_API_KEY="your-key" curl -sG "https://webreaderapi.com/api/v1/read" \ -H "Authorization: Bearer $WEBREADER_API_KEY" \ --data-urlencode "url=https://example.com/guides/compost" \ | jq -r '.markdown' > compost.md ``` ## How do I see what a call returned and cost? Ask jq for the metadata instead of the content. `final_url` shows where redirects led, `rendered` tells you whether a browser was used, and `credits_used` is what the call cost. If the answer is an error, the same filter prints `null` fields; print `.error` to read the code and message. *Metadata only, or the error if there is one* ```bash curl -sG "https://webreaderapi.com/api/v1/read" \ -H "Authorization: Bearer $WEBREADER_API_KEY" \ --data-urlencode "url=https://example.com/guides/compost" \ | jq '.error // {title, final_url, rendered, credits_used, word_count}' ``` ## How do I list a page's links? Add `include_links=true` and turn the `links` list into tab-separated lines. That gives you a quick map of a page's outgoing links, which you can sort, filter with grep or feed to a batch job. *Links as text and URL, one per line* ```bash curl -sG "https://webreaderapi.com/api/v1/read" \ -H "Authorization: Bearer $WEBREADER_API_KEY" \ --data-urlencode "url=https://example.com/docs/" \ -d include_links=true \ | jq -r '.links[] | "\(.text)\t\(.url)"' | sort -u ``` ## How do I read a JavaScript page or a PDF? Add `-d render=true` for a page that builds its content with JavaScript; it then costs 2 credits. A PDF needs nothing special: send its URL and the text comes back in `markdown`. PDF files up to 20 MB and 100 pages are converted to Markdown, at 1 credit per started block of 10 pages. *A rendered page, then a PDF* ```bash curl -sG "https://webreaderapi.com/api/v1/read" -H "Authorization: Bearer $WEBREADER_API_KEY" \ --data-urlencode "url=https://example.com/app/changelog" -d render=true \ | jq -r '.markdown' > changelog.md curl -sG "https://webreaderapi.com/api/v1/read" -H "Authorization: Bearer $WEBREADER_API_KEY" \ --data-urlencode "url=https://example.com/files/annual-report.pdf" \ | jq -r '.markdown' > annual-report.md ``` ## How do I run a batch job from the shell? Post a JSON body with your URLs to `/batch`, keep the `job_id`, then read the job. A small loop waits until the job is `done` and prints each page's title or error code. For long jobs, follow `next_cursor` as shown on the [batch and crawl page](https://webreaderapi.com/docs/crawl/). *Batch of three URLs, then wait for the result* ```bash JOB=$(curl -s "https://webreaderapi.com/api/v1/batch" \ -H "Authorization: Bearer $WEBREADER_API_KEY" \ -H "Content-Type: application/json" \ -d '{"urls": ["https://example.com/a", "https://example.com/b", "https://example.com/c"]}' \ | jq -r '.job_id') until [ "$(curl -s "https://webreaderapi.com/api/v1/jobs/$JOB" -H "Authorization: Bearer $WEBREADER_API_KEY" | jq -r '.status')" = "done" ]; do sleep 5 done curl -s "https://webreaderapi.com/api/v1/jobs/$JOB" -H "Authorization: Bearer $WEBREADER_API_KEY" \ | jq -r '.results[] | "\(.url)\t\(.title // .error.code)"' ``` ## How do I check my credits? Call `/credits`; it is free. It returns `credits_remaining`, `credits_used_today` and your `concurrency_limit`. Prepaid credits never expire: unused pages stay on your account until you use them. *Remaining credits* ```bash curl -s "https://webreaderapi.com/api/v1/credits" -H "Authorization: Bearer $WEBREADER_API_KEY" | jq '.credits_remaining' ``` ## What is the PowerShell version on Windows? `Invoke-RestMethod` parses the JSON for you, so no jq is needed. Build the query with `[uri]::EscapeDataString` to encode the page address, and pass the key in a headers table. *PowerShell: save one page* ```powershell $headers = @{ Authorization = "Bearer $env:WEBREADER_API_KEY" } $target = [uri]::EscapeDataString('https://example.com/guides/compost') $page = Invoke-RestMethod -Uri "https://webreaderapi.com/api/v1/read?url=$target" -Headers $headers $page.markdown | Out-File -Encoding utf8 compost.md "$($page.title): $($page.word_count) words, $($page.credits_used) credit(s)" ``` ## curl: common questions ### Why does my URL with & in it fail? Because the shell or curl split it. Quote the whole `url=…` argument and pass it with `--data-urlencode`, as in the recipes above; curl then encodes `&`, `?` and `=` inside the page address. ### How do I see the HTTP status of an error? Add `-w '\n%{http_code}\n'` to print it after the body, or `-i` to see all headers. The body's `error.code` is the stable value to test in scripts. ### Is a failed call billed? No. 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Errors, robots.txt refusals and retries > Every WebReaderAPI error code: robots.txt refusals, site refusals, timeouts and credit errors, which ones to retry, and why none of them is ever charged. - URL: https://webreaderapi.com/docs/errors/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Each error has an HTTP status, a stable `code` and a readable `message`. None of them uses a credit. Some are worth a retry; the refusals that come from a site are final, and we never work around them. **Quick answer — What do WebReaderAPI error codes mean, and which ones should I retry?** **WebReaderAPI** errors come in three groups, and none is ever charged. Request problems (`invalid_url`, `invalid_api_key`, `insufficient_credits`) need a fix on your side. Site decisions (`blocked_by_robots`, `blocked_by_site_policy`, `target_refused`) are final. Temporary failures (`fetch_failed`, `fetch_timeout`, `concurrency_limit`) can be retried with backoff. Credits start at $0.07/1K pages, paid in crypto with no KYC. ## Which error codes can the API return? The body is always `{"error": {"code", "message"}}`, plus `target_status` when the refusal came from the site. | HTTP | `code` | Cause | Retry? | | --- | --- | --- | --- | | 400 | `invalid_url` | Not http or https, too long, or a private, local or cloud metadata address | No: fix the URL | | 400 | `invalid_parameter` | A parameter has a value outside its list or range | No: fix the request | | 401 | `invalid_api_key` | Key missing, wrong or revoked | No: check the key | | 402 | `insufficient_credits` | Your balance has no credits left | After buying a pack | | 403 | `blocked_by_robots` | The site's robots.txt disallows the page for WebReaderAPI or for all robots | No: final | | 403 | `blocked_by_site_policy` | The site declares `Content-Signal: ai-input=no` | No: final | | 422 | `unsupported_content` | A type we do not read (video, archive) or a file too large | No | | 429 | `concurrency_limit` | Too many of your requests are running at the same time | Yes, with backoff | | 502 | `target_refused` | The site refused: 403, 429, a challenge page or a login wall; its status is in `target_status` | No: final | | 502 | `fetch_failed` | DNS, TLS or connection error, or the site answered 404 or 410 | Once or twice, with backoff | | 504 | `fetch_timeout` | The site did not answer within `timeout` | Yes, with backoff or a longer timeout | ## Why are robots.txt and site refusals final? Because they are the site owner's decision, and WebReaderAPI is built to follow it. We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged. If a site declares Content-Signal: ai-input=no in its robots.txt, we do not read its pages for you. When a site turns a request away in another way, with a 403, a 429, a challenge page or a login form, you get `target_refused` and the site's own status in `target_status`. If a site refuses our request, we return its answer: we never solve CAPTCHAs, log in to accounts, or try to get around a block or a paywall. Retrying the same URL a minute later will usually get the same answer, so treat these codes as a result, not a failure: log them and move on. ## How do I check why a page was refused? Open the site's `/robots.txt` and look for a group that names `WebReaderAPI`, or the `*` group that applies to all robots. A `Disallow` line that matches the path explains a `blocked_by_robots`. A line such as `Content-Signal: ai-input=no` explains a `blocked_by_site_policy`. We keep a copy of each robots.txt for at most 24 hours. If a site has just opened a section to robots, the change can take up to a day to apply. Site owners can read how our crawler behaves, and how to allow or block it, on [our crawler page](https://webreaderapi.com/bot/). ## How should I retry temporary failures? Retry only `concurrency_limit`, `fetch_timeout` and, once or twice, `fetch_failed`. Wait before each new attempt and double the wait each time, with a little random jitter so parallel workers do not retry together. Stop after 4 or 5 attempts. For `concurrency_limit`, the better fix is to send fewer requests at once: Every account has a limit on how many pages it reads at once; your dashboard shows it, and support can raise it. For `fetch_timeout`, raise `timeout` up to 60 seconds before retrying a slow site. Retries cost nothing, because a failed attempt is never billed. *Python: retry only the errors that can change* ```python import os import random import time import requests API = "https://webreaderapi.com/api/v1/read" HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]} RETRY = {"concurrency_limit", "fetch_timeout", "fetch_failed"} FINAL = {"blocked_by_robots", "blocked_by_site_policy", "target_refused", "unsupported_content"} def read_page(url, attempts=5, **options): wait = 1.0 for attempt in range(1, attempts + 1): resp = requests.get(API, headers=HEADERS, params={"url": url, **options}, timeout=75) if resp.status_code == 200: return resp.json() error = resp.json().get("error", {}) code = error.get("code", "unknown") if code in FINAL: return {"url": url, "skipped": code, "target_status": error.get("target_status")} if code not in RETRY or attempt == attempts or (code == "fetch_failed" and attempt >= 3): raise RuntimeError(f"{code}: {error.get('message')}") time.sleep(wait + random.uniform(0, wait / 2)) wait = min(wait * 2, 30) raise RuntimeError("unreachable") if __name__ == "__main__": print(read_page("https://example.com/blog/first-post", max_chars=20000).get("title")) ``` *Node.js: the same rule with exponential backoff* ```javascript const RETRY = new Set(['concurrency_limit', 'fetch_timeout', 'fetch_failed']); const FINAL = new Set(['blocked_by_robots', 'blocked_by_site_policy', 'target_refused', 'unsupported_content']); const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); export async function readPage(url, options = {}, attempts = 5) { let wait = 1000; for (let attempt = 1; attempt <= attempts; attempt++) { const params = new URLSearchParams({ url, ...options }); const res = await fetch(`https://webreaderapi.com/api/v1/read?${params}`, { headers: { Authorization: `Bearer ${process.env.WEBREADER_API_KEY}` }, }); const body = await res.json(); if (res.ok) return body; const code = body.error?.code ?? 'unknown'; if (FINAL.has(code)) return { url, skipped: code, target_status: body.error.target_status }; if (!RETRY.has(code) || attempt === attempts) throw new Error(`${code}: ${body.error?.message}`); await sleep(wait + Math.random() * (wait / 2)); wait = Math.min(wait * 2, 30000); } } ``` ## What about errors inside a batch or crawl? A job does not stop because one page fails. That page appears in `results` as `{url, error}` with the same codes as above, and it costs nothing. A crawl that meets `blocked_by_robots` on a section simply reads the rest of the site. At the end of a job, count the error codes. Many `target_refused` on one site mean the site does not accept automated readers; many `fetch_timeout` mean it is slow, and a longer `timeout` in the job options may help. ## Errors: common questions ### Am I charged for a page that robots.txt blocks? No. 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged. The same goes for every other error on this page. ### Can you read a page that blocks your crawler if I ask? No. We follow the site owner's choice, and the policy is the same for every customer. If you own the site, add a rule for `WebReaderAPI` in your robots.txt; the [crawler page](https://webreaderapi.com/bot/) shows how. ### Why do I get fetch_failed for a page that exists? The site may have been down, slow to accept a TLS connection or returning 404 to automated clients. Retry once or twice with backoff. If it keeps failing, check the address in a browser. ### What should my agent tell the user after a refusal? Something simple, such as the site does not allow automated reading of that page. Then let the model use other sources. Returning the error code as the tool result is enough for most models. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # LangChain and LlamaIndex: a web page loader and a read_url tool > LangChain and LlamaIndex web page loader: a read_url tool, a document loader and a small RAG index built from a crawl, all in Python. No KYC, pay in crypto. - URL: https://webreaderapi.com/docs/langchain/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Two ways to use the API in a Python AI stack: load pages as documents for retrieval, or give an agent a `read_url` tool. The same 20 lines of glue serve LangChain and LlamaIndex. **Quick answer — How do I load web pages into LangChain or LlamaIndex as clean Markdown?** Call **WebReaderAPI**'s `/read` endpoint and wrap each answer in a `Document`: the Markdown as content, `final_url` and the title as metadata. For agents, expose the same call as a `read_url` tool. Pages cost 1 credit each, 2 with a browser, from $0.07/1K pages as of October 2026. No package of ours is needed. No KYC; pay in crypto. ## Why Markdown documents rather than raw HTML? Because retrieval works on text, and Markdown keeps the structure that matters while dropping the rest. Headings survive, so a splitter can cut on topic changes. Lists and tables survive, so steps and specifications stay together. Navigation, cookie banners and ads are gone, so they do not pollute your embeddings or waste tokens. The metadata comes from the same call: `final_url` as the source to cite, `title`, `published_at` and `language` for filtering. *reader.py: the shared helper (save it next to your code)* ```python import os import requests API = "https://webreaderapi.com/api/v1/read" HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]} class PageError(Exception): pass def read_page(url, **options): resp = requests.get(API, headers=HEADERS, params={"url": url, **options}, timeout=75) body = resp.json() if resp.status_code != 200: raise PageError(body["error"]["code"]) return body ``` ## How do I build a LangChain document loader? Turn each page into a `langchain_core.documents.Document`, then split it by headings with `MarkdownHeaderTextSplitter` from the `langchain-text-splitters` package. Each chunk keeps the heading path in its metadata, and you add the source URL so every answer can point back to its page. Skipped pages, such as those refused by robots.txt, raise `PageError` with the code; log them and continue. *LangChain: load and split pages* ```python from langchain_core.documents import Document from langchain_text_splitters import MarkdownHeaderTextSplitter from reader import PageError, read_page splitter = MarkdownHeaderTextSplitter(headers_to_split_on=[("#", "h1"), ("##", "h2"), ("###", "h3")]) def load_pages(urls): chunks = [] for url in urls: try: page = read_page(url) except PageError as exc: print("skipped", url, exc) continue for part in splitter.split_text(page["markdown"]): part.metadata.update({"source": page["final_url"], "title": page.get("title", "")}) chunks.append(part) return chunks docs = load_pages(["https://example.com/docs/install", "https://example.com/docs/configure"]) print(len(docs), "chunks", isinstance(docs[0], Document) if docs else "") ``` ## How do I give a LangChain agent a read_url tool? Decorate a function with `@tool` from `langchain_core.tools`. The docstring becomes the tool description the model reads, so say what the tool returns and when to use it. Return plain text: a title line, the source and the Markdown, cut with `max_chars` to suit your model. Bind the tool to any chat model that supports tool calling, or pass it to your agent constructor. The model then calls `read_url` with a URL from the conversation or from a search result. *LangChain: the read_url tool* ```python import os from langchain.chat_models import init_chat_model from langchain_core.tools import tool from reader import PageError, read_page @tool def read_url(url: str, max_chars: int = 20000) -> str: """Read a public web page and return its main content as Markdown, with its title and source URL.""" try: page = read_page(url, max_chars=max_chars) except PageError as exc: return "Could not read the page: " + str(exc) return "# " + (page.get("title") or url) + "\nSource: " + page["final_url"] + "\n\n" + page["markdown"] model = init_chat_model(os.environ.get("CHAT_MODEL", "openai:gpt-4.1-mini")).bind_tools([read_url]) reply = model.invoke("Summarise https://example.com/blog/first-post in three bullet points.") print(reply.tool_calls) ``` ## How do I do the same in LlamaIndex? LlamaIndex uses its own `Document` class with `text` and `metadata`, and `FunctionTool.from_defaults` to turn a function into a tool. The helper stays the same. Build an index from the documents with `VectorStoreIndex.from_documents`, which uses whatever embedding model your LlamaIndex settings name. *LlamaIndex: index a crawl, and a read_url tool* ```python import os import time import requests from llama_index.core import Document, VectorStoreIndex from llama_index.core.tools import FunctionTool from reader import HEADERS, PageError, read_page def crawl_pages(start_url, limit=200): job = requests.post("https://webreaderapi.com/api/v1/crawl", headers=HEADERS, json={"url": start_url, "limit": limit}, timeout=30).json() pages, cursor = [], None while True: params = {"cursor": cursor} if cursor else {} res = requests.get("https://webreaderapi.com/api/v1/jobs/" + job["job_id"], headers=HEADERS, params=params, timeout=30).json() pages += [r for r in res["results"] if "error" not in r] cursor = res.get("next_cursor") if not cursor and res["status"] == "done": return pages if not cursor: time.sleep(5) pages = crawl_pages("https://docs.example.com/") docs = [Document(text=p["markdown"], metadata={"source": p["final_url"], "title": p.get("title", "")}) for p in pages] index = VectorStoreIndex.from_documents(docs) print(index.as_query_engine().query("How do I install the product?")) def read_url(url: str, max_chars: int = 20000) -> str: """Read a public web page and return its main content as Markdown.""" try: return read_page(url, max_chars=max_chars)["markdown"] except PageError as exc: return "Could not read the page: " + str(exc) read_tool = FunctionTool.from_defaults(fn=read_url, name="read_url") print(os.environ.get("CHAT_MODEL", "set CHAT_MODEL for your agent"), read_tool.metadata.name) ``` ## How large a crawl does a small RAG index need? Less than you might think. A product's documentation is often a few hundred pages, and a crawl with `include_paths` set to the docs section reads only those. At 1 credit a page, a 300-page docs site costs about 300 credits; 10,000 pages ($3.75) refreshes it many times. Re-crawl on a schedule that matches how often the source changes, and compare `final_url` and `word_count` with the previous run to re-embed only the pages that changed. Prepaid credits never expire: unused pages stay on your account until you use them. ## LangChain and LlamaIndex: common questions ### Is there a WebReaderAPI package for LangChain or LlamaIndex? No. The helper and tool on this page are the whole integration, so you can adapt them freely. WebReaderAPI is a plain HTTPS API that any Python HTTP client can call. ### Which LangChain version does this need? Any recent one with `langchain_core`, `langchain-text-splitters` and `init_chat_model`. The tool decorator and the `Document` class have been stable across recent releases. ### How do I avoid re-reading pages that did not change? Store `final_url`, `word_count` and the date of each read. Re-read on a schedule, and re-embed a page only when its word count or content hash changes. Each re-read costs a credit, so pick a rhythm that matches the source. ### What if some pages are refused? They raise `PageError` with a code such as `blocked_by_robots`, and they cost nothing. If a site refuses our request, we return its answer: we never solve CAPTCHAs, log in to accounts, or try to get around a block or a paywall. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Web page reader MCP server for Claude, Cursor and other MCP clients > Web page reader MCP server for Claude Desktop, Cursor and other MCP clients: a 40-line Python read_url tool that returns clean Markdown. No KYC, pay in crypto. - URL: https://webreaderapi.com/docs/mcp/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Give any MCP client a `read_url` tool that turns a link into clean Markdown. The server is one short Python file that you run on your own machine; your API key stays in its environment. **Quick answer — How do I add a web page reader to Claude Desktop or Cursor with MCP?** Run a small MCP server that exposes a `read_url` tool calling **WebReaderAPI**, then register it in your client's MCP settings with your key in `env`. The model can then read any public page as clean Markdown. Each page read costs 1 credit, 2 with a browser, from $0.07/1K pages as of October 2026. No KYC; pay in crypto. ## The MCP server at a glance | Attribute | Value | | --- | --- | | Language | Python 3.10+, with the `mcp` and `httpx` packages | | Transport | stdio: the client starts the server as a local process | | Tool | `read_url(url, max_chars)` → title, source URL, then the page's Markdown | | Why not web_fetch | Claude Code and OpenClaw already ship a built-in tool with that name | | Key | Read from `WEBREADER_API_KEY` in the server's environment | | Billing | 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged | ## Why a read_url tool and not web_fetch? Because two tools with the same name confuse both the client and the model. Claude Code and OpenClaw already include a tool called `web_fetch`. A distinct name, `read_url`, and a clear description let the model pick ours when it needs the main content of a page, JavaScript pages and PDFs included. The tool returns text, not JSON. Models handle a short header and Markdown better than nested fields, and the header keeps the source URL in view so the answer can cite it. *webreader_mcp.py* ```python import os import httpx from mcp.server.fastmcp import FastMCP API = "https://webreaderapi.com/api/v1/read" KEY = os.environ["WEBREADER_API_KEY"] mcp = FastMCP("webreaderapi") @mcp.tool() def read_url(url: str, max_chars: int = 20000) -> str: """Read a public web page and return its main content as Markdown. Use it when the user gives a link or when a page must be read to answer. max_chars caps the length of the returned text (1,000 to 500,000). """ params = {"url": url, "max_chars": max(1000, min(int(max_chars), 500000))} try: resp = httpx.get(API, params=params, headers={"Authorization": "Bearer " + KEY}, timeout=75) except httpx.HTTPError as exc: return "Network error while reading the page: " + str(exc) body = resp.json() if resp.status_code != 200: err = body.get("error", {}) return "Could not read the page: " + err.get("code", str(resp.status_code)) + ". " + err.get("message", "") title = body.get("title") or url return "# " + title + "\nSource: " + body.get("final_url", url) + "\n\n" + body.get("markdown", "") if __name__ == "__main__": mcp.run() ``` ## How do I install and test the server? Create a virtual environment, install `mcp` and `httpx`, and save the file above. The `mcp` package is the official Python SDK of the Model Context Protocol; `FastMCP` turns a typed Python function into a tool, using the docstring as its description. Before wiring it into a client, run the file once from a terminal with your key set. It waits silently for a client on stdin, which means it started correctly; stop it with Ctrl+C. *Install* ```bash python -m venv .venv . .venv/bin/activate pip install "mcp[cli]" httpx WEBREADER_API_KEY="your-key" python webreader_mcp.py ``` ## How do I register it in Claude Desktop? Open Claude Desktop's developer settings, edit `claude_desktop_config.json`, and add an entry under `mcpServers`. Use absolute paths: the Python of your virtual environment and the server file. Put the key in `env`, so it never appears in the tool's arguments or in the conversation. Restart Claude Desktop, and `read_url` appears in the tools list. Cursor reads the same `mcpServers` structure from its own MCP configuration file, and Claude Code can register the same command with its `claude mcp add` command. For OpenClaw, see the [OpenClaw page](https://webreaderapi.com/docs/openclaw/). *claude_desktop_config.json* ```json { "mcpServers": { "webreaderapi": { "command": "/home/you/webreader/.venv/bin/python", "args": ["/home/you/webreader/webreader_mcp.py"], "env": { "WEBREADER_API_KEY": "your-key" } } } } ``` ## How do I keep pages within the model's context? Lower `max_chars`. The default of 20,000 characters is enough for most articles and keeps a long page from filling the context. A model reading several pages in one answer does better with 8,000 to 12,000 per page. Ask the model to read the pages it needs, not every link it sees. A short sentence in your project instructions, such as "read at most three pages per question", keeps both cost and context in check. ## What happens when a site refuses the read? The tool returns a short message with the error code, such as `blocked_by_robots` or `target_refused`, and nothing is billed. We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged. If a site refuses our request, we return its answer: we never solve CAPTCHAs, log in to accounts, or try to get around a block or a paywall. The model can then say the page is not available to automated readers and use another source. ## MCP server: common questions ### Is there an official WebReaderAPI MCP package? No. The file on this page is the integration: 40 lines you can read, change and run yourself. Nothing to install from us, and your key never leaves your machine except in calls to our API. ### Can the model search the web with this server? No. `read_url` reads a page you or the model already have the address of. Pair it with any search tool your client offers; the model searches, then reads the pages that matter. ### How much does an assistant cost to run with it? One credit per page read, two when a browser is needed. An assistant that reads 50 pages a day uses about 1,500 credits a month, and 10,000 pages ($3.75) covers that for about six months. ### Does the server work offline? No. It calls `https://webreaderapi.com/api/v1/read` for every page, so it needs internet access and a valid key. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Read web pages in n8n AI agents with the WebReaderAPI > Read web pages in n8n AI agents: HTTP Request tool settings that turn any URL into clean Markdown, and a scheduled workflow. No KYC, pay in crypto. - URL: https://webreaderapi.com/docs/n8n/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) n8n's HTTP Request node can act as a tool for its AI Agent node. Point it at `/read`, let the model fill in the URL, and the agent can read any public page as clean Markdown, with no community node to install. **Quick answer — How do I let an n8n AI agent read web pages?** Attach an HTTP Request node to n8n's AI Agent as a tool, set it to `GET https://webreaderapi.com/api/v1/read` with a Header Auth credential holding your **WebReaderAPI** key, and let the model define the `url` parameter. Turn on Optimize Response to keep only `title`, `final_url` and `markdown`. Each page costs 1 credit, from $0.07/1K pages as of October 2026. No KYC; pay in crypto. ## Which HTTP Request settings do I use? Option names follow n8n's HTTP Request node documentation, read on 2026-10-22. | Setting | Value | | --- | --- | | Placement | Attached to the **AI Agent** node as a tool | | Method | `GET` | | URL | `https://webreaderapi.com/api/v1/read` | | Authentication | Generic credentials → **Header Auth** | | Header Auth credential | Name `Authorization`, Value `Bearer ` followed by your API key | | Send Query Parameters | On, **Using Fields Below** | | Parameter `url` | Defined by the model (see below) | | Parameter `max_chars` | A fixed value such as `15000` | | Optimize Response | On · response type **JSON** · Include Fields **Selected**: `title`, `final_url`, `markdown` | ## Why use the HTTP Request node as the tool? Because it already does everything this API needs: one GET request, one header and two query parameters. n8n documents that the HTTP Request node can run as a regular node or be attached to an AI agent as a tool, and that some options exist only in the tool form. There is nothing of ours to install or update, and the key sits in n8n's credential store rather than in the workflow. Give n8n its own key, so its usage is visible apart from your other scripts and you can revoke it alone. Create several API keys, for example one per agent or project, and revoke any of them from your dashboard. ## How does the model choose the URL to read? Let the model define the `url` parameter. n8n shows a button next to eligible parameter fields that hands the value to the model, and you can also write the expression yourself with n8n's `$fromAI()` function. It takes a key, a description, a type and an optional default, for example the key `url`, the description "The http or https address of the page to read" and the type `string`. n8n documents that `$fromAI()` works only in tools connected to the AI Agent node. Write the tool description in plain words too, such as "Reads a public web page and returns its main content as Markdown", so the agent knows when to call it. ## Why turn on Optimize Response? Because everything a tool returns goes into the model's context. Optimize Response is a tool-only option that reduces the data passed to the model. With the JSON response type you can include only selected fields; keep `title`, `final_url` and `markdown` and drop the rest. Cap the length on our side with `max_chars`, which cuts the Markdown before it leaves the API. The HTML and Text response types of the node have their own Truncate Response option, but with our JSON answer `max_chars` is the simpler control. ## How do I use the API in a scheduled workflow? Use the HTTP Request node as a regular node, not a tool, behind a Schedule Trigger. Give it a fixed `url`, such as a changelog or a pricing page you follow, and send the `markdown` field to the next node: a file, a database, a chat message or a comparison with the previous run. Each run reads one page per URL. Watching 20 pages once a day uses about 600 credits a month for pages without a browser, so 10,000 pages ($3.75) lasts more than a year. Prepaid credits never expire: unused pages stay on your account until you use them. ## What happens when a site refuses the read? The node receives an error response with a code such as `blocked_by_robots` or `target_refused`, and nothing is billed. We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged. In an agent, let the tool return the error so the model can explain it or try another source; in a scheduled workflow, route errors to a branch that logs them. The [errors page](https://webreaderapi.com/docs/errors/) lists every code. ## Can I try the call before building the workflow? Run the same call in a terminal with curl, as shown in the [curl recipes](https://webreaderapi.com/docs/curl/). If it works there and fails in n8n, the cause is in the node settings, most often the credential value missing the `Bearer ` prefix or the `url` parameter left empty. ## n8n: common questions ### Do I need a community node for WebReaderAPI? No. The built-in HTTP Request node is enough, as a tool for the AI Agent or as a regular node in any workflow. ### Can the agent read JavaScript pages or PDFs? Yes. The API decides when a browser is needed (`render=auto`) and reads PDFs as text. A rendered page uses 2 credits; PDF files up to 20 MB and 100 pages are converted to Markdown, at 1 credit per started block of 10 pages. ### How do I stop the agent from reading too many pages? Say how many pages it may read in the agent's system message, and keep `max_chars` modest. Each read is one tool call, so any limit you set on the agent's tool calls also caps the number of pages. ### How do I pay for credits? Top up from $25 in BTC, USDT, USDC, ETH, SOL, LTC, then buy a pack. WebReaderAPI asks for no ID, no selfie and no company documents. ## Sources 1. [HTTP Request node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.httprequest/) — n8n docs (read 2026-10-22) 2. [HTTP Request credentials](https://docs.n8n.io/integrations/builtin/credentials/httprequest/) — n8n docs (read 2026-10-22) 3. [AI Agent node](https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.agent/) — n8n docs (read 2026-10-22) 4. [Let AI specify tool parameters with $fromAI()](https://docs.n8n.io/advanced-ai/examples/using-the-fromai-function/) — n8n docs (read 2026-10-22) --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Web page to Markdown in Node.js > Convert web pages to Markdown in Node.js with native fetch: one page, a small concurrency pool and a crawl polling loop. From $0.07/1K pages, no KYC, crypto. - URL: https://webreaderapi.com/docs/node/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Node.js 18 and later ship `fetch`, so you need no dependency at all. Three recipes: read one page, read a list with a small concurrency pool, and follow a crawl job until it is done. **Quick answer — How do I convert a web page to Markdown in Node.js?** Use the built-in `fetch` of Node.js 18+: call `https://webreaderapi.com/api/v1/read?url=…` with your key in the `Authorization` header and read `markdown` from the JSON answer. **WebReaderAPI** returns the page's main content with its title, author and date. One page costs 1 credit, 2 with a browser, from $0.07/1K pages as of October 2026. No KYC; pay in crypto. ## What do I need? Node.js 18 or later, for the global `fetch`, and an API key in the `WEBREADER_API_KEY` environment variable. The examples are ES modules: save them with the `.mjs` extension, or set `"type": "module"` in your `package.json`, so that top-level `await` works. The JSDoc comments describe the response shape, so editors such as VS Code give you completion without a TypeScript build step. *read.mjs: one page* ```javascript /** * @typedef {Object} Page * @property {string} url * @property {string} final_url * @property {number} status_code * @property {boolean} rendered * @property {number} credits_used * @property {string} [title] * @property {string} [author] * @property {string} [published_at] * @property {string} [language] * @property {string} [markdown] * @property {number} word_count */ /** @returns {Promise} */ export async function readPage(url, options = {}) { const params = new URLSearchParams({ url, ...options }); const res = await fetch(`https://webreaderapi.com/api/v1/read?${params}`, { headers: { Authorization: `Bearer ${process.env.WEBREADER_API_KEY}` }, signal: AbortSignal.timeout(75_000), }); const body = await res.json(); if (!res.ok) throw Object.assign(new Error(body.error.message), { code: body.error.code }); return body; } const page = await readPage('https://example.com/docs/getting-started', { max_chars: '20000' }); console.log(page.title, page.word_count, 'words,', page.credits_used, 'credit(s)'); ``` ## How do I read a list of URLs without hitting the limit? Run a fixed number of workers that take URLs from a shared queue. It is simpler than `Promise.all` over the whole list, which would start every request at once and hit `429 concurrency_limit`. Every account has a limit on how many pages it reads at once; your dashboard shows it, and support can raise it. The pool below keeps errors next to the results, so one refused page does not stop the run. Refusals such as `blocked_by_robots` are final and free; count them and move on. *pool.mjs: a list of URLs, 8 at a time* ```javascript import { readPage } from './read.mjs'; export async function readAll(urls, workers = 8) { const queue = [...urls]; const results = []; async function worker() { while (queue.length) { const url = queue.shift(); try { const page = await readPage(url, { max_chars: '50000' }); results.push({ url, page }); } catch (err) { results.push({ url, error: err.code ?? err.name }); } } } await Promise.all(Array.from({ length: workers }, worker)); return results; } const urls = Array.from({ length: 30 }, (_, i) => `https://example.com/blog/post-${i + 1}`); for (const r of await readAll(urls)) console.log(r.url, '->', r.error ?? r.page.title); ``` ## How do I follow a crawl job in Node.js? Start the job with `POST https://webreaderapi.com/api/v1/crawl`, then read `https://webreaderapi.com/api/v1/jobs/{job_id}` in a loop. Pass `cursor` each time `next_cursor` comes back; when there is no cursor and the status is `done`, the job is finished. Wait between empty polls and double the wait up to 30 seconds, since a crawl that respects a site's Crawl-delay can be slow. The [batch and crawl page](https://webreaderapi.com/docs/crawl/) describes how the crawl chooses its pages and how `limit` caps what it reads. *crawl.mjs: start a crawl and collect every page* ```javascript const headers = { Authorization: `Bearer ${process.env.WEBREADER_API_KEY}`, 'Content-Type': 'application/json', }; const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); const start = await fetch('https://webreaderapi.com/api/v1/crawl', { method: 'POST', headers, body: JSON.stringify({ url: 'https://docs.example.com/', limit: 300, include_paths: ['/docs/*'] }), }); const { job_id } = await start.json(); const pages = []; let cursor = ''; let wait = 2000; for (;;) { const url = `https://webreaderapi.com/api/v1/jobs/${job_id}${cursor ? `?cursor=${encodeURIComponent(cursor)}` : ''}`; const job = await (await fetch(url, { headers })).json(); pages.push(...job.results.filter((r) => !r.error)); if (job.next_cursor) { cursor = job.next_cursor; wait = 2000; continue; } if (job.status === 'done') break; await sleep(wait); wait = Math.min(wait * 2, 30000); } console.log(pages.length, 'pages read, job', job_id); ``` ## How do I pass the result to a model? Send the Markdown, not the whole JSON. The `markdown` field is what a model should read; the metadata is better kept for citations. Prefix it with the title and `final_url`, so the model can name its source, and keep `max_chars` at a size your model's context can hold, such as 20,000 characters per page. The [OpenAI function-calling page](https://webreaderapi.com/docs/openai/) shows the full tool loop in Python; the same `readPage` function above can serve as the tool body in a Node.js agent. ## Node.js: common questions ### Does this work in Deno, Bun or the browser? Deno and Bun support the same `fetch` and ES modules. Do not call the API from a browser page: the key would be visible to every visitor. Call it from your server instead. ### Is there an npm package? No. The functions on this page are the whole integration. WebReaderAPI is plain HTTPS and JSON, so you keep control of retries, timeouts and logging. ### How do I handle 429 concurrency_limit? Lower the number of workers, or retry after a short wait that doubles each time. A 429 costs nothing, and the [errors page](https://webreaderapi.com/docs/errors/) has a ready backoff function. ### What does reading 10,000 pages cost? About 10,000 credits for plain pages, more if some need a browser. 10,000 pages cost $3.75; Prepaid credits never expire: unused pages stay on your account until you use them. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # OpenAI function calling: a read_url tool for your model > OpenAI function calling with a read_url tool: the JSON schema and the Python loop that turns links into clean Markdown for the model. No KYC, pay in crypto. - URL: https://webreaderapi.com/docs/openai/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Describe one function, `read_url`, and run it when the model asks. The model chooses which pages to read; your code calls the API and hands back the Markdown. The same schema works with other models that support tool calling. **Quick answer — How do I let an OpenAI model read web pages with function calling?** Declare a `read_url` function in the `tools` list, then run a loop: when the model returns a tool call, call **WebReaderAPI**'s `/read` endpoint with the URL and send the Markdown back as a `tool` message. Each page costs 1 credit, 2 with a browser, from $0.07/1K pages as of October 2026. No KYC; pay in crypto. ## What does the read_url schema look like? Two parameters are enough: the `url` to read, required, and an optional `max_chars` that lets the model keep a long page short. The description is the most important part: it tells the model what the tool returns and when to use it. Keep it factual and short. We name the tool `read_url` rather than `web_fetch` to avoid clashing with tools that some agent runtimes already provide under that name. *The tool definition* ```json { "type": "function", "function": { "name": "read_url", "description": "Read a public web page and return its main content as Markdown, with its title and source URL. Use it when the user gives a link or when a page must be read to answer.", "parameters": { "type": "object", "properties": { "url": { "type": "string", "description": "The http or https address of the page." }, "max_chars": { "type": "integer", "description": "Most characters to return, 1000 to 500000. Default 20000." } }, "required": ["url"], "additionalProperties": false } } } ``` ## How does the tool-calling loop work? Send the conversation with the `tools` list. If the reply contains tool calls, run each one, append the result as a message with role `tool` and the matching `tool_call_id`, and ask the model again. When a reply comes back with no tool call, it is the final answer. Cap the number of rounds. Three or four is plenty for a question about a handful of pages, and the cap protects you from a model that keeps asking for more. *Python: the full loop with the openai package* ```python import json import os import requests from openai import OpenAI READ_URL = { "type": "function", "function": { "name": "read_url", "description": "Read a public web page and return its main content as Markdown, with its title and source URL.", "parameters": { "type": "object", "properties": { "url": {"type": "string", "description": "The http or https address of the page."}, "max_chars": {"type": "integer", "description": "Most characters to return. Default 20000."}, }, "required": ["url"], "additionalProperties": False, }, }, } def read_url(url, max_chars=20000): resp = requests.get( "https://webreaderapi.com/api/v1/read", headers={"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}, params={"url": url, "max_chars": max_chars}, timeout=75, ) body = resp.json() if resp.status_code != 200: return "Could not read the page: " + body["error"]["code"] return "# " + (body.get("title") or url) + "\nSource: " + body["final_url"] + "\n\n" + body["markdown"] client = OpenAI() model = os.environ.get("OPENAI_MODEL", "gpt-4.1-mini") messages = [{"role": "user", "content": "What does https://example.com/blog/first-post announce? Answer in two sentences."}] for _ in range(4): reply = client.chat.completions.create(model=model, messages=messages, tools=[READ_URL]) msg = reply.choices[0].message if not msg.tool_calls: print(msg.content) break messages.append(msg) for call in msg.tool_calls: args = json.loads(call.function.arguments) messages.append({"role": "tool", "tool_call_id": call.id, "content": read_url(**args)}) ``` ## How do I keep token use and cost down? Three settings matter. Keep `max_chars` modest, around 8,000 to 20,000 characters per page, because every character you return is read by the model. Keep the default `main_content=true`, so menus and footers never reach the context. And cap the loop, as above, so one question cannot trigger dozens of reads. On our side, cost is per page, not per character: 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged. A support assistant that reads 300 pages a day uses about 9,000 credits a month, inside 10,000 pages ($3.75). ## Does the schema work with other models? Yes, with small changes in the wrapper. Most tool-calling APIs take a JSON Schema for the parameters and a name and description for the tool, so `read_url` moves across providers as is. The [MCP page](https://webreaderapi.com/docs/mcp/) packages the same function for clients that speak the Model Context Protocol, and the [LangChain page](https://webreaderapi.com/docs/langchain/) wraps it for LangChain and LlamaIndex agents. ## What should the model see when a page is refused? A short, clear sentence, as the loop above returns: "Could not read the page: blocked_by_robots". Models handle that well and fall back to other sources or tell the user. We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged. If a site refuses our request, we return its answer: we never solve CAPTCHAs, log in to accounts, or try to get around a block or a paywall. None of these refusals uses a credit. ## Function calling: common questions ### Does this need the Responses API or Chat Completions? The example uses Chat Completions with the `tools` parameter. The same function definition and the same `read_url` body work with other OpenAI interfaces that accept function tools; only the loop around them changes. ### Can the model read several pages in one turn? Yes. A reply can contain several tool calls; the loop runs each and returns every result before asking the model again. Keep `max_chars` lower when the model reads many pages at once. ### Is the API key exposed to the model? No. The key stays in your code's environment. The model only sees the tool's name, its description and the text your function returns. ### What does a refused page cost? Nothing. 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # OpenClaw: add a web page reader through MCP, next to web_fetch > Add a web page reader to OpenClaw through MCP: openclaw mcp add, doctor --probe, and how read_url complements the built-in web_fetch. No KYC, pay in crypto. - URL: https://webreaderapi.com/docs/openclaw/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) OpenClaw already has a `web_fetch` tool for light pages. Adding our MCP server gives the agent a second reader, `read_url`, for JavaScript pages, PDFs and longer documents. Setup is one command. **Quick answer — How do I add a web page reader API to OpenClaw?** Save the **WebReaderAPI** MCP server, then register it with `openclaw mcp add webreaderapi --command … --arg … --env WEBREADER_API_KEY=…` and check it with `openclaw mcp doctor --probe`. The agent gets a `read_url` tool next to the built-in `web_fetch`, able to render JavaScript and read PDFs. Pages cost 1 credit, from $0.07/1K pages as of October 2026. No KYC; pay in crypto. ## OpenClaw's web_fetch and read_url side by side | Attribute | Value | | --- | --- | | web_fetch (built in) | Runs locally, extracts the main content with Readability, returns Markdown or text | | web_fetch limits | No JavaScript execution; `maxChars` default and cap of 20,000 characters (OpenClaw docs) | | read_url (our MCP tool) | JavaScript rendering when needed, PDFs up to 100 pages, up to 500,000 characters | | read_url cost | 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged | | Our crawler | `WebReaderAPI/1.0 (+https://webreaderapi.com/bot/)`, robots.txt respected | | Tool profiles | MCP tools appear in the `coding` and `messaging` profiles, not in `minimal` | ## Why add a reader when OpenClaw has web_fetch? Because they cover different pages. OpenClaw's documentation says `web_fetch` always runs locally, uses Readability for main-content extraction, and does not execute JavaScript; it points to its Web Browser tool for JavaScript-heavy or login-protected sites. Its `maxChars` setting defaults to 20,000 characters and is capped there. Keep `web_fetch` for quick, light pages. Add `read_url` for pages that build their content with JavaScript, for PDFs, and for long documents where you want more than 20,000 characters. OpenClaw's docs also note that the official Firecrawl plugin is today the only bundled provider that `web_fetch` can fall back to. ## How do I register the MCP server in OpenClaw? First set up the Python server from the [MCP page](https://webreaderapi.com/docs/mcp/): one file plus the `mcp` and `httpx` packages. Then save it in OpenClaw with `openclaw mcp add`. OpenClaw's registry documentation describes stdio flags `--command`, `--arg`, `--env` and `--cwd`, and stores saved definitions under `mcp.servers` in its configuration. Run `openclaw mcp doctor --probe` afterwards. Without `--probe` the doctor only checks the saved configuration; with it, it also confirms that enabled servers connect. *Register and check the server* ```bash openclaw mcp add webreaderapi \ --command /home/you/webreader/.venv/bin/python \ --arg /home/you/webreader/webreader_mcp.py \ --env WEBREADER_API_KEY=your-key openclaw mcp doctor webreaderapi --probe ``` ## Will the agent see read_url? In the usual profiles, yes. OpenClaw's registry documentation says embedded OpenClaw exposes configured MCP tools in the `coding` and `messaging` profiles. The `minimal` profile hides them, and `tools.deny: ["bundle-mcp"]` turns them off, so check those two settings if the tool does not show up. The tool's description tells the model when to prefer it: pages that need JavaScript, PDFs and long reads. The model can still use `web_fetch` for everything else. ## How does read_url treat the sites it reads? The same way as every call to our API. We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged. If a site declares Content-Signal: ai-input=no in its robots.txt, we do not read its pages for you. If a site refuses our request, we return its answer: we never solve CAPTCHAs, log in to accounts, or try to get around a block or a paywall. Pages behind a login are therefore out of scope for `read_url`. For a site you are signed in to, OpenClaw's own browser tool, running with your session, is the right tool. ## What will reading pages through OpenClaw cost? One credit per page that comes back with content, two when a browser is needed, nothing for refusals and errors. An assistant that reads 30 pages a day across its chat channels uses about 900 credits a month: 10,000 pages ($3.75) lasts close to a year. Prepaid credits never expire: unused pages stay on your account until you use them. Give OpenClaw its own API key, so its usage shows apart from your other projects. ## OpenClaw: common questions ### Is WebReaderAPI a built-in OpenClaw provider? No. It connects as an MCP server, which OpenClaw supports without any plugin code. The MCP server is the short Python file from our [MCP page](https://webreaderapi.com/docs/mcp/). ### Should I disable web_fetch? No need. Both tools can stay on; the descriptions tell the model which one fits. Disable `web_fetch` only if you want every read to go through one place for logging. ### Can I manage the server from OpenClaw's web interface? OpenClaw's documentation describes MCP settings in its Control UI, which edit the same saved definitions as the command line. Use whichever you prefer. ### Does it work in a sandboxed OpenClaw setup? The MCP server is a local process that needs internet access to reach `https://webreaderapi.com/api/v1/read`. Check that your sandbox allows the process to start and to make that HTTPS call. ## Sources 1. [Web tools](https://docs.openclaw.ai/tools/web) — OpenClaw docs (read 2026-10-22) 2. [Web Fetch](https://docs.openclaw.ai/tools/web-fetch) — OpenClaw docs (read 2026-10-22) 3. [Manage saved MCP servers](https://docs.openclaw.ai/cli/mcp/registry) — OpenClaw docs (read 2026-10-22) 4. [MCP in the Control UI](https://docs.openclaw.ai/cli/mcp/control-ui) — OpenClaw docs (read 2026-10-22) --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Web page to Markdown in Python > Convert web pages to Markdown in Python: requests, async httpx with a concurrency cap, batch jobs and RAG chunking by headings. From $0.07/1K pages, no KYC. - URL: https://webreaderapi.com/docs/python/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Four recipes that cover most Python projects: read one page with requests, read many pages concurrently with httpx, hand a list to a batch job, and split the Markdown into chunks for a vector store. **Quick answer — How do I convert a web page to Markdown in Python?** Call **WebReaderAPI** with `requests.get("https://webreaderapi.com/api/v1/read", params={"url": ...})` and your key in the `Authorization` header; the JSON answer holds the page's main content in `markdown`. For many pages, use `httpx.AsyncClient` with a semaphore or a batch job. One page costs 1 credit, from $0.07/1K pages as of October 2026. No KYC; pay in crypto. ## What do I need installed? Python 3.9 or later and one HTTP client. `requests` is enough for scripts that read a page at a time; `httpx` adds async support for many pages in parallel. No package of ours is needed: the API is plain HTTPS and JSON. Keep your key out of the code. Read it from the `WEBREADER_API_KEY` environment variable, as every example below does, and give each project its own key so you can revoke one without touching the others. *Install* ```bash pip install requests httpx export WEBREADER_API_KEY="your-key" ``` ## How do I read one page with requests? Send `url` and any options as query parameters, raise on an HTTP error, and read the fields you need. The example keeps the main content, caps it at 20,000 characters for a model, and prints what the call cost. *requests: one page* ```python import os import requests API = "https://webreaderapi.com/api/v1/read" HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]} def read(url, **options): resp = requests.get(API, headers=HEADERS, params={"url": url, **options}, timeout=75) if resp.status_code != 200: error = resp.json()["error"] raise RuntimeError(error["code"] + ": " + error["message"]) return resp.json() page = read("https://example.com/docs/getting-started", max_chars=20000) print(page["title"], page["word_count"], "words", page["credits_used"], "credit(s)") print(page["markdown"][:500]) ``` ## How do I read many pages at once with httpx? Use one `httpx.AsyncClient` and an `asyncio.Semaphore` set at or below your concurrency limit. Every account has a limit on how many pages it reads at once; your dashboard shows it, and support can raise it. Errors from one page should not stop the others, so the example collects them next to the results instead of raising. If you go above the limit, the API answers `429 concurrency_limit`; nothing is billed, but the page is not read. Keeping the semaphore below the limit is simpler than retrying. *httpx: up to 10 pages in flight* ```python import asyncio import os import httpx API = "https://webreaderapi.com/api/v1/read" HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]} async def read_one(client, sem, url): async with sem: resp = await client.get(API, params={"url": url, "max_chars": 50000}) body = resp.json() if resp.status_code == 200: return url, body, None return url, None, body["error"]["code"] async def read_many(urls, limit=10): sem = asyncio.Semaphore(limit) async with httpx.AsyncClient(headers=HEADERS, timeout=75) as client: return await asyncio.gather(*(read_one(client, sem, u) for u in urls)) if __name__ == "__main__": urls = [f"https://example.com/blog/post-{n}" for n in range(1, 21)] for url, page, error in asyncio.run(read_many(urls)): print(url, "->", error or page["title"]) ``` ## When should I use a batch job instead? When you have a list of up to 100 URLs and do not need each page the moment it is ready. A batch job runs on our side, so your script holds no connections open and does no retry logic. Start it with `POST https://webreaderapi.com/api/v1/batch`, then collect results from the job with the cursor loop shown on the [batch and crawl page](https://webreaderapi.com/docs/crawl/). For a whole site, start a crawl instead and let it follow the sitemap and links; the same page has a complete loop that writes one `.md` file per page. *Start a batch job* ```python import os import requests urls = [f"https://example.com/changelog/{year}" for year in range(2020, 2027)] resp = requests.post( "https://webreaderapi.com/api/v1/batch", headers={"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}, json={"urls": urls, "options": {"format": "markdown", "main_content": True}}, timeout=30, ) resp.raise_for_status() print("job:", resp.json()["job_id"]) ``` ## How do I split the Markdown into chunks for RAG? Split on headings first, then on size. Headings mark where topics change, so a chunk that starts at a heading is easier for a retriever to match. Keep the page title and the heading path in each chunk's metadata: they help the model cite the right section. The function below needs no extra library. It cuts at `#` to `###` headings, then splits any long section on blank lines to stay under a character budget. Store `final_url` with each chunk, so two addresses that lead to the same page do not create duplicates. *Chunk Markdown by headings* ```python import re def chunk_markdown(page, max_chars=2000): """Split a /read response into chunks with a heading path and the source URL.""" chunks, path, buf = [], [], [] def flush(): text = "\n".join(buf).strip() while text: part = text[:max_chars] cut = part.rfind("\n\n") if len(text) > max_chars and cut > max_chars // 2: part = part[:cut] chunks.append({"text": part.strip(), "section": " > ".join(path), "title": page.get("title"), "source": page["final_url"]}) text = text[len(part):].strip() buf.clear() for line in page["markdown"].splitlines(): m = re.match(r"^(#{1,3}) (.+)", line) if m: flush() level = len(m.group(1)) path[:] = path[: level - 1] + [m.group(2).strip()] buf.append(line) flush() return chunks ``` ## How much does a Python job cost? Count pages, not requests. 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged. Reading 2,000 documentation pages without a browser uses about 2,000 credits, which 10,000 pages ($3.75) covers five times over. Prepaid credits never expire: unused pages stay on your account until you use them, so a pipeline that runs once a month uses only what it reads. ## Python: common questions ### Is there an official Python package? No. The code on this page is all you need: WebReaderAPI is a plain HTTPS API, and `requests` or `httpx` speak it directly. ### How do I handle a page that robots.txt refuses? Check the error code. `blocked_by_robots` and `target_refused` are final and free; record them and continue. The [errors page](https://webreaderapi.com/docs/errors/) has a retry helper that only retries the codes that can change. ### Can I use the API from a Jupyter notebook? Yes. The `requests` recipe works as is. In a notebook, `await read_many(urls)` replaces `asyncio.run`, because the notebook already runs an event loop. ### Which timeout should my client use? A little more than the API's own: the default is 30 seconds and the maximum 60, so 75 seconds on the client side leaves room for a slow site and the response. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Read endpoint reference: GET and POST /read > Read endpoint reference: every parameter of GET and POST /read, from format and render to country and max_chars, with examples. From $0.07/1K pages, no KYC. - URL: https://webreaderapi.com/docs/read/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) `/read` fetches one public page, extracts its main content and returns it as Markdown, plain text or cleaned HTML. This page lists every parameter, its default and what it costs. **Quick answer — What parameters does the WebReaderAPI read endpoint take?** **WebReaderAPI**'s `/read` endpoint needs one parameter, `url`. Optional ones choose the output (`format`), how much of the page to keep (`main_content`), whether a browser runs (`render`), links and images, the reading country, a character cap and a timeout. A plain page costs 1 credit, 2 with a browser, from $0.07/1K pages as of October 2026. No KYC; pay in crypto. ## /read at a glance | Attribute | Value | | --- | --- | | Methods | `GET` with query parameters, or `POST` with a JSON body using the same names | | URL | `https://webreaderapi.com/api/v1/read` | | Required | `url`: an http or https address, up to 2,048 characters | | Default output | Markdown of the main content | | Billing | 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged | | Country | Choosing the country a page is read from adds 10 credits, because the page is then read through a residential connection in that country | | PDF | PDF files up to 20 MB and 100 pages are converted to Markdown, at 1 credit per started block of 10 pages | | Crawler identity | `WebReaderAPI/1.0 (+https://webreaderapi.com/bot/)` | ## Read one page (GET) `GET /api/v1/read` Fetches one public page and returns its main content as Markdown, with the page's metadata. Parameters go in the query string, URL-encoded. The example reads a blog post; IDs, dates and values are illustrative. Authentication: API key in the `Authorization: Bearer API_KEY` header. **Parameters** | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | `url` | query | string | Yes | The page to read: http or https, at most 2,048 characters. Private, local and cloud metadata addresses are refused. | | `format` | query | string | No | `markdown` (default), `text` (plain text, no markup) or `html` (cleaned HTML of the content). | | `main_content` | query | boolean | No | `true` (default) keeps the article or main content only; `false` converts the whole page, menus and footer included. | | `render` | query | string | No | `auto` (default) starts a browser only when the raw HTML holds too little text; `true` always; `false` never. A rendered page costs 2 credits. | | `include_links` | query | boolean | No | `true` adds a `links` list of `{url, text}`. Default `false`. | | `include_images` | query | boolean | No | `true` keeps images as `![alt](src)` in the Markdown. Default `false`. | | `country` | query | string | No | Two-letter country code (ISO 3166-1 alpha-2). The page is read from an address in that country, for 10 extra credits (residential connection). | | `max_chars` | query | integer | No | Cuts the output at this many characters, from 1,000 to 500,000. Useful to fit a model's context. | | `timeout` | query | integer | No | Seconds to wait for the site, from 5 to 60. Default 30. | | `api_key` | query | string | No | Your key, if you do not send the `Authorization` header. | **Response** `200` ```json { "url": "https://example.com/blog/first-post", "final_url": "https://example.com/blog/first-post", "status_code": 200, "fetched_at": "2026-10-22T09:41:12Z", "rendered": false, "credits_used": 1, "title": "Our first post", "description": "Why we started this blog and what comes next.", "language": "en", "author": "Jane Doe", "published_at": "2026-10-01", "site_name": "Example Blog", "markdown": "# Our first post\n\nWe started this blog to share what we learn...\n\n## What comes next\n\n- A post every month\n- Notes from our team", "word_count": 612 } ``` *curl* ```bash curl -sG "https://webreaderapi.com/api/v1/read" \ -H "Authorization: Bearer $WEBREADER_API_KEY" \ --data-urlencode "url=https://example.com/blog/first-post" \ -d format=markdown -d max_chars=20000 ``` *Python* ```python import os import requests resp = requests.get( "https://webreaderapi.com/api/v1/read", headers={"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}, params={"url": "https://example.com/blog/first-post", "max_chars": 20000}, timeout=70, ) resp.raise_for_status() print(resp.json()["markdown"][:300]) ``` *Node.js* ```javascript const params = new URLSearchParams({ url: 'https://example.com/blog/first-post', max_chars: '20000' }); const res = await fetch(`https://webreaderapi.com/api/v1/read?${params}`, { headers: { Authorization: `Bearer ${process.env.WEBREADER_API_KEY}` }, }); const page = await res.json(); console.log(page.markdown.slice(0, 300)); ``` ## Read one page (POST) `POST /api/v1/read` Same endpoint, same parameters, sent as a JSON body. Prefer POST when the URL is very long or when your HTTP client builds JSON more easily than query strings. The response is identical to the GET form. Authentication: API key in the `Authorization: Bearer API_KEY` header. **Request body** | Field | Type | Required | Description | | --- | --- | --- | --- | | `url` | string | Yes | The page to read. | | `format`, `main_content`, `render`, `include_links`, `include_images`, `country`, `max_chars`, `timeout` | mixed | No | As in the GET table above, with JSON types: booleans as `true`/`false`, numbers unquoted. | *Example: a JavaScript-heavy page, read from Germany, with its links* ```json { "url": "https://example.com/app/pricing", "render": "true", "country": "de", "include_links": true, "max_chars": 30000 } ``` **Response** `200` ```json { "url": "https://example.com/app/pricing", "final_url": "https://example.com/de/app/pricing", "status_code": 200, "fetched_at": "2026-10-22T09:44:03Z", "rendered": true, "credits_used": 3, "title": "Pricing", "language": "de", "markdown": "# Pricing\n\n| Plan | Price |\n|---|---|\n| Basic | ... |", "word_count": 214, "links": [ { "url": "https://example.com/de/contact", "text": "Contact" } ] } ``` *curl* ```bash curl -s "https://webreaderapi.com/api/v1/read" \ -H "Authorization: Bearer $WEBREADER_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com/app/pricing", "render": "true", "country": "de", "include_links": true}' ``` *Python* ```python import os import requests resp = requests.post( "https://webreaderapi.com/api/v1/read", headers={"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}, json={"url": "https://example.com/app/pricing", "render": "true", "country": "de", "include_links": True}, timeout=70, ) print(resp.json()["credits_used"], "credits") ``` *Node.js* ```javascript const res = await fetch('https://webreaderapi.com/api/v1/read', { method: 'POST', headers: { Authorization: `Bearer ${process.env.WEBREADER_API_KEY}`, 'Content-Type': 'application/json', }, body: JSON.stringify({ url: 'https://example.com/app/pricing', render: 'true', country: 'de', include_links: true }), }); console.log((await res.json()).credits_used); ``` ## Which output format should I choose? Choose `markdown` for language models and RAG: headings, lists, tables and links survive, and the text stays compact. Choose `text` when you only count words, classify or search, and markup would be noise. Choose `html` when your own pipeline already parses HTML and you only want the clutter gone. The metadata fields are the same in all three formats. Only the name of the content field changes: `markdown`, `text` or `html`. ## When should I set main_content to false? Set it to `false` when the parts around the article are what you need. Examples are a site's navigation to map its sections, a footer with legal or contact details, or a product page whose specifications sit in a sidebar. With the default `true`, the extractor keeps the main content block and drops navigation, cookie banners, ads, share buttons and footers. That is what most agents and RAG pipelines want, and it often cuts a page to a fraction of its HTML size. ## How does render=auto decide to use a browser? It first fetches the raw HTML, which is fast and costs 1 credit. If that HTML holds too little readable text, typically an app that builds the page with JavaScript, it loads the page again in a headless browser and the call costs 2 credits. The response tells you which path ran: `rendered` is `true` or `false`. Set `render=false` when you know the site serves full HTML and want a fixed cost. Set `render=true` for single-page apps, to skip the first attempt. ## What do country and max_chars change? `country` reads the page from an address in the country you name, with the same User-Agent. Use it for pages whose prices, language or availability change by country. It adds 10 credits, because the page is then read through a residential connection, and it cannot be used with the countries the service does not operate in. `max_chars` cuts the output after that many characters. It does not change the price, but it protects an agent's context window: 20,000 characters is a sensible default for a single page passed to a model. ## Can /read handle PDF files? Yes. When the URL returns a PDF, the text comes back as Markdown in the same `markdown` field. PDF files up to 20 MB and 100 pages are converted to Markdown, at 1 credit per started block of 10 pages. Scanned PDFs without a text layer return little or no text, because there is no character recognition step. ## Read endpoint: common questions ### Does /read follow redirects? Yes, up to 5. The response gives the address you asked for in `url` and the address actually read in `final_url`. A redirect to a private or local address is refused. ### Can I read a page that needs a login? No. If a site refuses our request, we return its answer: we never solve CAPTCHAs, log in to accounts, or try to get around a block or a paywall. Pages behind a login or a paywall return the site's refusal, and the call is free. ### Is the robots.txt check done on every call? Yes, against a copy of the site's robots.txt that is at most 24 hours old. A disallowed URL returns `403 blocked_by_robots` and uses no credit. ### How many pages can I read at the same time? Every account has a limit on how many pages it reads at once; your dashboard shows it, and support can raise it. For many URLs, a [batch job](https://webreaderapi.com/docs/crawl/) is simpler than running many parallel calls. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Read API response format, field by field > Read API response format: every field returned for a page, from markdown and word_count to final_url and rendered, plus job results. No KYC, pay in crypto. - URL: https://webreaderapi.com/docs/response/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Every `/read` call returns one JSON object: the page's content in the format you asked for, its metadata, and three fields that tell you what happened and what it cost. Job results reuse the same object. **Quick answer — What does the WebReaderAPI read response contain?** A **WebReaderAPI** `/read` response holds the page content in `markdown` (or `text` or `html`), its metadata (`title`, `description`, `author`, `published_at`, `language`, `site_name`, `word_count`), and the facts of the fetch: `final_url`, `status_code`, `rendered` and `credits_used`. A plain page uses 1 credit, from $0.07/1K pages as of October 2026. No KYC; pay in crypto. ## Which fields describe the fetch? These fields are always present in a successful response. | Field | Type | Meaning | | --- | --- | --- | | `url` | string | The address you sent | | `final_url` | string | The address actually read, after redirects (5 at most) | | `status_code` | integer | The HTTP status the site returned for the final address, normally 200 | | `fetched_at` | string | Time of the fetch, ISO 8601 in UTC | | `rendered` | boolean | `true` when a headless browser built the page | | `credits_used` | integer | Credits billed for this page: 1, 2 with a browser, plus 10 with `country`; PDFs 1 per started 10 pages | ## Which fields describe the page? Metadata comes from the page itself: its title tag, meta tags, structured data and visible byline. A field the page does not provide is absent, so test for it before use. | Field | Type | Meaning | | --- | --- | --- | | `title` | string | The page or article title | | `description` | string | The summary the page gives of itself | | `language` | string | Language code of the content, such as `en` or `de` | | `author` | string | The author named on the page | | `published_at` | string | Publication date stated by the page | | `site_name` | string | The site's own name | | `word_count` | integer | Words in the returned content, after extraction and before `max_chars` | ## Where is the content itself? One content field is returned, named after the `format` you chose. | Field | When | Content | | --- | --- | --- | | `markdown` | `format=markdown` (default) | Headings, paragraphs, lists, tables and links in Markdown; images too with `include_images=true` | | `text` | `format=text` | Plain text, paragraphs separated by blank lines | | `html` | `format=html` | Cleaned HTML of the main content, without scripts, styles or ads | | `links` | `include_links=true` | List of `{url, text}` for the links found on the page, absolute URLs | *A complete response (values are illustrative)* ```json { "url": "https://example.com/guides/compost", "final_url": "https://example.com/guides/compost-at-home", "status_code": 200, "fetched_at": "2026-10-22T10:02:44Z", "rendered": false, "credits_used": 1, "title": "Compost at home: a beginner's guide", "description": "What to put in a compost bin, and what to leave out.", "language": "en", "author": "Sam Rivera", "published_at": "2026-09-14", "site_name": "Example Garden", "markdown": "# Compost at home: a beginner's guide\n\nA compost bin turns kitchen scraps into soil...\n\n## What goes in\n\n- Fruit and vegetable peels\n- Coffee grounds\n\nSee also [Choosing a bin](https://example.com/guides/bins).", "word_count": 1184, "links": [ { "url": "https://example.com/guides/bins", "text": "Choosing a bin" } ] } ``` ## How do I tell a rendered page from a plain one? Look at `rendered`. With `render=auto`, the API first reads the raw HTML; it starts a browser only when that HTML holds too little text, and then sets `rendered` to `true` and `credits_used` to 2. If a site you read often always comes back rendered, send `render=true` from the start: the cost is the same and you skip a wasted first step. If a page comes back unrendered but thin, compare `word_count` with what you see in a browser. A low count usually means the content needs JavaScript; retry once with `render=true`. ## Why can final_url differ from url? Because sites redirect. A short link, an old address, a missing trailing slash or a language switch all send you elsewhere. The API follows up to 5 redirects and reports the address it finally read in `final_url`. Store `final_url` as the canonical source of a document in a RAG index. Two different `url` values that end at the same `final_url` are the same page, so you can drop the duplicate before embedding it. ## What does a job result look like? `GET https://webreaderapi.com/api/v1/jobs/{job_id}` returns `job_id`, `status` (`queued`, `running` or `done`), `results`, `next_cursor` and the job's running `credits_used`. Each item of `results` is one of two shapes: - a full `/read` response, exactly as described above, for a page that was read; - `{url, error}` for a page that was not, where `error` has the same `code` and `message` as an [API error](https://webreaderapi.com/docs/errors/). `next_cursor` is present while more results are ready; send it back as `cursor` to get them. The [batch and crawl page](https://webreaderapi.com/docs/crawl/) has a complete polling loop. *A job result page (values are illustrative)* ```json { "job_id": "job_5c1e8a90d2", "status": "done", "results": [ { "url": "https://example.com/blog/post-1", "final_url": "https://example.com/blog/post-1", "status_code": 200, "rendered": false, "credits_used": 1, "title": "Post one", "markdown": "# Post one\n\n...", "word_count": 802 }, { "url": "https://example.org/members/area", "error": { "code": "target_refused", "message": "The site refused the request.", "target_status": 403 } } ], "credits_used": 1 } ``` ## What does an error response look like? An error comes back with an HTTP status of 400 or more and a JSON body `{"error": {"code", "message"}}`. When the site itself refused the request, `target_status` gives the status the site sent. No error is billed. The [errors page](https://webreaderapi.com/docs/errors/) lists every code with its cause and whether a retry can help. ## Response format: common questions ### Is the Markdown cut when I set max_chars? Yes. The content field stops at that many characters. `word_count` still describes the whole extracted content, so you can see how much was left out. ### Why is author or published_at missing? Because the page does not state it in a way that can be read. Many pages give no author or date. Absent fields are omitted rather than returned empty. ### Are the links in the Markdown absolute? Links in the `links` list are absolute. Write your code to accept both forms in the Markdown body, and resolve relative ones against `final_url`. ### Can I get the raw HTML of the whole page? The closest is `format=html` with `main_content=false`, which returns the whole page cleaned of scripts and styles. The original byte-for-byte HTML is not part of v1. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # WebReaderAPI FAQ > WebReaderAPI FAQ: 30 answers on the web reader API: price per page, JavaScript and PDFs, crawls, robots.txt, blocked sites, KYC, crypto payments and AI agents. - URL: https://webreaderapi.com/faq/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Answers to what developers, AI builders and site owners ask most about turning web pages into Markdown with WebReaderAPI. Each answer starts with the answer. **Quick answer — What do people ask most about WebReaderAPI?** Mostly the price per page, how JavaScript pages and PDFs are counted, and what happens when a site refuses robots. In short: **WebReaderAPI** charges one credit per page read, two with a browser, from $0.07/1K pages as of October 2026; refused or failed pages cost nothing; robots.txt is always respected. No KYC; pay in crypto. ## Price and credits ### How much does WebReaderAPI cost? Prepaid packs cost $3.75 for 10,000 pages, $21.75 for 100,000 pages and $74.25 for 1,000,000 pages, as of October 2026. The rate falls to $0.07/1K pages on the 5,000,000 pages pack. The [pricing page](https://webreaderapi.com/pricing/) lists every pack. ### What does one credit buy? One page read without a browser. A page that needs JavaScript rendering uses two credits, and reading from a chosen country adds one. The response tells you what was used in `credits_used`, so you can track cost per page in your own logs. ### Do credits expire? No. Prepaid credits never expire: unused pages stay on your account until you use them. A pack bought for a one-off crawl keeps its unused pages for the next project. ### Am I charged when a page cannot be read? No. Errors use no credit, whether the cause is our side, the network or the website. That includes pages refused by robots.txt, by an `ai-input=no` signal, or by the site's own answer. ### Is there a free tier? No. There is no free tier: the smallest pack is the way to test, paid from the minimum top-up, and the amount added only to reach that minimum can be sent back on request within 30 days, in USDT or USDC, while unspent. ### What is the smallest amount I can spend? The minimum top-up is $25. The smallest pack, 10,000 pages, costs $3.75; the rest of the top-up stays on your balance for later packs or can be asked back within 30 days if unspent. ## What the API reads and returns ### Can WebReaderAPI read pages built with JavaScript? Yes. JavaScript rendering in a headless browser when the page needs it (render=auto), or always, or never. With `render=auto`, a browser is used only when the raw HTML holds too little text, so simple pages stay at one credit. ### Can it convert PDF files? Yes. PDF files up to 20 MB and 100 pages are converted to Markdown, at 1 credit per started block of 10 pages. Send the PDF's URL to the same `/read` endpoint; the text comes back as Markdown. ### Can I read a whole website? Yes, with a crawl job. Crawl one website per job, up to 10,000 pages, following its sitemap, its robots.txt and its Crawl-delay. You can restrict it to paths such as `/docs/*`; each page read uses credits like a single request. ### How many URLs can I send at once? Up to 100 URLs per batch job. A batch runs as a job: you send the list, receive a `job_id`, and collect every result when the job is done. ### How do I keep a page within my model's context window? Set `max_chars`. The Markdown is cut at that length, from 1,000 to 500,000 characters. Combined with `main_content=true`, this keeps long pages within a predictable token budget. ### Does the Markdown keep tables, code blocks and links? Yes. Headings, lists, tables, code blocks and links keep their Markdown form. Links are inline; add `include_links=true` to also get them as a separate list, and `include_images=true` to keep images. ### What does "main content" mean? The part of the page a reader came for: the article, the documentation or the product description. With `main_content=true`, the default, menus, cookie banners, ads, sidebars and footers are removed. Set it to `false` to convert the whole page. ### Can I read a page as it appears in another country? Yes, with the `country` parameter and a two-letter country code. Choosing the country a page is read from adds 10 credits, because the page is then read through a residential connection in that country. Some sites change prices, language or content by country. ## robots.txt and refused pages ### Does WebReaderAPI respect robots.txt? Always. We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged. The [bot page](https://webreaderapi.com/bot/) shows the user agent and how site owners can allow or refuse it. ### What happens with sites protected by Cloudflare or another bot filter? If the site lets our crawler through, the page is read normally. If it refuses or answers with a challenge page, we return that answer as `target_refused` and charge nothing. We never try to get past a challenge. ### Can WebReaderAPI read pages behind a login or a paywall? No, never, even with credentials you provide. Content that requires an account or a subscription stays out of scope, and asking for it breaks the [Acceptable Use Policy](https://webreaderapi.com/acceptable-use/). ### What is the ai-input=no signal? A line some sites add to robots.txt, from the Content Signals Policy, to refuse the use of their pages as input for AI answers. If a site declares Content-Signal: ai-input=no in its robots.txt, we do not read its pages for you; the API returns `blocked_by_site_policy`, free of charge. ### Is reading web pages for AI legal? It depends on the content, the country and what you do with the text. Respecting robots.txt and staying away from personal data and paywalls lowers the risk. Our [overview of the rules](https://webreaderapi.com/guides/is-reading-web-pages-for-ai-legal/) summarises the main points; it is not legal advice. ## Account, payment and privacy ### Do I need to verify my identity? No. WebReaderAPI asks for no ID, no selfie, no company documents and no card. Account-number login; email is optional. The [no-KYC page](https://webreaderapi.com/no-kyc-web-reader-api/) explains what still applies. ### Which cryptocurrencies can I pay with? BTC, USDT, USDC, ETH, SOL, LTC. USDT works on TRC-20 or ERC-20, USDC on ERC-20. Send the exact amount shown, click "I've sent the payment", and the top-up is credited once it is confirmed. ### Can I get a refund? Yes, to your balance. A pack used at less than 20% goes back within 72 hours of purchase, and a pack that never worked is credited in full. The [refund policy](https://webreaderapi.com/refunds/) has the exact rule and three examples. ### How do API keys work? Create several API keys, for example one per agent or project, and revoke any of them from your dashboard. Send your API key in the Authorization header as a Bearer token, or in the api_key query parameter. One key per application or agent makes it easy to revoke one without stopping the others. ### What do you log about my requests? The host name of each page, never the full URL or its content, plus the time, status and credits, for 30 days. We keep no copy of the pages you read. The [privacy policy](https://webreaderapi.com/privacy/) has the details. ## Developers and AI agents ### Is there code for Python and Node.js? Yes. The [Python](https://webreaderapi.com/docs/python/) and [Node.js](https://webreaderapi.com/docs/node/) guides show single reads, batches, crawls and retries, and the [quick start](https://webreaderapi.com/docs/) has a curl example. No SDK package is needed: the API is plain HTTPS with JSON. ### Can Claude, Cursor or another MCP client use WebReaderAPI? Yes, through a small MCP server you run yourself, documented on the [MCP page](https://webreaderapi.com/docs/mcp/). It exposes a tool that takes a URL and returns the page as Markdown. ### Does it work with LangChain and LlamaIndex? Yes. The [LangChain and LlamaIndex page](https://webreaderapi.com/docs/langchain/) shows a document loader that turns URLs into documents with their metadata, ready for chunking and embedding. ### How many pages can I read at the same time? Every account has a limit on how many pages it reads at once; your dashboard shows it, and support can raise it. Crawls also follow each site's `Crawl-delay`, so a large crawl of one domain may take longer than its page count suggests. ### Who runs WebReaderAPI? SerpGoat LLC. See the [about page](https://webreaderapi.com/about/) for how the service works and how to reach a person. Email support answers within 12 hours, every day of the week, including help with crawls and integrations. ### How fast is a read? It depends mostly on the website and on whether a browser is needed. We do not publish a speed figure until we have measured one over a public sample; when we do, it will appear with its method and date on the [status page](https://webreaderapi.com/status/). --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Web reader API glossary > Glossary of 36 terms for reading web pages with an API: Markdown, main content, JavaScript rendering, robots.txt, Content Signals, RAG, MCP, credits and more. - URL: https://webreaderapi.com/glossary/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) The words you meet when you turn web pages into text for language models, from the HTML side to the AI side, with links to where each one matters in the docs. **Quick answer — What do the main web reader API terms mean?** A web reader API fetches a page, keeps its **main content** and returns it as **Markdown** a model can read. **WebReaderAPI** adds **JavaScript rendering** when a page needs a browser, follows **robots.txt** and **Content Signals**, and bills one **credit** per page, from $0.07/1K pages as of October 2026. The 36 terms below explain the rest. ## From the web page side **HTML** — The markup language of web pages. Raw HTML mixes the text you want with tags, scripts and styling, which is why it reads poorly for a language model. **DOM** — The Document Object Model: the tree of elements a browser builds from HTML and then changes with JavaScript. A rendered page is read from the DOM, not from the raw HTML. **JavaScript rendering** — Running a page's scripts in a browser so that content loaded after the first response appears. WebReaderAPI does it when needed (`render=auto`), for two credits instead of one. **Headless browser** — A real browser engine that runs without a screen, driven by software. It is how JavaScript-heavy pages are rendered on a server. **Main content extraction** — Finding the part of a page a reader came for, such as the article or the documentation, and leaving out everything around it. In our API this is `main_content=true`, the default. **Readability** — Both a quality of text and the name of a well-known family of algorithms that score page blocks to find the main content, as reading modes in browsers do. **Boilerplate** — Text repeated on every page of a site: menus, cookie notices, sidebars, footers, share buttons. Removing it is most of the work of a web reader. **Markdown** — A plain-text format where `#` marks a heading, `-` a list item and `[text](url)` a link. Models read it well and it uses far fewer tokens than HTML. **Metadata** — Facts about a page rather than its text: title, description, author, publication date, language. WebReaderAPI returns them next to the Markdown when the page states them. **PDF extraction** — Turning the text of a PDF file into Markdown. PDF files up to 20 MB and 100 pages are converted to Markdown, at 1 credit per started block of 10 pages. **Sitemap** — An XML file where a site lists its pages. A crawl job starts from it when it exists, which finds pages faster and with fewer requests. **Paywall** — A barrier that shows content only to subscribers. WebReaderAPI never reads behind one. ## From the crawler side **Crawler** — A program that fetches web pages for a purpose. Ours fetches only the pages customers ask for, and the [bot page](https://webreaderapi.com/bot/) describes it. **User agent** — The name a client sends with each request. Ours is `WebReaderAPI/1.0 (+https://webreaderapi.com/bot/)`, so site owners always know who is asking. **robots.txt** — A file at the root of a site that tells crawlers which paths they may fetch, standardised as RFC 9309. We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged. **Crawl-delay** — A robots.txt line asking crawlers to wait a number of seconds between requests. Our crawl jobs honour it. **Content Signals** — A policy for robots.txt lines such as `Content-Signal: search=yes, ai-input=no`, which state how a site's content may be used, including by AI. **ai-input** — The Content Signals setting that covers using a page as input for an AI answer. If a site declares Content-Signal: ai-input=no in its robots.txt, we do not read its pages for you. **llms.txt** — A Markdown file some sites publish at `/llms.txt` to give language models a short guide to their content and their key pages. **Crawl** — Reading many pages of one site by following its links and sitemap. Crawl one website per job, up to 10,000 pages, following its sitemap, its robots.txt and its Crawl-delay. **Batch job** — A list of URLs sent in one request and processed in the background. Up to 100 URLs per batch job. **HTTP 403** — The status a server sends to refuse a request. When a site answers 403 to our crawler, the API returns `target_refused` and charges nothing. **HTTP 429** — The status for too many requests. From a site, it means slow down; from our API (`concurrency_limit`), it means your account has reached its limit of simultaneous reads. **Exponential backoff** — Waiting longer after each failed attempt (for example 1, 2, then 4 seconds) before retrying, so that a busy server is not flooded. ## From the AI side **Token** — The unit a language model reads and is billed by, roughly a short word or part of one. Clean Markdown uses far fewer tokens than the same page in HTML. **Context window** — The amount of text a model can take in at once. The `max_chars` parameter keeps each page within the budget you set. **RAG** — Retrieval-augmented generation: the model answers from documents fetched for the question. A web reader supplies those documents from live pages. **Chunking** — Splitting a long document into passages before indexing it. Markdown headings give natural split points. **Embedding** — A list of numbers that represents the meaning of a passage, used to find passages similar to a question. **Vector store** — A database that stores embeddings and returns the closest passages to a query, the usual home of chunked web pages in a RAG system. **Tool calling** — A model's ability to ask the application to run a function, such as reading a URL, and to use the result. See [OpenAI function calling](https://webreaderapi.com/docs/openai/). **MCP** — The Model Context Protocol, a standard way to plug tools into AI clients such as Claude Desktop or Cursor. Our [MCP page](https://webreaderapi.com/docs/mcp/) shows a fetch tool built on the API. ## From the billing side **Credit** — The unit of use of the API. 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged. **Concurrency** — How many reads your account can run at the same moment. Every account has a limit on how many pages it reads at once; your dashboard shows it, and support can raise it. **KYC** — Know your customer: identity checks some services require, such as an ID document or a selfie. WebReaderAPI asks for none. **Stablecoin** — A cryptocurrency that tracks a currency such as the US dollar. USDT and USDC are the two accepted for top-ups; the full list of coins is BTC, USDT, USDC, ETH, SOL, LTC. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # How to convert a web page to Markdown for an LLM > How to convert a web page to Markdown for an LLM in 2026: three methods, what clean Markdown keeps, how to cap tokens, and Python code that chunks it. - URL: https://webreaderapi.com/guides/convert-web-page-to-markdown/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Copy and paste works for one page. A library works if you run your own crawler. An API works when you want one request per page. Here is how to choose, what good output looks like, and code you can reuse. **Quick answer — How do I convert a web page to Markdown for an LLM?** Fetch the page, keep only its main content, then turn headings, lists, tables and links into Markdown and cap the length to your model's budget. You can do it by hand, with open-source libraries, or with one API call: **WebReaderAPI** returns the Markdown and metadata for one credit per page, from $0.07/1K pages as of October 2026. No KYC; pay in crypto. ## Three ways to get Markdown from a web page | Attribute | Value | | --- | --- | | By hand | Copy the text from the browser or its reader view; fine for a few pages | | Open-source libraries | Readability, Trafilatura, MarkItDown or Crawl4AI inside your own fetcher | | A web reader API | `GET https://webreaderapi.com/api/v1/read?url=…` returns Markdown, title, author and date | | Length control | `max_chars` on WebReaderAPI, or cut the text yourself after conversion | | JavaScript pages | Need a headless browser; WebReaderAPI renders them when needed, for 2 credits | ## Which method should I use? Pick by volume and by how much infrastructure you want to run. | Method | Good for | You handle | |---|---|---| | Copy and paste | A handful of pages, a one-off prompt | Cleaning by hand | | Library in your code | Steady volumes, full control | Fetching, browsers, robots.txt, PDFs, retries | | Web reader API | Agents and pipelines that read many sites | Only the request and the result | Libraries give you control and no per-page fee. The cost is operational: a headless browser for JavaScript pages, a robots.txt check, character-set and redirect handling, and a separate path for PDFs. An API folds all of that into one HTTP call. ## What must good Markdown keep, and what should it drop? Good Markdown keeps every element that carries meaning and drops everything that only exists for layout or marketing. - **Keep headings** as `#`, `##` and `###`: they tell the model, and your chunker, where each topic starts. - **Keep lists and tables** as Markdown lists and pipe tables, so steps and numbers stay readable. - **Keep links** as `[text](url)`: they let an assistant cite and follow sources. - **Keep code blocks** fenced, with the language when the page shows it. - **Drop menus, cookie notices, ads, related-post widgets and footers**: they repeat on every page and pollute retrieval. Check the result on a few of your real pages. Extraction tools disagree most on forums, long tables and pages with several articles. ## How much smaller is Markdown than HTML? Usually much smaller, because page furniture and markup disappear. When Cloudflare launched its Markdown for Agents feature on 12 February 2026, its example page went from 16,180 tokens as HTML to 3,150 tokens as Markdown, about 80% fewer. Your own ratio depends on the site. Long articles shrink less, in percentage, than short pages wrapped in heavy templates. Measure on your sources before you size a budget. ## How do I keep a page within my model's token budget? Cap the text before it reaches the model, then split what remains along headings. With WebReaderAPI, the `max_chars` parameter cuts the Markdown at a set length, between 1,000 and 500,000 characters. The `word_count` field tells you how long the full article was, so an agent can decide whether reading the rest is worth it. For retrieval, split on headings first and only then cut long sections. A chunk that starts with its own heading keeps its topic; a chunk cut mid-sentence from a long page does not. *Python: read a page as Markdown and split it into heading-based chunks* ```python import os import re import requests API = "https://webreaderapi.com/api/v1/read" HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]} def page_to_chunks(url, max_chars=60000, chunk_len=1500): params = {"url": url, "format": "markdown", "max_chars": max_chars} resp = requests.get(API, headers=HEADERS, params=params, timeout=90) data = resp.json() if resp.status_code != 200: print("not read:", data["error"]["code"]) return [] chunks = [] for section in re.split(r"\n(?=#{1,3} )", data["markdown"]): section = section.strip() while section: chunks.append(section[:chunk_len]) section = section[chunk_len:] return [{"text": c, "source": data["final_url"], "title": data["title"]} for c in chunks] if __name__ == "__main__": for chunk in page_to_chunks("https://example.com/guides/compost")[:3]: print(chunk["title"], "|", chunk["text"][:80]) ``` ## What usually goes wrong, and how do I spot it? Four problems account for most bad conversions: - **Empty output.** The page builds its content with JavaScript. Look for a low `word_count`; render the page in a browser, or set `render=true`. - **Cookie or consent text in the result.** The extractor kept a banner. Prefer main-content extraction and check the first lines of each result. - **Missing tables.** Some sites draw tables as images or canvas elements; no text extractor can read those. - **A refusal instead of a page.** The site disallowed robots in robots.txt or refused the request. Respect it: ask the owner for access, or use an official API or feed if the site offers one. None of these is fixed by retrying faster. Read the error, then change the method or the source. ## Web page to Markdown: common questions ### Is Markdown better than plain text for an LLM? For most pages, yes. Markdown keeps headings, lists, tables and links at a small token cost, so the model can see structure and cite sources. Plain text is slightly shorter but loses that structure. ### Can I convert a whole website, not just one page? Yes. Crawl it and convert each page. WebReaderAPI's [crawl API](https://webreaderapi.com/crawl-api/) does both in one job: Crawl one website per job, up to 10,000 pages, following its sitemap, its robots.txt and its Crawl-delay. ### Can I convert a PDF the same way? Yes, by sending the PDF's URL to the same endpoint. PDF files up to 20 MB and 100 pages are converted to Markdown, at 1 credit per started block of 10 pages. ### What about pages behind a login? Do not convert them through a third-party service unless they are your own and you have exported them yourself. WebReaderAPI reads public pages only and never logs in to accounts. ### Is there a free way to do this? Open-source libraries are free to use and run on your own machines. WebReaderAPI has no free tier; its smallest pack is 10,000 pages for $3.75, paid in crypto with no KYC. ## Sources 1. [Introducing Markdown for Agents](https://blog.cloudflare.com/markdown-for-agents/) — Cloudflare (2026-02-12) --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Is reading web pages for AI legal? The dated facts in 2026 > Is it legal to read web pages for AI agents and RAG in 2026? EU opt-out rules, the GPAI code on robots.txt, and US cases in progress. Not legal advice. - URL: https://webreaderapi.com/guides/is-reading-web-pages-for-ai-legal/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) There is no single answer, and several cases are still open. This page lists the rules and court events that matter for reading public pages with an AI tool, with dates and sources, and what we do on our side. It is not legal advice. **Quick answer — Is it legal to read web pages with an AI tool?** Reading public pages is widely practised, but the rules depend on the country, the content and how it is accessed. In the EU, machine-readable opt-outs such as robots.txt limit text and data mining. In the US, several 2025–2026 cases about AI and automated access are open, with no final judgment. **WebReaderAPI** respects robots.txt and AI opt-outs; no KYC, crypto. Not legal advice. ## Key dates | Attribute | Value | | --- | --- | | 17 April 2019 | EU DSM Directive, article 4(3): commercial text and data mining can be reserved by machine-readable means | | 10 July 2025 | EU GPAI Code of Practice: signatories commit to crawlers that read and follow robots.txt | | 5 December 2025 | The New York Times sued Perplexity over retrieval of its content (allegations) | | 31 July 2026 | Reddit v. SerpApi and Perplexity: DMCA §1201(a) claims survive a motion to dismiss (allegations) | | Reported 5 August 2026 | Ninth Circuit vacates Amazon's injunction against Perplexity's Comet agent | ## What does EU law say about reading web content? EU law allows text and data mining of lawfully accessible content, but lets rightholders reserve it. Article 4(3) of the DSM Directive (2019/790) says the commercial exception applies only where use has not been expressly reserved "in an appropriate manner, such as machine-readable means" for content published online. The EU AI Act's Code of Practice for general-purpose AI models, published on 10 July 2025, goes further for its signatories. They commit to crawlers that read and follow robots.txt, and to other appropriate machine-readable protocols. That is why honouring robots.txt and declared AI preferences has become a baseline for European use. ## Which US cases matter, and where do they stand? None has produced a final judgment on reading public pages for AI. The facts so far, all allegations unless a court decided them: - **Reddit v. SerpApi, Perplexity and others (S.D.N.Y.).** On 31 July 2026 the court let claims under DMCA §1201(a) proceed at the pleading stage, treating Google's challenge-based protections as measures that control access. No defendant has been found liable. - **The New York Times v. Perplexity.** Filed on 5 December 2025. The Times alleges retrieval of its content at query time after it had blocked Perplexity in robots.txt. - **Amazon v. Perplexity.** As reported on 5 August 2026, the Ninth Circuit vacated an injunction against Perplexity's Comet shopping agent, noting the assistant acted on its user's instructions. The panel said it set no broad rule for agentic AI. The direction is mixed, and appeals continue. Treat any summary, including this one, as dated. ## What lowers the risk when an AI tool reads pages? A few practices come up in every rule and case above: - **Read public pages only.** No logins, no members areas, no paywalls. - **Honour machine-readable opt-outs.** robots.txt and declared AI preferences, such as `ai-input=no`. - **Never defeat a protection.** No disguised user agents, no rotating addresses to escape blocks, no challenge solving. - **Avoid personal data.** Do not collect profiles, contact details or other data about private people. - **Credit your sources.** Keep the URL with each page and show it when an answer relies on it. These are good practice, not a guarantee. Ask a lawyer about your own use case. ## What does WebReaderAPI do on its side? We built the service around those practices. We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged. If a site declares Content-Signal: ai-input=no in its robots.txt, we do not read its pages for you. If a site refuses our request, we return its answer: we never solve CAPTCHAs, log in to accounts, or try to get around a block or a paywall. Our [Acceptable Use Policy](https://webreaderapi.com/acceptable-use/) forbids collecting personal data about private people and reading content behind a login or a paywall. Every request identifies itself as `WebReaderAPI/1.0 (+https://webreaderapi.com/bot/)`, and our [crawler page](https://webreaderapi.com/bot/) tells site owners how to reach us. ## Legal questions ### Does respecting robots.txt make my use legal? Not by itself. It answers the EU opt-out question and shows good faith, but copyright, contracts, privacy law and each site's terms still apply. This is not legal advice. ### Can I store the pages I read? That depends on the content, your purpose and your jurisdiction. WebReaderAPI keeps no copy of the pages it reads for you; what you store on your side is your decision and your responsibility. ### Is reading a page for one user different from crawling a site? Some vendors and one appellate panel have treated a user-directed agent differently from a crawler. The law is not settled, so WebReaderAPI applies the same robots.txt rules to both. ### Where can I read more? Our [robots.txt guide](https://webreaderapi.com/guides/robots-txt-for-ai-agents/) covers the standards, and the sources below link to the texts and reports. For your own situation, ask a lawyer in your jurisdiction. ## Sources 1. [Directive (EU) 2019/790, article 4](https://www.legislation.gov.uk/eudr/2019/790/article/4) — legislation.gov.uk (read 2026-10-22) 2. [AI Office publishes final version of the Code of Practice for general-purpose AI models](https://www.globalpolicywatch.com/2025/07/ai-office-publishes-final-version-of-the-code-of-practice-for-general-purpose-ai-models/) — Global Policy Watch (2025-07) 3. [Reddit v. SerpApi LLC](https://www.loeb.com/en/insights/publications/2026/08/reddit-v-serpapi-llc) — Loeb & Loeb (2026-08) 4. [NYT sues AI search engine Perplexity](https://the-decoder.com/nyt-sues-ai-search-engine-perplexity-for-alleged-content-misuse/) — The Decoder (read 2026-10-22) 5. [US court overturns Amazon injunction against Perplexity](https://www.retail-insight-network.com/news/us-court-overturns-amazon-injunction-perplexity/) — Retail Insight Network (2026-08-05) --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # llms.txt generator: build the file from a crawl of your own site > Generate an llms.txt file for your own website from a crawl: a Python script that reads every page as Markdown and lists it with a one-line summary. 2026 data. - URL: https://webreaderapi.com/guides/llms-txt-generator/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) An llms.txt file lists a site's key pages, with a line on each, in Markdown that models read easily. This guide gives honest adoption figures, then a script that crawls your site and writes the file for you. **Quick answer — How do I generate an llms.txt file for my website?** Crawl your own site, take each page's title and first paragraph, and write them as a Markdown list under your site's name in a file at `/llms.txt`. **WebReaderAPI** does the crawl: up to 10,000 pages per job, as Markdown with metadata, at one credit per page from $0.07/1K pages as of October 2026. The Python script below writes the file. No KYC; pay in crypto. ## llms.txt in brief | Attribute | Value | | --- | --- | | What it is | A proposed Markdown file at the root of a site that lists its key pages for language models | | Adoption, top 1,000 sites | 9.3% (93 sites) on 18 September 2026, per Rankability | | Sites with the file | From 4,088 to 36,120 between June 2025 and May 2026, per Originality.ai | | Measured use | 97% of llms.txt files got zero requests in an Ahrefs study of 137,000 domains | | Cost to build with WebReaderAPI | One credit per page crawled; 10,000 pages cost $3.75 | ## Is llms.txt worth making? It is cheap to make, and its measured effect so far is small. Adoption is rising: Rankability found the file on 9.3% of the top 1,000 sites on 18 September 2026, against 0.3% in June 2025, although its method changed between the two counts. Originality.ai counted sites with the file growing from 4,088 to 36,120 between June 2025 and May 2026. Use is another matter. An Ahrefs study of 137,000 domains, reported by PPC Land on 2 July 2026, found that 97% of llms.txt files received no requests at all, and that AI retrieval bots made 1.1% of the requests the files did get. Make one if your docs are large and you want a clean index; do not expect it to change how assistants cite you on its own. ## What goes into an llms.txt file? A title, a one-paragraph summary, then grouped lists of links with a short description each. A minimal file looks like this: - `# Example Docs` on the first line, the name of the site or project. - `> One sentence on what the site covers.` as a quoted summary. - `## Guides`, `## API reference` and other sections, one per part of the site. - `- [Page title](https://example.com/page): one line on what the page answers.` for each page. Keep descriptions factual and short. Do not write instructions to AI systems in the file; assistants increasingly treat such text as manipulation. *Python: crawl your own site and write llms.txt* ```python import os import time from collections import defaultdict from urllib.parse import urlparse import requests API = "https://webreaderapi.com/api/v1" HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]} SITE = "https://docs.example.com/" def first_paragraph(markdown): for block in markdown.split("\n\n"): block = block.strip() if block and not block.startswith(("#", "-", "|", "!", ">")): return " ".join(block.split())[:160] return "" job = requests.post(API + "/crawl", headers=HEADERS, json={"url": SITE, "limit": 300}, timeout=60).json() pages, cursor, delay = [], None, 5 while True: params = {"cursor": cursor} if cursor else {} batch = requests.get(API + "/jobs/" + job["job_id"], headers=HEADERS, params=params, timeout=60).json() pages += [p for p in batch.get("results", []) if "markdown" in p] cursor = batch.get("next_cursor") if batch["status"] == "done" and not cursor: break if not cursor: time.sleep(delay) delay = min(delay * 2, 60) sections = defaultdict(list) for p in pages: path = urlparse(p["final_url"]).path.strip("/") section = path.split("/")[0].replace("-", " ").title() if path else "Home" summary = p.get("description") or first_paragraph(p["markdown"]) sections[section].append("- [" + p["title"] + "](" + p["final_url"] + "): " + summary) lines = ["# Example Docs", "", "> Documentation for the Example product.", ""] for name in sorted(sections): lines += ["## " + name, ""] + sorted(sections[name]) + [""] with open("llms.txt", "w", encoding="utf-8") as fh: fh.write("\n".join(lines)) print(len(pages), "pages written to llms.txt") ``` ## How does the script work? It starts one crawl job, collects the results page by page, then groups pages by the first folder of their path. For each page it uses the meta description when the site provides one, and otherwise the first paragraph of the Markdown, cut to 160 characters. Change `SITE`, the title line and the summary line to match your project, and raise `limit` for a larger site, up to 10,000 pages per job. Review the output before you publish it: section names come from your URL structure and may need a human touch. ## Does my robots.txt need to allow the crawl? Yes. WebReaderAPI reads only what robots.txt allows for `WebReaderAPI` or for all robots, and it waits any `Crawl-delay` you set. If your own site blocks robots, add a group that allows our token for the time of the crawl; our [crawler page](https://webreaderapi.com/bot/) shows the exact lines. Pages that robots.txt disallows are skipped and not charged, so a strict file costs you nothing extra; it only leaves those pages out of llms.txt. ## How much does it cost? One credit per page read, two for pages that need a browser. A documentation site of 300 pages costs 300 credits, or up to 600 if it is a JavaScript application. That fits easily in 10,000 pages for $3.75, as of October 2026, and the credits left over stay on your account: they never expire. Rebuild the file when your docs change. If you keep the crawl output, you can re-read only the changed pages with a batch job instead of crawling everything again. ## llms.txt: common questions ### Where do I put the llms.txt file? At the root of your site, so it is served at `/llms.txt`, as plain text or Markdown. Some sites also publish `llms-full.txt` with the full text of the listed pages. ### Does llms.txt replace robots.txt? No. robots.txt says which paths robots should skip; llms.txt lists pages you want models to find. Keep both. Our [robots.txt guide](https://webreaderapi.com/guides/robots-txt-for-ai-agents/) explains the difference. ### Will llms.txt make ChatGPT or Claude cite my site? There is no evidence that it does on its own. The Ahrefs study reported in July 2026 found 97% of files received no requests. Clear pages with direct answers and dated facts matter more. ### Can I generate llms.txt for someone else's site? Only for your own sites or with the owner's agreement. The file speaks for the site, so it belongs to the site owner to publish. ## Sources 1. [llms.txt adoption data](https://www.rankability.com/data/llms-txt-adoption/) — Rankability (2026-09-18) 2. [llms.txt adoption rises 8.8x but 97% of files get zero AI requests](https://ppc.land/llms-txt-adoption-rises-8-8x-but-97-of-files-get-zero-ai-requests/) — PPC Land (2026-07-02) --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # robots.txt for AI agents and fetchers: what applies in 2026 > robots.txt for AI agents in 2026: crawlers vs user-triggered fetchers, what OpenAI, Anthropic, Google and Perplexity say, Content Signals, aipref and RSL. - URL: https://webreaderapi.com/guides/robots-txt-for-ai-agents/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) robots.txt was written for crawlers, then AI agents arrived that fetch a page because a person asked. Vendors now draw that line differently. This guide sets out the standard, each vendor's stated policy, the newer signals, and how our own crawler behaves. **Quick answer — Do AI agents have to follow robots.txt?** It depends on the vendor. robots.txt is a voluntary standard (RFC 9309, September 2022). Crawlers that build indexes generally follow it; for fetches a user triggers, OpenAI says its rules may not apply, Google and Perplexity say their fetchers generally ignore them, and Anthropic says its bots honour them. **WebReaderAPI** follows robots.txt on every read, with no KYC and payment in crypto. ## robots.txt and AI in October 2026 | Attribute | Value | | --- | --- | | Standard | RFC 9309, Robots Exclusion Protocol, September 2022 | | OpenAI ChatGPT-User | robots.txt rules "may not apply" to user-initiated actions | | Anthropic (incl. Claude-User) | Says its bots honour robots.txt directives, including Crawl-delay | | Google user-triggered fetchers | "Generally ignore robots.txt rules" (page updated 2026-08-19) | | Perplexity-User | "Generally ignores robots.txt rules" | | Content Signals | Cloudflare policy of 24 September 2025: `search`, `ai-input`, `ai-train` | | IETF AI preferences | Working-group drafts, not yet an RFC | | WebReaderAPI | Follows robots.txt and `ai-input=no` on every read; token `WebReaderAPI` | ## What does robots.txt actually do? It tells automated clients which paths a site would rather they skip. The file sits at the root of a host, groups rules by user-agent token, and uses `Allow` and `Disallow` lines. RFC 9309, published in September 2022, made the long-standing convention a standard. The RFC is explicit that the file does not lock anything: "These rules are not a form of access authorization." Compliance is voluntary. That is why each vendor's stated policy matters, and why sites that want hard limits also use firewalls and bot management. ## Why do crawlers and user-triggered fetchers differ? A crawler decides by itself which pages to visit; a user-triggered fetcher visits one page because a person asked. Many vendors treat the second case like a browser acting for its user, and say robots.txt does not fully apply. | Fetcher | Stated policy (read 2026-10-22) | |---|---| | OpenAI ChatGPT-User | robots.txt rules "may not apply" to user-initiated actions; sites are pointed to OAI-SearchBot for search opt-out | | Anthropic Claude-User | Anthropic says its bots respect robots.txt directives; blocking Claude-User stops retrieval for user queries | | Google user-triggered fetchers | "Generally ignore robots.txt rules" because a user requested the fetch | | Perplexity-User | "Generally ignores robots.txt rules" for the same reason | | MCP reference fetch server | Obeys robots.txt for model-initiated fetches by default, not for user-initiated ones | The split is real and documented. If you build an agent, decide which side you are on and say so in your user agent and docs. ## What are Content Signals, aipref and RSL? They are newer ways to say what AI systems may do with content, not just whether a robot may fetch it. - **Content Signals.** Cloudflare published this policy on 24 September 2025. A robots.txt can state `search`, `ai-input` (use as input to an AI answer, such as grounding) and `ai-train`. Cloudflare's managed robots.txt serves `search=yes, ai-train=no` by default. - **IETF AI preferences (aipref).** The working group's vocabulary and attachment drafts were still Internet-Drafts in October 2026, intended as Proposed Standards. - **Web Bot Auth.** A way for bots to sign their requests; the IETF working group adopted its protocol draft on 1 September 2026. - **RSL (Really Simple Licensing).** Its site says the RSL 1.0 specification is available; it lets publishers state licensing terms, including pay-per-crawl, through robots.txt, headers or RSS. ## Is llms.txt the same thing as robots.txt? No. robots.txt says what not to fetch; llms.txt is a proposed file that lists a site's key pages for language models. Adoption is growing but measured use is low. Rankability counted llms.txt on 9.3% of the top 1,000 sites on 18 September 2026. An Ahrefs study of 137,000 domains, reported by PPC Land on 2 July 2026, found that 97% of llms.txt files received no requests at all. Our [llms.txt guide](https://webreaderapi.com/guides/llms-txt-generator/) shows how to build one from a crawl of your own site, if you want one anyway. ## How does WebReaderAPI treat robots.txt? It follows it on every read, including reads a customer triggers. We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged. If a site declares Content-Signal: ai-input=no in its robots.txt, we do not read its pages for you. For crawl jobs, it also waits the `Crawl-delay` a site sets. Every request carries the same User-Agent, `WebReaderAPI/1.0 (+https://webreaderapi.com/bot/)`, so site owners can see us in their logs and write rules for us. Our [crawler page](https://webreaderapi.com/bot/) gives the details and a contact for problems. *robots.txt examples for WebReaderAPI* ```text # Block WebReaderAPI from the whole site User-agent: WebReaderAPI Disallow: / # Or: allow it, but keep it out of one folder and slow it down User-agent: WebReaderAPI Disallow: /account/ Crawl-delay: 5 # Or: allow reading for search, but not as AI input (Content Signals) User-agent: * Content-Signal: search=yes, ai-input=no Allow: / ``` ## robots.txt for AI: common questions ### Does blocking GPTBot stop ChatGPT from reading my page for a user? Not necessarily. OpenAI documents separate tokens: GPTBot for training, OAI-SearchBot for search and ChatGPT-User for user actions, and says robots.txt rules may not apply to user-initiated actions. Check each vendor's current documentation. ### Is robots.txt legally binding? Not by itself; RFC 9309 says it is not access authorization. In the EU, machine-readable opt-outs matter for text and data mining under the DSM Directive. See our [legal overview](https://webreaderapi.com/guides/is-reading-web-pages-for-ai-legal/); it is not legal advice. ### How do I block WebReaderAPI? Add a `User-agent: WebReaderAPI` group with `Disallow: /` to your robots.txt. Our reads check it before every page; disallowed pages are never read. ### Does WebReaderAPI read pages that say ai-input=no? No. If a site declares Content-Signal: ai-input=no in its robots.txt, we do not read its pages for you. The API returns `403 blocked_by_site_policy` to the customer, without charge. ## Sources 1. [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) — RFC Editor (September 2022) 2. [Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) — OpenAI (read 2026-10-22) 3. [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) — Anthropic (2026-04-07) 4. [Google user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers) — Google (updated 2026-08-19) 5. [Perplexity crawlers](https://docs.perplexity.ai/guides/bots) — Perplexity (read 2026-10-22) 6. [MCP fetch server](https://github.com/modelcontextprotocol/servers/tree/main/src/fetch) — Model Context Protocol (read 2026-10-22) 7. [Content Signals Policy](https://blog.cloudflare.com/content-signals-policy/) — Cloudflare (2025-09-24) 8. [AI Preferences working group documents](https://datatracker.ietf.org/wg/aipref/documents/) — IETF (read 2026-10-22) 9. [Web Bot Auth working group documents](https://datatracker.ietf.org/wg/webbotauth/documents/) — IETF (read 2026-10-22) 10. [Really Simple Licensing](https://rslstandard.org/) — rslstandard.org (read 2026-10-22) 11. [llms.txt adoption data](https://www.rankability.com/data/llms-txt-adoption/) — Rankability (2026-09-18) 12. [llms.txt adoption rises 8.8x but 97% of files get zero AI requests](https://ppc.land/llms-txt-adoption-rises-8-8x-but-97-of-files-get-zero-ai-requests/) — PPC Land (2026-07-02) --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Why AI agents get 403 errors when fetching web pages > Why web fetch fails with 403 in Claude Code, OpenClaw and other AI agents in 2026: Cloudflare defaults, robots.txt, rate limits, JavaScript, and what to do. - URL: https://webreaderapi.com/guides/web-fetch-403-errors/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) A 403 is a site saying no. Since 2025 more sites say it to AI agents by default, and since September 2026 Cloudflare blocks agent traffic on many new ad-supported sites. Here is how to tell the causes apart and what you can legitimately do. **Quick answer — Why does my AI agent get a 403 when it fetches a page?** Usually because the site, or its CDN, refuses automated or AI traffic. Cloudflare has blocked AI crawlers by default for new domains since 1 July 2025, and from 15 September 2026 blocks "Agent" traffic on ad-supported pages of new domains. Other causes are rate limits and login walls. The fix is permission, not disguise. **WebReaderAPI** returns the site's answer, uncharged; no KYC, crypto. ## Common causes of a failed web fetch | Attribute | Value | | --- | --- | | 403 from a CDN | Bot management blocks AI crawlers or agents by default on many sites | | 403 or 401 from the site | Login wall, members area, region rules or a hard block on known agents | | 429 | Too many requests in a short time; the site is rate limiting you | | 402 | The site asks for payment, for example through Cloudflare pay per crawl | | 200 but empty text | The page builds its content with JavaScript; not an error, but nothing to read | | Refused before fetching | Your own tool honoured robots.txt and skipped the page | ## What changed on the web in 2025 and 2026? Large infrastructure providers turned AI blocking into a default. Three dated changes explain most new 403s: - **1 July 2025:** Cloudflare said it now blocks AI crawlers by default, asking every new domain at sign-up whether to allow them. The same day it launched pay per crawl in private beta, which answers with HTTP 402 and a price. - **1 July 2026:** Cloudflare introduced three categories of AI traffic, Search, Agent and Training. "Agent" explicitly includes chat fetchers such as ChatGPT-User. - **15 September 2026:** for new domains, Cloudflare's presets block Training and Agent traffic on pages that display ads; Search stays allowed. An agent that fetches a page for a user falls in the Agent category. On ad-supported sites onboarded after September 2026, expect refusals. ## How do I tell a block from a bug? Read the status code and the body before changing anything. | What you see | Likely meaning | Next step | |---|---|---| | 403 with a challenge or "access denied" page | The site or its CDN refuses automated traffic | Respect it; look for an API or ask for access | | 402 with price headers | The site sells access per request | Decide whether to pay through its channel | | 429 | You sent too many requests | Slow down and retry later with backoff | | 401, or a login page | The content needs an account | Out of scope for a public reader | | 200 with almost no text | Content built by JavaScript | Render the page in a browser | | Your tool says robots.txt disallows | The site asked robots to skip it | Respect it | Only the 429 and the empty JavaScript page are technical problems you can solve on your side. The others are decisions made by the site owner. ## What can an agent builder legitimately do? Work with the site, not around it. These steps fix many failures and keep your agent on the right side of site owners: 1. **Use the site's own channel.** Many sites offer an API, an RSS or Atom feed, or a sitemap with the content you need. 2. **Ask for access.** Site owners can allow a named bot in robots.txt or in their CDN settings. A clear User-Agent and a contact page make that easy. 3. **Render pages that are allowed but empty.** For JavaScript-only pages, use a reader that runs a headless browser. 4. **Slow down and cache.** Read the same page once, store it, and respect `Crawl-delay` and 429 responses. 5. **Pay where the site sells access.** A 402 is an offer; take it through the site's channel or leave the page out. Each of these leaves the site owner in control, which is also what keeps your agent useful over time. ## What should I never do to get past a refusal? Never disguise the agent or defeat a protection. Faking a browser's User-Agent, rotating IP addresses to escape a block, or solving challenge pages turns a polite client into a tool built to defeat access controls. In 2026, two US courts treated such techniques as circumvention in lawsuits over automated access to search results. The cases are still open and nothing is decided, but the risk is not theoretical. It also breaks trust with the sites your users depend on. A refusal is information: record it, tell the user the page was not available, and move on to another source. ## What do built-in fetch tools do when a site refuses? Most return the error to the model, which then tells the user it could not read the page. OpenClaw's documentation, read on 22 October 2026, says its built-in web_fetch extracts content with Readability, does not run JavaScript and caps output at 20,000 characters; it can fall back to a fetch provider, and plugins can register new ones. The MCP reference fetch server obeys robots.txt for model-initiated fetches by default. When these tools fail on pages that allow robots but need JavaScript, a reader that renders pages helps. When they fail because the site refused, no tool should change the answer. ## What does WebReaderAPI return when a site refuses? It returns the refusal clearly, and you are not charged. A page disallowed by robots.txt gives `403 blocked_by_robots`. A site declaring `ai-input=no` gives `403 blocked_by_site_policy`. A site that refuses the request in any other way gives `502 target_refused`, with the site's own status code in `target_status`. If a site refuses our request, we return its answer: we never solve CAPTCHAs, log in to accounts, or try to get around a block or a paywall. Your agent can show the user exactly why a page was skipped. The [errors page](https://webreaderapi.com/docs/errors/) lists every code with the retry rule for each. ## Web fetch errors: common questions ### Why does a page open in my browser but fail in my agent? Because the site treats people and automated clients differently. Bot management can allow browsers and refuse agents, and some sites block AI traffic by default. The page is public to readers, not necessarily to robots. ### Will retrying fix a 403? Rarely. A 403 is a decision, not a glitch. Retry 429, timeouts and network errors with exponential backoff; treat 403 as final unless the site owner changes it. ### Can WebReaderAPI read pages my agent cannot? Only pages that allow robots but need a browser to show their text, because WebReaderAPI renders JavaScript when needed. It cannot read pages a site refuses, and does not try to. ### How can my bot get allowed on a site? Use a stable, descriptive User-Agent with a contact URL, follow robots.txt, keep request rates low, and ask the owner. Signed-agent schemes such as Web Bot Auth let sites verify who you are. ## Sources 1. [Press release: AI crawlers blocked by default](https://www.cloudflare.com/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/) — Cloudflare (2025-07-01) 2. [Introducing pay per crawl](https://blog.cloudflare.com/introducing-pay-per-crawl/) — Cloudflare (2025-07-01) 3. [Blog: Search, Agent and Training traffic options](https://blog.cloudflare.com/content-independence-day-ai-options/) — Cloudflare (2026-07-01) 4. [Blog: new defaults for AI crawlers and agents](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) — Cloudflare (2026-09-15) 5. [Web fetch tool](https://docs.openclaw.ai/tools/web-fetch.md) — OpenClaw (read 2026-10-22) 6. [MCP fetch server](https://github.com/modelcontextprotocol/servers/tree/main/src/fetch) — Model Context Protocol (read 2026-10-22) 7. [Reddit v. SerpApi LLC](https://www.loeb.com/en/insights/publications/2026/08/reddit-v-serpapi-llc) — Loeb & Loeb (2026-08) 8. [Google lawsuit against SerpApi dismissed](https://seroundtable.com/google-lawsuit-serpapi-dismissed-41731.html) — Search Engine Roundtable (2026-07-22) --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # What is a web reader API? A URL in, readable Markdown out > What a web reader API does: a URL goes in, the page's main content comes out as clean Markdown for an LLM. How it differs from scraping and search APIs. - URL: https://webreaderapi.com/guides/what-is-a-web-reader-api/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) A web reader API turns a page meant for human eyes into text a model can use. This guide explains what it does step by step, where it sits next to scraping and search APIs, and how the market prices it. **Quick answer — What is a web reader API?** A web reader API is a service that takes a URL, loads the page and returns its main content as clean text, usually Markdown, with metadata such as title, author and date. AI agents and RAG pipelines use it instead of raw HTML. **WebReaderAPI** is one: one credit per page, from $0.07/1K pages as of October 2026, with robots.txt respected, no KYC and payment in crypto. ## A web reader API in one table | Attribute | Value | | --- | --- | | Input | The URL of a public web page or PDF | | Output | Markdown of the main content, or plain text, with title, author, date and language | | Removed | Navigation, cookie banners, ads, sidebars, footers, scripts and styles | | JavaScript pages | Rendered in a headless browser when the raw HTML holds too little text | | Typical billing | Per page or per credit; some providers bill per token of output | | Example request | `GET https://webreaderapi.com/api/v1/read?url=https://example.com/post` | ## What does a web reader API do, step by step? It does five jobs that every page-to-text pipeline needs, in one call: 1. **Check permission.** A careful reader looks at the site's robots.txt for its own name and for all robots before it fetches anything. 2. **Fetch the page.** It downloads the HTML, follows redirects, and stops at a timeout. 3. **Render when needed.** If the HTML is an empty shell filled by JavaScript, it loads the page in a headless browser and reads the result. 4. **Extract the main content.** A readability-style step finds the article or the documentation body and drops the page furniture around it. 5. **Convert and describe.** It writes the content as Markdown and adds metadata: title, author, publication date, language and word count. The caller gets one JSON object with the Markdown inside. A model can read it at once, or a pipeline can split it into chunks for a vector store. ## Why do models need Markdown rather than HTML? Because most of an HTML page is not content. Menus, script tags, tracking code and layout markup use tokens that a model must read and that you must pay for. Markdown keeps the parts that carry meaning: headings, lists, tables, links and emphasis. Cloudflare gave a concrete example when it launched its "Markdown for Agents" feature on 12 February 2026: one page dropped from 16,180 tokens as HTML to 3,150 tokens as Markdown, about 80% fewer. Results vary by page, but the direction is the same on most article and documentation pages. ## How is a web reader API different from a scraping API? A scraping API returns what the server sent; a reader returns what a person would read. Scraping APIs are built to fetch pages at scale, often to pull specific fields such as prices or product names, and they usually hand back raw HTML or structured JSON that you define. A web reader is narrower and more opinionated. It decides what the main content is, removes the rest and writes clean text. That is the right tool when the next consumer is a language model, a search index or a person reading a summary. It is the wrong tool when you need every field of a product grid; for that you want the full HTML or a schema-based extraction. ## How is it different from a search API? A search API answers "which pages talk about this?"; a reader answers "what does this page say?". Search returns a list of URLs with titles and snippets for a query. A reader takes one of those URLs and returns its full text. Agents often chain the two: search to find candidate pages, then read the best three or four to answer with citations. Keeping the steps separate lets you choose how many pages to read, which saves tokens and credits. ## Can I run open-source tools instead of an API? Yes, and many teams start that way. Mozilla's Readability extracts the main content of a page in a browser context; Trafilatura does extraction in Python; Microsoft's MarkItDown converts documents, including PDFs, to Markdown; Crawl4AI combines crawling and Markdown output. The library is the easy part. Running it in production means hosting headless browsers for JavaScript pages, checking robots.txt, handling timeouts, redirects and character sets, converting PDFs, and keeping all of it patched. An API is worth it when you would rather pay per page than run that machinery; a library is worth it when volumes are steady and you have the time to operate it. ## How do web reader APIs charge? Three billing models are common in 2026. Credit-based plans charge a set number of credits per page: Firecrawl's pricing page, read on 22 October 2026, counts 1 credit per scraped page and 4 extra credits per page for its JSON format. Token-based billing charges for the text returned: Jina Reader has priced this way since 6 May 2025 and gives 10 million free tokens to each new key. Monthly plans reset their credits each cycle, while prepaid packs keep them. WebReaderAPI uses prepaid credits that never expire: 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged. 10,000 pages cost $3.75 and the rate falls to $0.07/1K pages on 5,000,000 pages, as of October 2026. ## What should I check before choosing a web reader API? Test it on twenty of your own URLs and look at six things: - **Extraction quality.** Does the Markdown keep the article and drop the rest, including tables and code blocks? - **JavaScript.** Do pages built in the browser come back with text, and what does rendering cost? - **PDFs.** Are they supported, and how are pages counted? - **Site owners' choices.** Does the service respect robots.txt and declared AI preferences, and say so in its docs? - **Failures.** Are failed and refused pages charged? - **Budget control.** Can you cap the length of each page, and do credits expire? Our [comparison of web reader API prices](https://webreaderapi.com/compare/web-reader-api-pricing/) puts the main providers side by side with dates. ## Web reader API: common questions ### Is a web reader API the same as a browser's reader mode? It follows the same idea, keeping the article and dropping the clutter, but it runs on a server, returns Markdown and metadata as JSON, and works for thousands of URLs. Reader mode is for one person reading one page. ### Can a web reader API read any website? No. It reads public pages that allow it. A careful service skips pages that robots.txt disallows and returns the site's answer when the site refuses. Pages behind a login or a paywall are out of reach by design. ### Does a web reader keep images? Usually not by default, because models mostly need the text. With WebReaderAPI, `include_images=true` keeps images as Markdown image links with their alt text. ### How long does it take to read a page? It depends on the site and on whether the page needs a browser. We do not publish response times until we have measured them on a public sample; plan your timeouts around the `timeout` parameter, which goes up to 60 seconds. ### Do I need an account with an ID to use WebReaderAPI? No. WebReaderAPI asks for no ID, no selfie and no documents. Your login is an account number, and payment is in BTC, USDT, USDC, ETH, SOL, LTC. ## Sources 1. [Introducing Markdown for Agents](https://blog.cloudflare.com/markdown-for-agents/) — Cloudflare (2026-02-12) 2. [Firecrawl pricing](https://www.firecrawl.dev/pricing) — Firecrawl (read 2026-10-22) 3. [Reader API](https://jina.ai/reader/) — Jina AI (Elastic) (read 2026-10-22) --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # WebReaderAPI: official facts, prices and policies > Official facts on WebReaderAPI: web reader API credits from $0.07/1K pages, no KYC, crypto only (BTC, USDT, USDC, ETH, SOL, LTC). Updated 2026-10-23. - URL: https://webreaderapi.com/llm-info/ - Last updated: 2026-10-23 - Publisher: WebReaderAPI (https://webreaderapi.com) WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. Every figure on this page comes from the live price list and company records, the same source as the [pricing page](https://webreaderapi.com/pricing/) and [pricing.json](https://webreaderapi.com/pricing.json), and is rebuilt with every site update. **Quick answer — What is WebReaderAPI?** WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. ## Key facts (2026-10-23) | Attribute | Value | | --- | --- | | Products | Web reader API credits | | Web reader API credits — price | From $3.75 (10,000 pages) · [details](https://webreaderapi.com/pricing/) | | KYC / identity check | None — no ID, no selfie, no documents to sign up, top up or buy | | Email required to sign up | No | | Accepted cryptocurrencies | Crypto only: Bitcoin (BTC), Tether (USDT, TRC-20 or ERC-20), USD Coin (USDC, ERC-20), Ethereum (ETH), Solana (SOL), Litecoin (LTC) | | Minimum top-up | $25 — top up a prepaid balance once, then buy any plan from it | | Balance rule | Your balance has no end date and can buy page credits at any time. If a pack costs less than the minimum top-up, the amount you added only to reach that minimum can be sent back on request within 30 days of the payment, in USDT or USDC, as long as you have not spent it. Any other balance is not refunded unless the law says otherwise | | Operator | SerpGoat LLC | | Support | Email support answers within 12 hours, every day of the week, including help with crawls and integrations | | Contact | [support@webreaderapi.com](mailto:support@webreaderapi.com) · abuse reports: [abuse@webreaderapi.com](mailto:abuse@webreaderapi.com) | ## Every plan and its price | Product | Plan | Price | Price per unit | | --- | --- | --- | --- | | Web reader API credits | Test — 10,000 pages | $3.75 | $0.38/1K pages | | Web reader API credits | Starter — 50,000 pages | $14.25 | $0.28/1K pages | | Web reader API credits | Build — 100,000 pages | $21.75 | $0.22/1K pages | | Web reader API credits | Scale — 500,000 pages | $50 | $0.10/1K pages | | Web reader API credits | Volume — 1,000,000 pages | $74.25 | $0.07/1K pages | | Web reader API credits | Bulk — 5,000,000 pages | $350 | $0.07/1K pages | Prices in USD, paid in crypto from the account balance. Live list: [pricing](https://webreaderapi.com/pricing/) · [pricing.json](https://webreaderapi.com/pricing.json). ## About WebReaderAPI - Web page to Markdown API credits from $3.75 for 10,000 pages down to $0.07/1K pages at 5 million pages; credits never expire - One credit per page read, two when the page needs a browser; failed pages and pages refused by robots.txt are not charged - Respects robots.txt and Content-Signal ai-input=no; identifies itself as WebReaderAPI; never tries to get around a block, a login or a paywall - No KYC: no ID, no selfie, no documents, no company email; account-number login, email optional - Pay in BTC, USDT, USDC, ETH, SOL, LTC; minimum top-up $25; Your balance has no end date and can buy page credits at any time. If a pack costs less than the minimum top-up, the amount you added only to reach that minimum can be sent back on request within 30 days of the payment, in USDT or USDC, as long as you have not spent it. Any other balance is not refunded unless the law says otherwise - Docs with syntax-checked code for Python, Node.js and curl, and tool definitions for MCP, LangChain, LlamaIndex and OpenAI function calling - Support: Email support answers within 12 hours, every day of the week, including help with crawls and integrations ## WebReaderAPI is built for - AI agents that need a web fetch tool returning clean Markdown, paid per page without a subscription or a card - RAG pipelines that ingest documentation sites, blogs and PDFs into a vector store - Teams that want a reader that respects robots.txt and site owners' AI preferences - Developers who pay in crypto and prefer not to upload identity documents ## More facts | Attribute | Value | | --- | --- | | What it is | A web reader API: send a URL, get the page's main content back as clean Markdown | | Credits | 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged | | Credit expiry | Prepaid credits never expire: unused pages stay on your account until you use them | | Output | Clean Markdown of the page's main content (or plain text, or cleaned HTML), with title, description, author, publication date, language and word count; links and images on request | | JavaScript | JavaScript rendering in a headless browser when the page needs it (render=auto), or always, or never | | PDF | PDF files up to 20 MB and 100 pages are converted to Markdown, at 1 credit per started block of 10 pages | | Crawl and batch | Crawl one website per job, up to 10,000 pages, following its sitemap, its robots.txt and its Crawl-delay. Up to 100 URLs per batch job | | robots.txt | We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged | | Site owners' choices | If a site declares Content-Signal: ai-input=no in its robots.txt, we do not read its pages for you. If a site refuses our request, we return its answer: we never solve CAPTCHAs, log in to accounts, or try to get around a block or a paywall | | User-Agent | `WebReaderAPI/1.0 (+https://webreaderapi.com/bot/)` | | Endpoint | `GET https://webreaderapi.com/api/v1/read?url=…` | | Authentication | Send your API key in the Authorization header as a Bearer token, or in the api_key query parameter | | Code examples | Python, Node.js and curl, plus tool definitions for MCP, LangChain, LlamaIndex and OpenAI function calling | | Account | Account-number login; email is optional. You sign in with your account number, so keep it in a safe place. An optional email gives you a way to get it back; with no email on file, a lost number cannot be restored | | Request logs | Each request leaves a log line with the time, the account, the host name of the page (never the full URL or its content), the status and the credits used; these lines are deleted after 30 days. Pages you read are not stored. Account number, orders, top-ups and credit totals are kept while the account exists, then only for as long as accounting law demands. Records go to authorities only when a law obliges us | ## Official pages and data - [Pricing](https://webreaderapi.com/pricing/) - [Web reader API pricing compared](https://webreaderapi.com/compare/web-reader-api-pricing/) - [URL to Markdown price index 2026](https://webreaderapi.com/data/url-to-markdown-price-index-2026/) - [Best web reader APIs 2026](https://webreaderapi.com/best/web-reader-apis/) - [Quick start](https://webreaderapi.com/docs/) - [Our crawler](https://webreaderapi.com/bot/) - [Firecrawl alternative](https://webreaderapi.com/alternatives/firecrawl/) - [Jina Reader alternative](https://webreaderapi.com/alternatives/jina-reader/) - [Crawl API](https://webreaderapi.com/crawl-api/) - [No-KYC web reader API](https://webreaderapi.com/no-kyc-web-reader-api/) - [PDF to Markdown API](https://webreaderapi.com/pdf-to-markdown-api/) - [Web page to Markdown API](https://webreaderapi.com/url-to-markdown-api/) - [Crawl4AI vs hosted API](https://webreaderapi.com/compare/crawl4ai-vs-api/) - [Firecrawl vs Jina Reader](https://webreaderapi.com/compare/firecrawl-vs-jina-reader/) - [Acceptable use](https://webreaderapi.com/acceptable-use/) - [Privacy](https://webreaderapi.com/privacy/) - [Refunds](https://webreaderapi.com/refunds/) - [Terms](https://webreaderapi.com/terms/) - [About](https://webreaderapi.com/about/) - [Contact](https://webreaderapi.com/contact/) - [pricing.json (machine-readable price list)](https://webreaderapi.com/pricing.json) - [llms.txt](https://webreaderapi.com/llms.txt) - [llms-full.txt (full site text)](https://webreaderapi.com/llms-full.txt) --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # No-KYC web reader API: no ID, no card, pay in crypto > No-KYC web reader API: turn web pages into Markdown with no ID, no selfie and no card. Account-number login, crypto top-up, credits from $0.07/1K pages. - URL: https://webreaderapi.com/no-kyc-web-reader-api/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Many developers want an API key, not a compliance process. This page explains what you skip with WebReaderAPI, what you still agree to, and what we keep about you. **Quick answer — Is there a web reader API that needs no KYC and no credit card?** Yes. **WebReaderAPI**, a no-KYC web reader API, asks for no ID, no selfie, no company documents and no card: you sign in with an account number and pay in BTC, USDT, USDC, ETH, SOL, LTC. Credits cost $3.75 for 10,000 pages and fall to $0.07/1K pages, as of October 2026. The acceptable use policy and robots.txt still apply to every request. ## What you need, and what you do not | Attribute | Value | | --- | --- | | Identity document or selfie | Not needed | | Company papers or a work email | Not needed | | Credit card | Not needed: crypto only | | Login | An account number; adding an email is optional | | Payment | BTC, USDT, USDC, ETH, SOL, LTC, from a $25 top-up | | Rules that still apply | Acceptable use policy, robots.txt, site owners' refusals | | What we keep | Account number, optional email, payment records, 30 days of logs with host names only | ## Why does an API need no identity check? Because nothing in the service depends on knowing who you are. You pay before you use anything, so there is no credit to recover. The crawler's behaviour does not change with the customer: it follows each site's robots.txt and stops at every refusal, whoever sent the request. An identity check would add a delay and a store of documents worth protecting, without making the service safer for the sites we read. We ask for what the service actually needs: a payment and a key. ## How does an account work without an email? You get an account number when you sign up, and that number is your login. If you add an email, it gives you a way back into the account; without one, a lost number cannot be recovered. Inside the account you create API keys, one per application or agent if you like, and revoke any of them when you want. Losing a key is not a problem; losing the account number without an email is, so store it in a password manager. ## What rules still apply without KYC? Every rule that protects websites and people applies to every account. We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged. If a site refuses our request, we return its answer: we never solve CAPTCHAs, log in to accounts, or try to get around a block or a paywall. The [Acceptable Use Policy](https://webreaderapi.com/acceptable-use/) bans, among other things, Reading pages to collect personal data about private people, reading content behind a login or a paywall, child sexual abuse material, spam, fraud, and any use that breaks the law where you or we operate. Reports are checked by a person, and breaches end in suspension. Skipping the identity check never means skipping these rules. ## What do we keep about you? Less than a typical API provider, because there is less to keep: your account number, an optional email, your payments and credits, and request logs that hold host names only, deleted after 30 days. No page content is stored. We never hold an ID document, a selfie or a card number, so none can leak. The [privacy policy](https://webreaderapi.com/privacy/) lists every item and how long it stays. ## Who chooses a no-KYC web reader? Mostly people for whom paperwork is the slowest part of trying a tool: - **Independent developers** building an AI agent who want a key in minutes. - **Small teams** without a company card, or in countries where card payments to foreign APIs fail. - **Autonomous projects and bots** funded by a crypto wallet rather than a bank account. - **Privacy-minded builders** who prefer to share as little as possible with each supplier. ## How do I get started without an ID? 1. Create an account and save the account number. 2. Top up from $25 in BTC, USDT, USDC, ETH, SOL, LTC: send the exact amount shown on the network shown, then click "I've sent the payment". The top-up is credited once it is confirmed. 3. Buy a pack, such as 10,000 pages for $3.75, and copy your API key. 4. Send your first URL to `https://webreaderapi.com/api/v1/read`, as shown in the [quick start](https://webreaderapi.com/docs/). ## What does the no-KYC web reader cost? The same prices apply to every account: 10,000 pages for $3.75, 100,000 pages for $21.75, down to $0.07/1K pages on the largest pack. Credits never expire. | Plan | Quantity | Price | Price per unit | | --- | --- | --- | --- | | Test | 10,000 pages | $3.75 | $0.38/1K pages | | Build | 100,000 pages | $21.75 | $0.22/1K pages | | Volume | 1,000,000 pages | $74.25 | $0.07/1K pages | In every plan: Main-content extraction · JavaScript rendering when needed · Failed and refused pages not charged · Credits never expire Packs are bought from a crypto balance topped up with at least $25. Prices in USD, set on 2026-10-22. ## No-KYC web reader: common questions ### Can I use WebReaderAPI without a credit card? Yes, and in fact only without one: payments are in BTC, USDT, USDC, ETH, SOL, LTC. There is no card on file and nothing renews by itself. ### Does no KYC mean anything goes? No. Every request follows robots.txt and stops when a site refuses, and the acceptable use policy applies to every account. Breaking it gets an account suspended. ### Will you ask for documents later, for example above some spend? Our sign-up and top-up process does not ask for documents. We disclose records to authorities only when the law requires it, as the privacy policy says. ### What if I lose my account number? Then the account cannot be recovered unless you added an email to it. Keep the number in a password manager, and add an email if you want a safety net. ### Can a bot or an agent pay for its own credits? Yes, as long as someone sends the top-up from a wallet. The account then works like any other: buy a pack, use a key, watch the balance with `GET https://webreaderapi.com/api/v1/credits`. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # PDF to Markdown API: send a PDF link, get its text back as Markdown > PDF to Markdown API for LLMs and RAG: send a PDF URL, get its text as Markdown with headings and tables, up to 100 pages, from $0.07/1K pages. Crypto. - URL: https://webreaderapi.com/pdf-to-markdown-api/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Reports, papers, manuals and price lists are often PDFs. Pass the link to the same endpoint you use for web pages and get the document's text as Markdown, ready to chunk and embed. **Quick answer — Is there an API that converts a PDF from a URL into Markdown?** Yes. With **WebReaderAPI**, send the PDF's URL to `https://webreaderapi.com/api/v1/read`: the API downloads the file and returns its text as Markdown, with headings, paragraphs, lists and simple tables. Files up to 20 MB and 100 pages are supported, at 1 credit per started block of 10 pages, from $0.07/1K pages as of October 2026. No KYC; pay in crypto. ## PDF to Markdown at a glance | Attribute | Value | | --- | --- | | Endpoint | `GET https://webreaderapi.com/api/v1/read?url=https://…/report.pdf` (the same as for web pages) | | Limits | PDF files up to 20 MB and 100 pages are converted to Markdown, at 1 credit per started block of 10 pages | | Output | Markdown text with headings, paragraphs, lists and simple tables; `format=text` for plain text | | Not supported | Scanned PDFs without a text layer (no OCR), password-protected files | | robots.txt | Applies to PDF links as to any other page | | Payment | BTC, USDT, USDC, ETH, SOL, LTC, from a $25 top-up; no KYC | ## How are PDF pages counted? A PDF costs 1 credit per started block of 10 pages. A 4-page brochure costs 1 credit, a 25-page report 3 credits, and a 100-page manual 10 credits. The response gives the exact figure in `credits_used`. Files larger than 20 MB or longer than 100 pages return `422 unsupported_content` and use no credit. For a long document, look for an HTML version of the same content first: it is often cheaper and better structured. ## What does the Markdown look like? The converter rebuilds the reading order from the PDF's text layer. Large or bold lines at the start of a section become headings, bulleted and numbered items become Markdown lists, and simple grid tables become Markdown tables. Running headers, footers and page numbers that repeat on every page are removed, so they do not split sentences in your chunks. Two-column layouts are read column by column. Footnotes stay as text at the point where the page puts them. *Python: read a PDF and split it into chunks by heading* ```python import os import re import requests resp = requests.get( "https://webreaderapi.com/api/v1/read", headers={"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}, params={"url": "https://example.com/annual-report.pdf"}, timeout=120, ) resp.raise_for_status() doc = resp.json() chunks = [c.strip() for c in re.split(r"\n(?=#{1,3} )", doc["markdown"]) if c.strip()] print(len(chunks), "chunks,", doc["credits_used"], "credit(s)") for c in chunks[:3]: print(c.splitlines()[0]) ``` ## Which PDFs work best, and which do not work? PDFs exported from a word processor, a website or a publishing tool work best: they carry a text layer with fonts and positions. Annual reports, research papers, product manuals, policy documents and price lists usually fall in this group. Scanned documents are images of pages and have no text layer. WebReaderAPI does not run optical character recognition, so a scan returns little or no text; the `word_count` field shows it at once. Password-protected and encrypted PDFs are refused with `422 unsupported_content`, without charge. ## How do PDFs fit into a RAG pipeline? Treat them like web pages. Read the PDF, split the Markdown along its headings, add the source URL and the page title to each chunk's metadata, then embed. Keeping the URL lets your assistant cite the document it used. When a crawl meets links to PDFs on the same domain, it reads them with the same rules and the same robots.txt checks. See the [RAG guide](https://webreaderapi.com/use-cases/rag/) for a full ingestion example. ## What does PDF conversion cost? PDFs draw from the same credits as web pages: 1 credit per started block of 10 pages. With 10,000 pages for $3.75, you can convert about 10,000 short PDFs or 1,000 documents of 100 pages. | Plan | Quantity | Price | Price per unit | | --- | --- | --- | --- | | Test | 10,000 pages | $3.75 | $0.38/1K pages | | Starter | 50,000 pages | $14.25 | $0.28/1K pages | | Build | 100,000 pages | $21.75 | $0.22/1K pages | | Scale | 500,000 pages | $50 | $0.10/1K pages | In every plan: Main-content extraction · JavaScript rendering when needed · Failed and refused pages not charged · Credits never expire **Before you pay** - Minimum top-up $25. Top up once in crypto, then buy any plan from your balance. - Your balance has no end date and can buy page credits at any time. If a pack costs less than the minimum top-up, the amount you added only to reach that minimum can be sent back on request within 30 days of the payment, in USDT or USDC, as long as you have not spent it. Any other balance is not refunded unless the law says otherwise. - Credited once your payment is checked on the blockchain. Typical network confirmation time: BTC ≈ 20 min · USDT TRC-20 ≈ 1 min · USDT ERC-20 ≈ 5 min · USDC ERC-20 ≈ 5 min · ETH ≈ 5 min · SOL ≈ 1 min · LTC ≈ 15 min ## PDF to Markdown API: common questions ### Can I upload a PDF file instead of sending a link? Not in version 1. The API reads PDFs from a public URL. To convert a private file, put it at a URL you control for the time of the request. ### Does it extract text from scanned PDFs? No. WebReaderAPI reads the text layer of a PDF and does not run OCR. A scanned document returns little or no text, which the `word_count` field shows. ### Are tables kept? Simple grid tables become Markdown tables. Tables with merged cells or complex layouts are kept as text in reading order, which models still read well. ### Is a PDF charged if it fails? No. Files that are too large, too long, encrypted or refused by the site are not charged. 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Web reader API pricing: prepaid page credits that never expire > WebReaderAPI pricing: $3.75 for 10,000 pages down to $0.07/1K pages, credits that never expire, failed pages free. No KYC, crypto. - URL: https://webreaderapi.com/pricing/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) One credit reads one page; a page that needs a browser takes two. Pages that fail or that a site refuses cost nothing. Buy a pack from your crypto balance and use it at your own pace. No KYC — no ID, no selfie, no documents. Pay in crypto. **Quick answer — How much does WebReaderAPI cost?** **WebReaderAPI** sells prepaid page credits: 10,000 pages for $3.75 ($0.38/1K pages), 100,000 pages for $21.75, 1,000,000 pages for $74.25 and $0.07/1K pages at 5 million, as of October 2026. A page read with a browser uses two credits. Failed pages and pages refused by robots.txt are free, and credits never expire. No KYC; pay in crypto. ## Which credit pack should I buy? Six packs, the same API and options in each. The price per page falls from $0.38/1K pages on the smallest pack to $0.07/1K pages on the largest. | Plan | Quantity | Price | Price per unit | | --- | --- | --- | --- | | Test | 10,000 pages | $3.75 | $0.38/1K pages | | Starter | 50,000 pages | $14.25 | $0.28/1K pages | | Build | 100,000 pages | $21.75 | $0.22/1K pages | | Scale | 500,000 pages | $50 | $0.10/1K pages | | Volume | 1,000,000 pages | $74.25 | $0.07/1K pages | | Bulk | 5,000,000 pages | $350 | $0.07/1K pages | In every plan: Main-content extraction · JavaScript rendering when needed · Failed and refused pages not charged · Credits never expire **Before you pay** - Minimum top-up $25. Top up once in crypto, then buy any plan from your balance. How payment works ## How are pages counted? Every response says what it cost in `credits_used`. The rules: - **Static page: 1 credit.** The page's HTML held its content. - **Page that needed a browser: 2 credits.** With `render=auto`, the browser only runs when the raw HTML is nearly empty; `render=false` never uses it. - **Country chosen: +10 credits.** Choosing the country a page is read from adds 10 credits, because the page is then read through a residential connection in that country. - **PDF: 1 credit per started block of 10 pages.** A 25-page report costs 3 credits. - **Errors and refusals: 0 credit.** Invalid URLs, timeouts, pages refused by robots.txt or by the site, and files we cannot read are free. - **Checking your balance: free.** `GET https://webreaderapi.com/api/v1/credits` uses no credit. ## Do unused credits expire? No. Prepaid credits never expire: unused pages stay on your account until you use them. A pack bought for one crawl keeps its remaining value for the next one, and an agent that is idle for a month loses nothing. If we ever stop selling the service, the [terms](https://webreaderapi.com/terms/) give you at least 30 days' notice to use your credits or move their value. A price increase applies only to packs bought after it is published. ## Which pack fits my workload? Typical volumes and the pack that covers them. Count two credits for pages that need a browser. | Workload | Pages | Suggested pack | | --- | --- | --- | | Prototype agent or a one-off crawl of a small site | under 5,000 | 10,000 pages ($3.75) | | AI agent used every day by a small team | 10,000 to 30,000 a month | 50,000 pages ($14.25) | | RAG index of several documentation sites, with weekly refreshes | 30,000 to 80,000 | 100,000 pages ($21.75) | | Product that reads pages for its own users | 100,000 to 400,000 a month | 500,000 pages ($50) | | Research corpus or large archive | 1 million and more | 1,000,000 pages ($74.25) or 5,000,000 pages ($350) | ## How do these prices compare with other web reader APIs? Price per 1,000 pages when you read 100,000 pages a month, from each provider's public pricing page. The full table, with 21 APIs and three volumes, is on [web reader API pricing compared](https://webreaderapi.com/compare/web-reader-api-pricing/). | Provider and plan | Plain pages | Pages that need a browser | | --- | --- | --- | | WebReaderAPI, 100,000 pages | $0.22/1K pages | 2 credits per page | | Firecrawl Standard ($99 for 100,000 credits) | $0.99 | $0.99 (every page rendered) | | Olostep Standard ($99) | $0.99 | $0.99 (every page rendered) | | Crawl4AI Cloud Builder ($99) | $0.50 | $0.99 | | Scrape.do Hobby ($29 for 250,000 credits) | $0.29 | $0.99 on Pro | | ScrapingBee Freelance ($49) | $0.49 | $0.99 on Startup | | Exa contents | $1.00 | No browser option documented | | Tavily Extract (basic, with pay as you go) | $1.40 | $2.36 (advanced) | *Prices checked on October 22, 2026.* We do not claim to be the cheapest at every volume or for every page type. Sources: each provider's pricing page, listed on the comparison page. ## How payment works 1. **Top up $25 or more** — Choose the amount and the coin (BTC, USDT, USDC, ETH, SOL, LTC). The crypto amount and its rate are locked for 30 minutes. 2. **Send it to the address shown, on the exact network** — The payment page shows our deposit address for the coin and network you picked: BTC on Bitcoin, USDT on Tron (TRC-20) or Ethereum (ERC-20), USDC on Ethereum (ERC-20), ETH on Ethereum, SOL on Solana, LTC on Litecoin. Send only that coin on that network: funds sent on another network or in another coin can’t be recovered. The network fee comes on top of the amount. 3. **Send the exact amount shown, then click “I’ve sent the payment”** — Each top-up has its own exact amount (down to the last decimals): that is how we find your payment and link it to your account. 4. **Credited once your payment is checked on the blockchain** — Your balance is credited once we have checked the transaction on the blockchain, after the network has confirmed it. Typical network confirmation times: 5. **One balance for all your page credits** — When you begin from a pack, the top-up equals its price (or the minimum, if the pack is smaller); the credits are added once the payment is credited, and the API key waits in your dashboard. The remainder stays on the balance for later packs. Your balance has no end date and can buy page credits at any time. If a pack costs less than the minimum top-up, the amount you added only to reach that minimum can be sent back on request within 30 days of the payment, in USDT or USDC, as long as you have not spent it. Any other balance is not refunded unless the law says otherwise. | Crypto | Network | Confirmations | Usual time | | --- | --- | --- | --- | | Bitcoin (BTC) | Bitcoin | 2 | ≈ 20 min | | Tether (USDT) | Tron (TRC-20) | 20 | ≈ 1 min | | Tether (USDT) | Ethereum (ERC-20) | 12 | ≈ 5 min | | USD Coin (USDC) | Ethereum (ERC-20) | 12 | ≈ 5 min | | Ethereum (ETH) | Ethereum | 12 | ≈ 5 min | | Solana (SOL) | Solana | 1 | ≈ 1 min | | Litecoin (LTC) | Litecoin | 6 | ≈ 15 min | Closed the tab? The pending top-up stays under Billing in your account: open it to see the address and the exact amount again. ## Pricing questions ### Is there a free tier? No. There is no free tier: the smallest pack is the way to test, paid from the minimum top-up, and the amount added only to reach that minimum can be sent back on request within 30 days, in USDT or USDC, while unspent. The 10,000 pages pack costs $3.75 and includes every option of the largest pack. ### Why must I top up $25 if a pack costs less? Every payment goes through your balance, and the minimum top-up is $25. Your balance has no end date and can buy page credits at any time. If a pack costs less than the minimum top-up, the amount you added only to reach that minimum can be sent back on request within 30 days of the payment, in USDT or USDC, as long as you have not spent it. Any other balance is not refunded unless the law says otherwise. ### Can I get a refund? A credit pack bought less than 72 hours ago returns to your balance if under 20% of its credits were used. If a pack does not work and support cannot solve it within 24 hours, all of it returns to your balance. Top-ups stay as balance and are not refunded unless the law says otherwise; the amount added only to reach the minimum top-up can be sent back on request within 30 days, in USDT or USDC, while unspent. The [refund policy](https://webreaderapi.com/refunds/) has worked examples. ### Do JavaScript pages always cost two credits? Only when the browser actually runs. With `render=auto`, most news, blog and documentation pages are read from their HTML for one credit; the browser starts only when that HTML holds too little text. The `rendered` field tells you which happened. ### Which cryptocurrencies can I pay with? BTC, USDT, USDC, ETH, SOL, LTC. USDT works on TRC-20 or ERC-20, and USDC on ERC-20. The [crypto payments page](https://webreaderapi.com/crypto-payments/) explains each step. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Privacy policy > WebReaderAPI privacy policy: we log host names, never full URLs or page content, for 30 days. No ID, no card, account-number login. What we keep and why. - URL: https://webreaderapi.com/privacy/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) A web reader sees two kinds of data: yours as a customer, and the pages you ask it to read. This policy covers both, in that order. **Quick answer — What does WebReaderAPI keep about me and the pages I read?** **WebReaderAPI** keeps your account number, an optional email, your payment and credit records, and request logs that hold the host name of each page, never the full URL or its content, for 30 days. We keep no copy of the pages you read. We ask for no ID document and no card, and we do not sell or share your data. ## What data do we collect about you? Only what running the service requires: - **Account:** your account number, and an email address if you choose to add one. - **Payments:** top-ups, their transaction IDs, coins and networks, and the packs you buy. - **Usage:** Each request leaves a log line with the time, the account, the host name of the page (never the full URL or its content), the status and the credits used; these lines are deleted after 30 days. Pages you read are not stored. Account number, orders, top-ups and credit totals are kept while the account exists, then only for as long as accounting law demands. Records go to authorities only when a law obliges us. - **API keys:** stored so we can check your requests, revocable at any time. ## What do we never collect? We never collect identity documents, selfies, company papers or card numbers, because we do not need them. Account-number login; email is optional. We never store the full address of the pages you read, their content, or the Markdown we return to you. Once the answer leaves our servers, the page text is gone from our side. ## Why keep the host name of each page? For two reasons: to show you your usage history, and to answer site owners who report a problem with our crawler. A host name such as `example.com` is enough for both, so we cut every URL down to it before writing the log. Logs are deleted after 30 days. ## Who sees the requests we make for you? The website you ask us to read sees a request from our infrastructure, with our user agent `WebReaderAPI/1.0 (+https://webreaderapi.com/bot/)`. It does not see your account, your key or your identity. When you use the `country` option, the request leaves from an address in that country through our infrastructure, with the same user agent. Our infrastructure receives only what is needed to fetch the page: the URL and the options you chose. ## Do you use cookies? The public pages use no advertising or tracking cookies. The client area uses the storage it needs to keep you signed in and to remember interface choices such as your language. Payment pages show addresses and amounts but set no third-party trackers. ## Do you share or disclose data? We do not sell or rent any data. Records go to authorities only under a legal obligation, and we tell you when the law allows. Account and payment records are kept while your account is open, then only as long as accounting rules require. ## How do I see or delete my data? Your usage history and payments are in the client area. To delete your account, ask from the client area or write to support@webreaderapi.com with an order ID; we will not ask for your account number or a key. Deleting the account removes its credits, as the [terms](https://webreaderapi.com/terms/) explain. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Refund policy > WebReaderAPI refund policy: credit packs back to your balance within 72 hours if under 20% used, full credit for a pack that does not work. Worked examples. - URL: https://webreaderapi.com/refunds/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Refunds go to your balance, where they stay and never expire. This page gives the rule, three examples with real pack sizes, and how to ask. **Quick answer — Can I get a refund on WebReaderAPI credits?** Yes, in two cases. Within 72 hours of purchase, a **WebReaderAPI** credit pack goes back to your balance if you used less than 20% of it. A pack that does not work, and that support cannot fix within 24 hours, is credited back in full. Top-ups stay on your balance; only the part added to reach the $25 minimum can be paid back. ## What is the exact rule? A credit pack bought less than 72 hours ago returns to your balance if under 20% of its credits were used. If a pack does not work and support cannot solve it within 24 hours, all of it returns to your balance. Top-ups stay as balance and are not refunded unless the law says otherwise; the amount added only to reach the minimum top-up can be sent back on request within 30 days, in USDT or USDC, while unspent. Refund requests are handled by a person, and the amount appears on your balance once approved. Failed requests never need a refund, because they use no credit in the first place. That includes pages refused by robots.txt, by a site's AI preference, or by the site itself. ## Example 1: a pack bought by mistake You buy 100,000 pages for $21.75, read 3,000 pages to test, and decide a smaller pack suits you. You ask the next day. You used 3% of the pack, under the 20% limit and within 72 hours, so the full $21.75 goes back to your balance. You can then buy 10,000 pages for $3.75 from it. ## Example 2: a pack that does not work You buy 50,000 pages for $14.25, and every request returns an error on our side for a reason support cannot fix within 24 hours. The pack is credited back in full, whatever share of it you had tried to use. ## Example 3: the minimum top-up You only need 10,000 pages at $3.75, but the minimum top-up is $25. The difference stays on your balance for later packs. If you do not spend it, you can ask for that difference to be paid back within 30 days of the payment, in USDT or USDC. Nothing else in a top-up is paid back, except where the law requires it. ## How do I ask for a refund? Use the support form in the client area, or write to support@webreaderapi.com with the order ID shown in your order history (never your account number or an API key). Support replies within 12 hours; once a refund is approved, the amount appears on your balance. ## Why do refunds go to the balance? Because crypto payments cannot be reversed like card payments, and sending coins back costs network fees each time. A balance credit is immediate and costs you nothing. The one exception, paid back in stablecoins, is the part of a top-up added only to reach the minimum, within 30 days and if unspent. Payments are not paid back otherwise, except where the law requires it. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # WebReaderAPI status: how to check the API is working > How to check that the WebReaderAPI web reader API works: the free credits endpoint, what each error code means, and where incidents are announced. - URL: https://webreaderapi.com/status/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) When a page does not come back, the cause is usually the website, sometimes the network, and rarely us. This page shows how to tell which, in a few seconds and without spending a credit. **Quick answer — How do I know if WebReaderAPI is working?** Call `GET https://webreaderapi.com/api/v1/credits` with your key: it is free, and a normal answer means the API and your key work. If a page fails, the error code tells you why, and **WebReaderAPI** charges nothing for it. Incidents on our side are recorded on the [changelog](https://webreaderapi.com/changelog/) with their date. ## What is the fastest health check? The credits endpoint. It answers without fetching any website, so it tests only our side: authentication, the API and your account. *curl: check the API and your remaining credits* ```bash curl -s https://webreaderapi.com/api/v1/credits -H "Authorization: Bearer $WEBREADER_API_KEY" ``` ## Which errors come from the website, and which from us? The error code says where the problem is. None of these errors uses a credit. | Code | Where the problem is | What to do | |---|---|---| | `blocked_by_robots` | The site's robots.txt refuses our crawler | Nothing: the site has said no | | `blocked_by_site_policy` | The site declares `ai-input=no` | Nothing: the site has said no | | `target_refused` | The site answered 403, 429, a challenge or a login | Try later, or ask the site for access | | `fetch_failed` | DNS, TLS, connection, or a 404/410 page | Check the URL; retry later | | `fetch_timeout` | The site did not answer in time | Retry with a longer `timeout` | | `concurrency_limit` | Too many requests at once from your account | Slow down or ask for a higher limit | | `invalid_api_key`, `insufficient_credits` | Your key or your balance | Check the dashboard | The [errors page](https://webreaderapi.com/docs/errors/) gives each case in detail, with a retry loop. ## Why is there no live uptime figure here? Because we have not measured one yet over a public sample, and an invented figure would mislead you. The success of a read depends mostly on the site being read, so a single percentage would say little. Once we publish figures, they will come with the method, the sample and the date. ## Where are incidents announced? On the [changelog](https://webreaderapi.com/changelog/), with the date, the endpoints affected and what was done. If you see repeated errors on our side (codes other than those caused by the target site), write to support@webreaderapi.com with the request time and the error code. ## Status: common questions ### Does checking my credits cost anything? No. `GET https://webreaderapi.com/api/v1/credits` never uses a credit, so you can call it from a health check as often as you like, within your concurrency limit. ### A page fails for me but opens in my browser. Why? Usually because the site treats robots differently from people: it may refuse our crawler in robots.txt or answer automated requests with a challenge page. We return its answer and charge nothing. The error code and `target_status` show what the site sent. ### Are failed reads ever charged? No. 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Terms of service > WebReaderAPI terms of service: account, page credits that never expire, crypto top-ups, robots.txt rules, refunds and liability, written in plain English. - URL: https://webreaderapi.com/terms/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) These terms cover the website, the client area and the API. Each section answers one question, and nothing important is hidden in a footnote. **Quick answer — What do the WebReaderAPI terms say in short?** You pay in crypto for prepaid page credits for the **WebReaderAPI** web reader API, after a top-up of at least $25. 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged. Prepaid credits never expire: unused pages stay on your account until you use them. Every request follows the target site's robots.txt, and the [Acceptable Use Policy](https://webreaderapi.com/acceptable-use/) is part of these terms. The service is sold by SerpGoat LLC. ## 1. Who is bound by these terms? You and SerpGoat LLC, which operates WebReaderAPI. "You" is the holder of an account, a person or an organisation. Creating an account, paying a top-up or calling the API means you accept these terms and the [Acceptable Use Policy](https://webreaderapi.com/acceptable-use/). ## 2. What does the service do? WebReaderAPI fetches public web pages and PDF files that you name and returns their content as Markdown, plain text or cleaned HTML, through `https://webreaderapi.com/api/v1/read`, `https://webreaderapi.com/api/v1/batch` and `https://webreaderapi.com/api/v1/crawl`. Websites decide what their pages contain and whether robots may read them. We do not control, correct or guarantee the content we return. The same URL can give a different result from one request to the next. ## 3. How do accounts and API keys work? Account-number login; email is optional. You sign in with your account number, so keep it in a safe place. An optional email gives you a way to get it back; with no email on file, a lost number cannot be restored. You can make several keys and revoke any of them from the dashboard. Every request sent with one of your keys uses your credits, whoever sends it, so keep keys private and revoke any key that leaks. One account per person or organisation. ## 4. How are credits counted and when do they expire? 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged. Choosing the country a page is read from adds 10 credits, because the page is then read through a residential connection in that country. PDF files up to 20 MB and 100 pages are converted to Markdown, at 1 credit per started block of 10 pages. Prepaid credits never expire: unused pages stay on your account until you use them. Credits are lost only if you delete your account or if we close it for a breach of these terms. If we stop selling the service, we give at least 30 days' notice so you can use your remaining credits or move their value to another product. ## 5. How do payments work? Prices are in US dollars on the [pricing page](https://webreaderapi.com/pricing/). You top up a balance in BTC, USDT, USDC, ETH, SOL, LTC, with a minimum of $25, and buy credit packs from that balance. Send the exact amount shown on the network shown, then click "I've sent the payment"; the payment is matched by its exact amount and credited once it is confirmed on the blockchain. Your balance keeps its value with no end date. When a pack costs less than the minimum top-up, the extra part can be asked back within 30 days, in USDT or USDC, as long as it is unspent; other top-ups are not paid back unless the law requires it. Network fees are paid by the sender. ## 6. Can I get a refund? Yes, to your balance, in two cases: a pack used at less than 20% within 72 hours of purchase, and a pack that does not work and that support cannot repair within a day. The [refund policy](https://webreaderapi.com/refunds/) states the full rule, with worked examples. ## 7. What rules apply to the pages I read? Our crawler applies three rules to every request, and they cannot be turned off. We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged. If a site declares Content-Signal: ai-input=no in its robots.txt, we do not read its pages for you. If a site refuses our request, we return its answer: we never solve CAPTCHAs, log in to accounts, or try to get around a block or a paywall. You are responsible for how you use the content you receive, including the rights of its owners. The [Acceptable Use Policy](https://webreaderapi.com/acceptable-use/) lists forbidden uses. ## 8. Are there limits on volume or speed? Every account has a limit on how many pages it reads at once; your dashboard shows it, and support can raise it. Crawl one website per job, up to 10,000 pages, following its sitemap, its robots.txt and its Crawl-delay. Up to 100 URLs per batch job. We may slow or pause requests to a single website to protect it, for example by honouring its `Crawl-delay`. ## 9. When can you suspend my account? When we find a breach of these terms or of the acceptable use policy, when a payment is reversed or fraudulent, or when the law requires it. We tell you the reason when the law allows, and a person reviews the facts before any suspension. ## 10. What do you guarantee? We provide the service as it is. We do not guarantee that a given page can be read, that a site will accept our requests, or that the service will run without interruption. Failed requests and pages refused by a site are not charged, which limits your cost when a page cannot be read. ## 11. What is your liability? Where the law allows, our total liability for any claim is limited to the amounts you paid us in the 12 months before the event that caused it. We are not liable for indirect losses, such as lost profits or lost data. Nothing in these terms limits a liability that the law does not allow us to limit. ## 12. How do these terms change? We date every version and record changes in the [changelog](https://webreaderapi.com/changelog/). A change that reduces your rights applies to credits bought after the change is published. Questions go to support@webreaderapi.com. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Web page to Markdown API: the main content of any URL, ready for a model > Web page to Markdown API for LLMs and AI agents: any URL to clean Markdown with title, author and date, from $0.07/1K pages. No KYC, crypto. - URL: https://webreaderapi.com/url-to-markdown-api/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Pass a URL and get the article back as Markdown: headings, paragraphs, lists, tables and links kept; menus, cookie banners, ads and footers removed. With the page's title, author, date and language. **Quick answer — Which API converts a web page into clean Markdown for an LLM?** A web page to Markdown API fetches a page, keeps its main content and returns it as Markdown. **WebReaderAPI** does it in one request: `GET https://webreaderapi.com/api/v1/read?url=…` returns the article with its headings, lists, tables and links, plus title, author, date and word count, for one credit per page from $0.07/1K pages as of October 2026. robots.txt is respected. No KYC; pay in crypto. ## URL to Markdown API at a glance | Attribute | Value | | --- | --- | | Endpoint | `GET https://webreaderapi.com/api/v1/read?url=…` (or `POST` with a JSON body) | | Output | Clean Markdown of the page's main content (or plain text, or cleaned HTML), with title, description, author, publication date, language and word count; links and images on request | | JavaScript | JavaScript rendering in a headless browser when the page needs it (render=auto), or always, or never | | Billing | 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged | | Token budget | `max_chars` cuts the Markdown to the length your model can take | | robots.txt | We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged | | Price | $3.75 for 10,000 pages, $21.75 for 100,000 pages, from $0.07/1K pages at 5,000,000 pages | | Payment | BTC, USDT, USDC, ETH, SOL, LTC, from a $25 top-up; no KYC | ## Why convert web pages to Markdown before giving them to a model? Markdown keeps the structure a model needs and drops the markup it does not. A typical article page carries navigation, scripts, tracking tags, cookie notices and related-post widgets around a few hundred words of text. Sent as raw HTML, most of the tokens you pay for are markup. Markdown keeps headings as `#`, lists as `-`, tables as pipes and links as `[text](url)`. A model can still see which sentence belongs to which section, quote the source link and split the text into chunks along its headings. The same text in plain prose loses that structure; the same text in HTML costs far more tokens. ## What does the API keep, and what does it remove? With `main_content=true`, the default, the API keeps the part of the page that a reader came for and removes the rest. | Kept | Removed | |---|---| | Title and headings | Site navigation and menus | | Paragraphs, quotes and lists | Cookie and consent banners | | Tables, as Markdown tables | Ads and promoted widgets | | Code blocks, with their language when shown | Sidebars and related-post lists | | Links inside the text | Footers and legal boilerplate | | Images, only with `include_images=true` | Share buttons and comment forms | Set `main_content=false` to convert the whole page instead, for example a home page or a product listing where everything matters. Set `format=text` for plain text without Markdown, or `format=html` for cleaned HTML of the main content. *curl: one page as Markdown, cut to 20,000 characters* ```bash curl -G "https://webreaderapi.com/api/v1/read" \ -H "Authorization: Bearer $WEBREADER_API_KEY" \ --data-urlencode "url=https://example.com/blog/pour-over-coffee" \ -d format=markdown -d max_chars=20000 ``` *Python: read a page and print its metadata and Markdown* ```python import os import requests resp = requests.get( "https://webreaderapi.com/api/v1/read", headers={"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}, params={"url": "https://example.com/blog/pour-over-coffee", "format": "markdown"}, timeout=90, ) resp.raise_for_status() page = resp.json() print(page["title"], "|", page.get("author"), "|", page.get("published_at")) print(page["word_count"], "words,", page["credits_used"], "credit(s)") print(page["markdown"][:500]) ``` ## What about pages built with JavaScript? Many pages send almost no text in their first HTML and build the content in the browser. With `render=auto`, the default, the API first reads the raw HTML; if it holds too little text, it opens the page in a headless browser and reads the result. The response says which one happened in `rendered`. A page that needed the browser costs two credits instead of one. If you know a site always needs it, set `render=true`; if you only want what the server sends, set `render=false` and never pay the second credit. ## What happens when a site does not want to be read? We respect its choice and you are not charged. Our crawler identifies itself as `WebReaderAPI` and checks robots.txt before every read. A page disallowed for our token, or for all robots, returns `403 blocked_by_robots`. A site that declares `Content-Signal: ai-input=no` returns `403 blocked_by_site_policy`. If the site refuses the request in another way, such as a 403 status, a challenge page or a login wall, you get `502 target_refused` with the site's own status. If a site refuses our request, we return its answer: we never solve CAPTCHAs, log in to accounts, or try to get around a block or a paywall. Our [crawler page](https://webreaderapi.com/bot/) explains all of this to site owners, with the exact User-Agent. ## How many pages will my project need? Count one credit per page, two for pages that need the browser. Some typical volumes: - **An AI agent** that reads 5 pages per task and runs 100 tasks a day reads about 15,000 pages a month: 50,000 pages ($14.25) last more than three months. - **A RAG index** of three documentation sites with 2,000 pages each is 6,000 pages, then a weekly refresh of changed pages. - **A research corpus** of 400,000 news and blog articles fits in 500,000 pages ($50). Credits never expire, so a pack bought for a one-off crawl keeps its remaining value for later jobs. ## How much does the URL to Markdown API cost? 10,000 pages cost $3.75, 100,000 pages cost $21.75 and 1,000,000 pages cost $74.25, as of October 2026. Failed pages and pages refused by robots.txt are free. See [all packs](https://webreaderapi.com/pricing/). | Plan | Quantity | Price | Price per unit | | --- | --- | --- | --- | | Test | 10,000 pages | $3.75 | $0.38/1K pages | | Starter | 50,000 pages | $14.25 | $0.28/1K pages | | Build | 100,000 pages | $21.75 | $0.22/1K pages | | Volume | 1,000,000 pages | $74.25 | $0.07/1K pages | In every plan: Main-content extraction · JavaScript rendering when needed · Failed and refused pages not charged · Credits never expire **Before you pay** - Minimum top-up $25. Top up once in crypto, then buy any plan from your balance. - Your balance has no end date and can buy page credits at any time. If a pack costs less than the minimum top-up, the amount you added only to reach that minimum can be sent back on request within 30 days of the payment, in USDT or USDC, as long as you have not spent it. Any other balance is not refunded unless the law says otherwise. - Credited once your payment is checked on the blockchain. Typical network confirmation time: BTC ≈ 20 min · USDT TRC-20 ≈ 1 min · USDT ERC-20 ≈ 5 min · USDC ERC-20 ≈ 5 min · ETH ≈ 5 min · SOL ≈ 1 min · LTC ≈ 15 min ## URL to Markdown API: common questions ### Does the Markdown keep tables and code blocks? Yes. HTML tables become Markdown tables, and code blocks stay fenced, with their language when the page marks it. Very complex tables, such as ones with merged cells, are flattened row by row. ### Can I get the whole page instead of the article? Yes. Set `main_content=false` and the API converts everything on the page, including navigation and footer. This suits home pages, category pages and product listings, where the main-content detector would drop useful parts. ### How do I keep pages within my model's context window? Use `max_chars`, between 1,000 and 500,000 characters, to cut the Markdown at a set length. The `word_count` field tells you how long the full article was, so your agent can decide whether to read more. ### Can it read pages behind a login or a paywall? No. WebReaderAPI reads public pages only. It never logs in to accounts and never tries to get around a paywall or a block; such pages return the site's refusal and are not charged. ### Do I need an ID or a card to get a key? No. WebReaderAPI asks for no ID, no selfie, no company documents and no card. Account-number login; email is optional. You top up from $25 in BTC, USDT, USDC, ETH, SOL, LTC. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # What can you build with a web reader API? > What teams build with a web reader API: fetch tools for AI agents, RAG ingestion, page change monitoring and website-to-Markdown exports. From $0.07/1K pages. - URL: https://webreaderapi.com/use-cases/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Four jobs account for most of what people do with clean page text: give an agent a way to read, feed a knowledge base, watch pages for changes, and move a site's content into Markdown. Each has its own page with code and costs. **Quick answer — What is a web reader API used for?** Teams use a web reader API to give AI agents a fetch tool, to ingest sites into RAG indexes, to monitor public pages for changes, and to convert their own websites to Markdown. **WebReaderAPI** covers all four with one key: `/read` for a page, `/batch` for a list, `/crawl` for a site, at one credit per page from $0.07/1K pages as of October 2026. No KYC; pay in crypto. ## Which use case is yours? - **[AI agents](https://webreaderapi.com/use-cases/ai-agents/)** — A read_url tool for Claude, Cursor, OpenClaw, n8n or your own agent, with a length cap per page. - **[RAG ingestion](https://webreaderapi.com/use-cases/rag/)** — Crawl documentation and blogs into Markdown chunks that keep their headings and source URL. - **[Change monitoring](https://webreaderapi.com/use-cases/monitoring/)** — Read pricing pages, policies and changelogs on a schedule and get the differences as text. - **[Website to Markdown](https://webreaderapi.com/use-cases/website-to-markdown/)** — Export your own site to Markdown files for a migration, a backup or your own assistant. - **[PDFs](https://webreaderapi.com/pdf-to-markdown-api/)** — Reports, papers and manuals from a link, converted to Markdown text. - **[llms.txt](https://webreaderapi.com/guides/llms-txt-generator/)** — Build an llms.txt file for your own site from a crawl, with one line per page. ## Which endpoint and how many pages does each use case need? Typical volumes for a small team, to size a first pack. One credit per page, two when the page needs a browser. | Use case | Endpoint | Typical pages per month | | --- | --- | --- | | Agent that reads 5 pages per task, 100 tasks a day | `/read` | About 15,000 | | RAG index of 3 documentation sites, refreshed weekly | `/crawl`, then `/batch` | 6,000 to build, a few hundred a week after | | Watching 200 competitor and policy pages daily | `/batch` | About 6,000 | | Exporting a 2,000-page site once | `/crawl` | 2,000, one time | ## Which problem do these four jobs share? They all need the readable part of a page, not its layout. An agent wastes tokens on menus; an index ranks boilerplate; a change monitor raises false alarms when an ad rotates; a migration copies cookie banners into the new site. Main-content extraction fixes the same problem in each case. They also share the same limits. Pages behind a login or a paywall are out of reach, and sites that refuse robots are skipped and not charged. We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged. ## How do I choose between /read, /batch and /crawl? Use the smallest tool that does the job. - **`/read`** returns one page synchronously. It suits agents, which decide page by page what to read next. - **`/batch`** reads up to 100 URLs you already know, as a background job. It suits monitoring lists and refreshes. - **`/crawl`** discovers and reads a whole site from a start URL, up to 10,000 pages, following its sitemap and robots.txt. It suits first builds and exports. All three take the same options, such as `format`, `main_content`, `render` and `max_chars`, and bill the same way. ## What will a first project cost? Most first projects fit in one small pack. A prototype agent, a single documentation index or a monitoring list of a few hundred pages uses a few thousand to fifteen thousand credits a month. 10,000 pages cost $3.75 and 50,000 pages cost $14.25, as of October 2026. Credits never expire, so buying the larger pack for a project that may grow costs nothing extra in a quiet month. Failed pages and pages refused by robots.txt are free. ## Packs for a first project Start small; credits carry over with no end date. 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged. | Plan | Quantity | Price | Price per unit | | --- | --- | --- | --- | | Test | 10,000 pages | $3.75 | $0.38/1K pages | | Starter | 50,000 pages | $14.25 | $0.28/1K pages | | Build | 100,000 pages | $21.75 | $0.22/1K pages | In every plan: Main-content extraction · JavaScript rendering when needed · Failed and refused pages not charged · Credits never expire **Before you pay** - Minimum top-up $25. Top up once in crypto, then buy any plan from your balance. - Your balance has no end date and can buy page credits at any time. If a pack costs less than the minimum top-up, the amount you added only to reach that minimum can be sent back on request within 30 days of the payment, in USDT or USDC, as long as you have not spent it. Any other balance is not refunded unless the law says otherwise. - Credited once your payment is checked on the blockchain. Typical network confirmation time: BTC ≈ 20 min · USDT TRC-20 ≈ 1 min · USDT ERC-20 ≈ 5 min · USDC ERC-20 ≈ 5 min · ETH ≈ 5 min · SOL ≈ 1 min · LTC ≈ 15 min ## Use cases: common questions ### Can I use one key for several projects? Yes, or create one key per project so you can see usage separately and revoke one without touching the others. Create several API keys, for example one per agent or project, and revoke any of them from your dashboard. ### Which use case needs JavaScript rendering? Any that reads modern web apps, single-page documentation sites or pages that load content after the first response. With `render=auto`, the API renders only when the raw HTML holds too little text. ### Is there a use case WebReaderAPI is not for? Yes: collecting personal data about private people, reading content behind a login or a paywall, and anything that requires getting around a site's refusal. Our [Acceptable Use Policy](https://webreaderapi.com/acceptable-use/) lists the rules. ### Do I need an ID to start? No. WebReaderAPI asks for no ID, no selfie and no documents. A top-up starts at $25, in BTC, USDT, USDC, ETH, SOL, LTC. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Web fetch tool for AI agents: read pages as clean Markdown > A web fetch tool for AI agents: read any public page as Markdown with a length cap, via MCP, OpenAI tools, LangChain or n8n, from $0.07/1K pages. Crypto. - URL: https://webreaderapi.com/use-cases/ai-agents/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) An agent that can search but not read answers from snippets. Give it a tool that turns any public URL into short, clean Markdown, and it can read the source, quote it and cite it, within a budget you set. **Quick answer — What is the best way to give an AI agent a web fetch tool?** Wrap a web reader API as a tool that takes a URL and returns the page's main content as Markdown, cut to a set length. **WebReaderAPI** does the reading, renders JavaScript when needed, respects robots.txt, and charges one credit per page from $0.07/1K pages as of October 2026. Ready tool code covers MCP, OpenAI function calling, LangChain, n8n and OpenClaw. No KYC; crypto. ## WebReaderAPI as an agent tool | Attribute | Value | | --- | --- | | Tool | `read_url(url, max_chars)` calling `https://webreaderapi.com/api/v1/read` | | Returns | Markdown of the main content, plus title, author, date and word count | | Length cap | `max_chars`, 1,000 to 500,000 characters per page | | JavaScript pages | JavaScript rendering in a headless browser when the page needs it (render=auto), or always, or never | | Refusals | Returned as clear errors, never charged | | Integrations | [MCP](https://webreaderapi.com/docs/mcp/), [OpenAI](https://webreaderapi.com/docs/openai/), [LangChain and LlamaIndex](https://webreaderapi.com/docs/langchain/), [n8n](https://webreaderapi.com/docs/n8n/), [OpenClaw](https://webreaderapi.com/docs/openclaw/) | | Price | $3.75 for 10,000 pages; credits never expire | ## Why does an agent need a reader rather than raw HTML? Because raw HTML spends the context window on things the model does not need. A news article or a docs page arrives wrapped in navigation, scripts, consent banners and related links. Passed as is, it crowds out the conversation and makes the model quote menus. A reader solves three problems at once. It keeps only the main content, it renders pages that are empty until JavaScript runs, and it turns PDFs into text. The agent gets one compact block of Markdown with a title and a date it can cite. ## Why do built-in fetch tools sometimes fail? Mostly because sites refuse automated traffic. Since 1 July 2025, Cloudflare blocks AI crawlers by default for new domains, and from 15 September 2026 its presets block "Agent" traffic on ad-supported pages of new domains. Built-in tools in coding agents and assistants meet those refusals like any other client. Other failures are technical. OpenClaw's documentation, read on 22 October 2026, says its built-in web_fetch extracts content with Readability, does not run JavaScript and caps output at 20,000 characters. Pages built in the browser, long documents, slow servers and rate limits all trip such tools. A reader that renders pages fixes the JavaScript case; caching and patience fix the rest. Refusals are the site owner's call. Our [guide to web fetch 403 errors](https://webreaderapi.com/guides/web-fetch-403-errors/) explains each case. ## Why is the tool called read_url and not web_fetch? To avoid a clash with tools agents already have. Claude Code and OpenClaw both ship a built-in tool named `web_fetch`. A second tool with the same name confuses the model about which one it is calling. Naming ours `read_url` keeps both available: the agent can use its built-in fetcher for quick checks and `read_url` when it needs clean Markdown, PDFs or JavaScript pages. The tool definitions on our docs pages all use this name, so you can switch between frameworks without changing your prompts. *Python: a compact read_url tool with a length cap* ```python import os import requests API = "https://webreaderapi.com/api/v1/read" HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]} def read_url(url: str, max_chars: int = 12000) -> str: """Return a web page as Markdown for a model, or a short reason it was not read.""" resp = requests.get(API, headers=HEADERS, params={"url": url, "max_chars": max_chars}, timeout=90) data = resp.json() if resp.status_code != 200: err = data.get("error", {}) return "Not read (" + err.get("code", "error") + "): " + err.get("message", "") head = "# " + (data.get("title") or url) + "\nSource: " + data["final_url"] if data.get("published_at"): head += " | Published: " + data["published_at"] return head + "\n\n" + data["markdown"] if __name__ == "__main__": print(read_url("https://example.com/blog/pour-over-coffee", max_chars=2000)[:500]) ``` ## How should an agent decide what to read? Give it a budget and a rule. Three habits keep costs and context in check: - **Read few, read well.** Two to four sources per question are usually enough to answer with citations. - **Cap each page.** Start with `max_chars` around 10,000 to 15,000 characters; raise it only when the `word_count` shows the answer is likely further down. - **Report refusals.** When a page is not read, let the model say so and move to another source instead of retrying. The tool above returns refusals as one plain sentence, so the model can explain them to the user. ## What does it cost to run an agent on WebReaderAPI? One credit per page, two when the page needs a browser. An assistant that answers 100 questions a day and reads 4 pages for each reads about 12,000 pages a month. 50,000 pages ($14.25) cover that for four months, as of October 2026. Credits never expire, so an agent that is busy one week and idle the next does not waste a monthly quota. Pages that fail or that robots.txt refuses cost nothing. ## Which pack suits an agent? Prototype on 10,000 pages; move to 50,000 pages or 100,000 pages when the agent serves real users. Credits never expire. | Plan | Quantity | Price | Price per unit | | --- | --- | --- | --- | | Test | 10,000 pages | $3.75 | $0.38/1K pages | | Starter | 50,000 pages | $14.25 | $0.28/1K pages | | Build | 100,000 pages | $21.75 | $0.22/1K pages | In every plan: Main-content extraction · JavaScript rendering when needed · Failed and refused pages not charged · Credits never expire **Before you pay** - Minimum top-up $25. Top up once in crypto, then buy any plan from your balance. - Your balance has no end date and can buy page credits at any time. If a pack costs less than the minimum top-up, the amount you added only to reach that minimum can be sent back on request within 30 days of the payment, in USDT or USDC, as long as you have not spent it. Any other balance is not refunded unless the law says otherwise. - Credited once your payment is checked on the blockchain. Typical network confirmation time: BTC ≈ 20 min · USDT TRC-20 ≈ 1 min · USDT ERC-20 ≈ 5 min · USDC ERC-20 ≈ 5 min · ETH ≈ 5 min · SOL ≈ 1 min · LTC ≈ 15 min ## AI agents: common questions ### Can I add WebReaderAPI to Claude Desktop or Cursor? Yes, through a small MCP server that exposes `read_url`. The [MCP page](https://webreaderapi.com/docs/mcp/) has the code and the configuration snippet; your API key stays in an environment variable. ### Does it work with OpenClaw? Yes, as an MCP server next to OpenClaw's built-in web_fetch. The [OpenClaw page](https://webreaderapi.com/docs/openclaw/) shows the commands from OpenClaw's own documentation. ### Can the agent read pages that block AI bots? No. If robots.txt disallows the page, the API returns `blocked_by_robots`; if the site refuses, it returns `target_refused` with the site's status. Neither is charged, and we never try to get around the refusal. ### How do I stop a long page from filling the context window? Set `max_chars`. The Markdown is cut at that length, and `word_count` tells the agent how long the full page was, so it can ask for more only when needed. ### Do I need a card or an ID? No. WebReaderAPI asks for no ID, no selfie, no documents and no card. Payment is in BTC, USDT, USDC, ETH, SOL, LTC, with a minimum top-up of $25. ## Sources 1. [Press release: AI crawlers blocked by default](https://www.cloudflare.com/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/) — Cloudflare (2025-07-01) 2. [Blog: new defaults for AI crawlers and agents](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) — Cloudflare (2026-09-15) 3. [Web fetch tool](https://docs.openclaw.ai/tools/web-fetch.md) — OpenClaw (read 2026-10-22) --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Website change monitoring: diff public pages as clean Markdown > Monitor public web pages for changes: read pricing pages, policies and changelogs as Markdown on a schedule, diff the text and alert. From $0.07/1K pages. - URL: https://webreaderapi.com/use-cases/monitoring/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Comparing raw HTML raises an alert every time an ad or a token changes. Comparing the main content as Markdown only flags what a reader would notice: a new price, a reworded clause, a fresh changelog entry. **Quick answer — How can I monitor web pages for content changes?** Read each page on a schedule as Markdown of its main content, store the text, and compare it with the previous version; alert only when the text changes. **WebReaderAPI** reads up to 100 URLs per batch job, removes ads and menus so they cannot trigger false alarms, and charges one credit per page from $0.07/1K pages as of October 2026. No KYC; pay in crypto. ## Monitoring with WebReaderAPI | Attribute | Value | | --- | --- | | Endpoint | `POST https://webreaderapi.com/api/v1/batch`, then `GET https://webreaderapi.com/api/v1/jobs/{job_id}` | | Batch size | Up to 100 URLs per batch job | | What is compared | Markdown of the main content, without ads, menus or banners | | Billing | 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged | | Example volume | 200 pages checked daily = about 6,000 pages a month | | Price | $3.75 for 10,000 pages; credits never expire | ## What is worth monitoring? Pages whose wording matters to your business and that change without notice: - **Pricing pages** of suppliers and competitors, to see new plans and price moves. - **Terms, privacy and refund policies** you depend on, to catch clauses that change. - **Changelogs and release notes** of tools you build on, to react to breaking changes. - **Documentation pages** your product links to, to fix broken guidance quickly. - **Public notices** from regulators or standards bodies in your field. Keep the list to public pages. Personal profiles and pages behind a login are out of scope. ## Why diff Markdown instead of HTML? Because HTML changes all the time without the content changing. Rotating ads, session tokens, timestamps in footers and A/B test markup make two loads of the same page differ. A diff on HTML turns that noise into alerts, and people stop reading them. Main-content Markdown removes most of the noise before you compare. What is left is the text a person reads: headings, paragraphs, lists and tables. A diff on that text is short, readable and worth an alert. *Python: check a list of pages and print what changed* ```python import difflib import json import os import pathlib import time import requests API = "https://webreaderapi.com/api/v1" HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]} STORE = pathlib.Path("snapshots.json") URLS = [ "https://example.com/pricing", "https://example.com/legal/terms", "https://example.com/changelog", ] old = json.loads(STORE.read_text(encoding="utf-8")) if STORE.exists() else {} job = requests.post(API + "/batch", headers=HEADERS, json={"urls": URLS}, timeout=60).json() results, cursor, delay = [], None, 5 while True: params = {"cursor": cursor} if cursor else {} page = requests.get(API + "/jobs/" + job["job_id"], headers=HEADERS, params=params, timeout=60).json() results += page.get("results", []) cursor = page.get("next_cursor") if page["status"] == "done" and not cursor: break if not cursor: time.sleep(delay) delay = min(delay * 2, 60) new = dict(old) for item in results: if "markdown" not in item: print("not read:", item["url"], item.get("error", {}).get("code")) continue before = old.get(item["url"], "") if before and before != item["markdown"]: diff = difflib.unified_diff(before.splitlines(), item["markdown"].splitlines(), lineterm="", n=0) print("CHANGED:", item["url"]) print("\n".join(list(diff)[:40])) new[item["url"]] = item["markdown"] STORE.write_text(json.dumps(new, ensure_ascii=False), encoding="utf-8") ``` ## How often should I check, and how politely? Check as often as the page realistically changes, and no more. Daily suits pricing pages and changelogs; weekly suits policies and documentation. A page read every hour rarely tells you more than one read every day, and it costs 24 times as much. WebReaderAPI follows robots.txt for every page and honours a site's `Crawl-delay` in crawl jobs. Pages the site disallows are skipped and not charged, so a strict site simply drops out of your report. Ask the owner if you need access to a page that refuses robots. ## What does monitoring cost? One credit per page per check. Some examples at the prices of October 2026: - **50 pages, daily:** about 1,500 pages a month. 10,000 pages ($3.75) last more than six months. - **200 pages, daily:** about 6,000 pages a month. 50,000 pages ($14.25) last about eight months. - **1,000 pages, weekly:** about 4,300 pages a month, close to the previous case. Pages that need a browser use two credits; set `render=false` for pages you know are plain HTML to keep the count at one. ## Which pack suits monitoring? Multiply pages by checks per month, then pick a pack that lasts a few months. Credits never expire. | Plan | Quantity | Price | Price per unit | | --- | --- | --- | --- | | Test | 10,000 pages | $3.75 | $0.38/1K pages | | Starter | 50,000 pages | $14.25 | $0.28/1K pages | | Build | 100,000 pages | $21.75 | $0.22/1K pages | In every plan: Main-content extraction · JavaScript rendering when needed · Failed and refused pages not charged · Credits never expire **Before you pay** - Minimum top-up $25. Top up once in crypto, then buy any plan from your balance. - Your balance has no end date and can buy page credits at any time. If a pack costs less than the minimum top-up, the amount you added only to reach that minimum can be sent back on request within 30 days of the payment, in USDT or USDC, as long as you have not spent it. Any other balance is not refunded unless the law says otherwise. - Credited once your payment is checked on the blockchain. Typical network confirmation time: BTC ≈ 20 min · USDT TRC-20 ≈ 1 min · USDT ERC-20 ≈ 5 min · USDC ERC-20 ≈ 5 min · ETH ≈ 5 min · SOL ≈ 1 min · LTC ≈ 15 min ## Change monitoring: common questions ### Does WebReaderAPI send alerts? No. It reads pages and returns Markdown; your script compares versions and sends the alert by email, chat or ticket. The code above prints the changed lines as a starting point. ### How do I avoid false alarms from dates and counters? Compare main-content Markdown, not HTML, and strip lines you know change on every load, such as "updated 5 minutes ago", before diffing. Most rotating elements are already removed by main-content extraction. ### Can I monitor pages that need JavaScript? Yes. With `render=auto`, pages that are empty without JavaScript are rendered in a browser for two credits. Pages that work without it stay at one. ### Can I monitor a whole site rather than a list of pages? Yes: run a crawl job instead of a batch, then compare page by page. For large sites, compare the sitemap's `lastmod` dates first and re-read only the pages that moved. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Web pages for RAG: turn sites into clean, chunkable Markdown > Feed RAG pipelines with web pages: crawl docs and blogs into clean Markdown, chunk by heading, keep source URLs. From $0.07/1K pages, no KYC, crypto. - URL: https://webreaderapi.com/use-cases/rag/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Retrieval works on what you put in the index. Clean Markdown with headings, metadata and a source URL gives better chunks, fewer wasted tokens and answers that can cite their page. **Quick answer — How do I ingest web pages into a RAG pipeline?** Read each page as Markdown of its main content, split it along its headings, attach the source URL and title to every chunk, then embed and store. **WebReaderAPI** handles the reading: one request per page, or one crawl job for a whole site of up to 10,000 pages, at one credit per page from $0.07/1K pages as of October 2026. robots.txt is respected. No KYC; pay in crypto. ## RAG ingestion with WebReaderAPI | Attribute | Value | | --- | --- | | Whole site | `POST https://webreaderapi.com/api/v1/crawl`: Crawl one website per job, up to 10,000 pages, following its sitemap, its robots.txt and its Crawl-delay | | List of URLs | `POST https://webreaderapi.com/api/v1/batch`: Up to 100 URLs per batch job | | Per page | Markdown of the main content, title, author, date, language, word count | | PDFs | PDF files up to 20 MB and 100 pages are converted to Markdown, at 1 credit per started block of 10 pages | | Billing | 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged | | Price | $14.25 for 50,000 pages; credits never expire | ## Why does clean Markdown give better retrieval? Chunks made of boilerplate retrieve badly. When a page is split with its menus, cookie text and footer still in it, many chunks contain the same navigation words, and the vector search returns them for unrelated questions. Main-content extraction removes that noise before you embed. Headings matter just as much. A chunk that starts with `## Refund policy` carries its topic with it, so both the embedding and the model reading it later know what it is about. Plain text loses that signal; raw HTML hides it under markup. ## What does an ingestion pipeline look like? Five steps, whatever vector store you use: 1. **Collect URLs.** Crawl the site, or send the list you already have as a batch. 2. **Read as Markdown.** Keep `main_content=true` for articles and docs; use `false` for listing pages. 3. **Chunk by heading.** Split on `#`, `##` and `###`, then cut long sections to your size limit with a small overlap. 4. **Attach metadata.** Store `url`, `title`, `published_at` and the heading path with each chunk, so answers can cite their source. 5. **Refresh.** Re-read pages whose sitemap `lastmod` changed, and delete chunks of pages that now return an error. *Python: read pages and split them into chunks with their source* ```python import os import re import requests API = "https://webreaderapi.com/api/v1/read" HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]} URLS = [ "https://docs.example.com/getting-started", "https://docs.example.com/billing", ] def chunks_for(url, max_len=2000): page = requests.get(API, headers=HEADERS, params={"url": url}, timeout=90).json() if "markdown" not in page: return [] out = [] for section in re.split(r"\n(?=#{1,3} )", page["markdown"]): section = section.strip() heading = section.splitlines()[0] if section else "" for start in range(0, len(section), max_len): out.append({ "text": section[start:start + max_len], "url": page.get("final_url", url), "title": page.get("title"), "heading": heading, }) return out all_chunks = [c for u in URLS for c in chunks_for(u)] print(len(all_chunks), "chunks ready to embed") ``` ## How much does it cost to build and refresh an index? Building costs one credit per page, two for pages that need a browser. Refreshing costs only the pages you re-read. Three examples, at the prices of October 2026: - **A product's help center** of 800 pages, refreshed monthly for changed pages (about 10%): 800 credits once, then about 80 a month. 10,000 pages ($3.75) last for years. - **Ten documentation sites** of 2,000 pages each, refreshed weekly at 5%: 20,000 credits, then about 4,000 a month. 50,000 pages ($14.25) cover the build and seven months of refreshes. - **A news archive** of 300,000 articles that never change: 300,000 credits once, which fit in 500,000 pages ($50). Embedding and storage are billed by your own providers and are not included. ## Can I index sites that do not want to be indexed? Not through WebReaderAPI. Pages disallowed by robots.txt for our crawler, or for all robots, are skipped and not charged; sites that declare `Content-Signal: ai-input=no` are not read for you. For your own sites or partner sites that block robots, ask the owner to allow `WebReaderAPI` in robots.txt; our [crawler page](https://webreaderapi.com/bot/) shows the lines to add. This matters for your users too. A RAG assistant that cites sources is more useful when those sources agreed to be read. ## Which pack fits a RAG project? Count the pages to build the index, add the monthly refresh, and pick the pack that covers a few months. Credits never expire. | Plan | Quantity | Price | Price per unit | | --- | --- | --- | --- | | Test | 10,000 pages | $3.75 | $0.38/1K pages | | Starter | 50,000 pages | $14.25 | $0.28/1K pages | | Build | 100,000 pages | $21.75 | $0.22/1K pages | | Scale | 500,000 pages | $50 | $0.10/1K pages | In every plan: Main-content extraction · JavaScript rendering when needed · Failed and refused pages not charged · Credits never expire **Before you pay** - Minimum top-up $25. Top up once in crypto, then buy any plan from your balance. - Your balance has no end date and can buy page credits at any time. If a pack costs less than the minimum top-up, the amount you added only to reach that minimum can be sent back on request within 30 days of the payment, in USDT or USDC, as long as you have not spent it. Any other balance is not refunded unless the law says otherwise. - Credited once your payment is checked on the blockchain. Typical network confirmation time: BTC ≈ 20 min · USDT TRC-20 ≈ 1 min · USDT ERC-20 ≈ 5 min · USDC ERC-20 ≈ 5 min · ETH ≈ 5 min · SOL ≈ 1 min · LTC ≈ 15 min ## RAG ingestion: common questions ### What chunk size should I use? Split on headings first, then cut sections longer than about 500 to 1,000 tokens with a small overlap. Heading-based chunks keep one topic each, which usually retrieves better than fixed-size windows over the whole page. ### Does WebReaderAPI create embeddings or store vectors? No. WebReaderAPI reads pages and returns Markdown with metadata. You embed with the model of your choice and store in your own vector database; the [LangChain and LlamaIndex page](https://webreaderapi.com/docs/langchain/) shows both. ### How do I keep the index up to date? Re-read only the pages that changed. Most sitemaps carry a `lastmod` date per URL; compare it with your last read and send the changed URLs as a batch. Pages that now fail should be removed from the index. ### Can I include PDFs from the same site? Yes. A crawl reads PDF links on the same domain with the same robots.txt checks. PDF files up to 20 MB and 100 pages are converted to Markdown, at 1 credit per started block of 10 pages. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto. --- # Convert your website to Markdown: crawl it and export every page > Convert your website to Markdown files: crawl your own site and export each page with front matter, for a migration, a backup or your own assistant. No KYC. - URL: https://webreaderapi.com/use-cases/website-to-markdown/ - Last updated: 2026-10-22 - Publisher: WebReaderAPI (https://webreaderapi.com) Moving to a static site generator, a docs platform or a knowledge base often starts with one question: how do I get my pages out as clean Markdown? A crawl does it in one job, with titles, dates and URLs kept in front matter. **Quick answer — How do I convert my whole website to Markdown?** Crawl your site, read each page's main content as Markdown, and write one `.md` file per page with its title, URL and date in front matter. **WebReaderAPI** crawls up to 10,000 pages per job, follows your sitemap and robots.txt, and charges one credit per page from $0.07/1K pages as of October 2026. A Python script below writes the files. No KYC; pay in crypto. ## Website export with WebReaderAPI | Attribute | Value | | --- | --- | | Endpoint | `POST https://webreaderapi.com/api/v1/crawl`, results from `GET https://webreaderapi.com/api/v1/jobs/{job_id}` | | Scope | Crawl one website per job, up to 10,000 pages, following its sitemap, its robots.txt and its Crawl-delay | | Per page | Markdown of the main content, plus title, description, author, date and language | | Whole pages | `main_content=false` keeps navigation and footers when you need them | | Your robots.txt | Must allow `WebReaderAPI` for the pages you export | | Price | $3.75 for 10,000 pages; credits never expire | ## When is a Markdown export useful? Whenever content has to leave the system that renders it. Common cases: - **Migrations.** Moving from a CMS to a static site generator or a documentation platform that stores pages as Markdown. - **Backups.** A readable archive of your published pages that does not depend on any database or theme. - **Your own assistant.** Feeding your site's text into a help bot or an internal search, with each chunk linked to its page. - **Audits.** Reading all your pages as plain text to find outdated claims, broken wording or duplicate content. In each case you want the words and the structure, not the theme around them. *Python: crawl your site into Markdown files with front matter* ```python import os import pathlib import time from urllib.parse import urlparse import requests API = "https://webreaderapi.com/api/v1" HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]} SITE = "https://www.example.com/" OUT = pathlib.Path("export") def target_path(url): path = urlparse(url).path.strip("/") or "index" return OUT / (path + ".md") def front_matter(page): fields = {"title": page.get("title", ""), "source": page["final_url"], "date": page.get("published_at") or ""} lines = ["---"] + [key + ": " + repr(value) for key, value in fields.items()] + ["---", ""] return "\n".join(lines) job = requests.post(API + "/crawl", headers=HEADERS, json={"url": SITE, "limit": 2000, "max_depth": 5}, timeout=60).json() cursor, delay, written = None, 5, 0 while True: params = {"cursor": cursor} if cursor else {} batch = requests.get(API + "/jobs/" + job["job_id"], headers=HEADERS, params=params, timeout=60).json() for page in batch.get("results", []): if "markdown" not in page: continue dest = target_path(page["final_url"]) dest.parent.mkdir(parents=True, exist_ok=True) dest.write_text(front_matter(page) + page["markdown"] + "\n", encoding="utf-8") written += 1 cursor = batch.get("next_cursor") if batch["status"] == "done" and not cursor: break if not cursor: time.sleep(delay) delay = min(delay * 2, 60) print(written, "files in", OUT) ``` ## How do I keep the folder structure and the URLs? Mirror the URL path in the file path, as the script does: `/guides/setup/` becomes `export/guides/setup.md`. Keep the original URL in the front matter as `source`, so you can set up redirects after the migration and check that nothing was lost. Links inside the Markdown still point to your current URLs. After the move, replace the domain or the path prefix in one pass, then test a sample of pages. Tables and code blocks stay as Markdown, so most static generators render them without changes. ## What if my own robots.txt blocks crawlers? Allow our token for the time of the export. WebReaderAPI reads only what robots.txt permits for `WebReaderAPI` or for all robots, including on your own site. Add a group such as `User-agent: WebReaderAPI` with `Allow: /`, run the crawl, then remove it if you like. Our [crawler page](https://webreaderapi.com/bot/) shows the exact lines and the User-Agent your server logs will show. Pages that stay disallowed are skipped and not charged. ## How much does an export cost? One credit per page, two for pages that need a browser to show their text. A 2,000-page site rendered on the server costs 2,000 credits once; the same site as a JavaScript app can cost up to 4,000. Both fit in 10,000 pages for $3.75, as of October 2026. A site of 50,000 pages needs several jobs, split by section with `include_paths`, and fits in 100,000 pages ($21.75). Credits that remain after the export stay on your account for later refreshes. ## Which pack covers my export? Count your pages, double the count if the site needs JavaScript to render, and pick the pack above it. Credits never expire. | Plan | Quantity | Price | Price per unit | | --- | --- | --- | --- | | Test | 10,000 pages | $3.75 | $0.38/1K pages | | Starter | 50,000 pages | $14.25 | $0.28/1K pages | | Build | 100,000 pages | $21.75 | $0.22/1K pages | In every plan: Main-content extraction · JavaScript rendering when needed · Failed and refused pages not charged · Credits never expire **Before you pay** - Minimum top-up $25. Top up once in crypto, then buy any plan from your balance. - Your balance has no end date and can buy page credits at any time. If a pack costs less than the minimum top-up, the amount you added only to reach that minimum can be sent back on request within 30 days of the payment, in USDT or USDC, as long as you have not spent it. Any other balance is not refunded unless the law says otherwise. - Credited once your payment is checked on the blockchain. Typical network confirmation time: BTC ≈ 20 min · USDT TRC-20 ≈ 1 min · USDT ERC-20 ≈ 5 min · USDC ERC-20 ≈ 5 min · ETH ≈ 5 min · SOL ≈ 1 min · LTC ≈ 15 min ## Website to Markdown: common questions ### Can I export the whole page, including menus and footer? Yes. Set `main_content=false` in the crawl options and each file holds the full page converted to Markdown. Most migrations prefer the main content only, because the new theme supplies its own navigation. ### Are images downloaded? No. With `include_images=true`, image links stay in the Markdown with their alt text; download the files themselves with your own script if the migration needs them. ### Can I export a site I do not own? Only public pages that its robots.txt allows, and only for uses its terms and the law permit. For a migration or a backup, export your own sites or sites whose owner agreed. ### How large a site can one job handle? Up to 10,000 pages per crawl job, within one domain. Split larger sites by section with `include_paths` and run several jobs. --- WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto.