# Crawl4AI or a hosted API? When to run your own web reader

> Crawl4AI or a hosted API for web pages to Markdown? Licences, robots.txt defaults, what self-hosting costs, and hosted prices checked on 2026-10-22.

- URL: https://webreaderapi.com/compare/crawl4ai-vs-api/
- Last updated: 2026-10-22
- Publisher: WebReaderAPI (https://webreaderapi.com)

Open-source tools can turn web pages into Markdown for free, on your own servers. Hosted APIs charge per page and run the servers for you. This page sets out what each choice really costs, from licences and robots.txt defaults to browsers and maintenance.

**Quick answer — Should I self-host Crawl4AI or use a hosted web reader API?**

Self-host Crawl4AI (Apache-2.0, with an attribution requirement) when you have steady volume and people to run browsers and servers; note that its robots.txt check is off by default. Use a hosted API when volume is uneven or you want no operations work: Crawl4AI Cloud starts at $10 for 10,000 credits, and **WebReaderAPI** at $3.75 for 10,000 pages, paid in crypto with no KYC (22 October 2026).

## Which open-source tools turn web pages into Markdown?

The main options in October 2026, with the licence and defaults their own repositories and docs state.

| Tool | Licence | Renders JavaScript | robots.txt by default | MCP server |
| --- | --- | --- | --- | --- |
| Crawl4AI | Apache-2.0, plus a README attribution requirement | Yes (Playwright) | Off: `check_robots_txt` defaults to False | Yes, on your own server |
| Firecrawl (self-hosted) | AGPL-3.0 core; SDKs and some UI parts MIT | Yes (Playwright service) | On | Yes (firecrawl-mcp-server, MIT) |
| Microsoft MarkItDown | MIT | No | Not documented | Yes (markitdown-mcp) |
| Trafilatura | Apache-2.0 (GPLv3+ before v1.8.0) | No | Its spider follows the Robots Exclusion Standard | None mentioned |
| Mozilla Readability | Apache-2.0 | No; needs a DOM and does not fetch | Not applicable | None found |
| Jina Reader (open-source branch) | Apache-2.0 | Yes (Puppeteer) | Not mentioned | Not in the repository |

*Prices checked on October 22, 2026.*

From the GitHub repositories and documentation listed below, read on 2026-10-22.

## What does self-hosting a web reader really cost?

The software is free; running it is not. Before you compare a hosted price with zero, count these:

- **Servers and browsers.** Static pages need little CPU, but JavaScript pages need a headless browser per request, with memory to match. Peaks need spare capacity that sits idle the rest of the time.
- **Maintenance.** Browser versions, extraction rules and dependencies change. Crawl4AI shipped v0.9.5 on 14 October 2026; MarkItDown 0.1.9 on 21 October 2026. Someone has to test and deploy each update.
- **Network.** Outbound bandwidth, DNS, timeouts and retries are yours to tune. Sites can refuse traffic from your servers' addresses, and the right answer then is to respect the refusal.
- **Policy.** Crawl4AI leaves `check_robots_txt` off unless you turn it on, and lets you set or randomise the user agent. Deciding and enforcing your own robots.txt and identification rules is part of the job.
- **Security.** MarkItDown's README warns that it performs I/O with the privileges of the current process and that inputs should be sanitised in untrusted environments. A service that fetches arbitrary URLs must also refuse private and local addresses.

## What do hosted readers charge in comparison?

Hosted APIs turn those costs into a price per page. Two examples from 22 October 2026:

| | Crawl4AI Cloud | WebReaderAPI (ours) |
|---|---|---|
| Entry | Starter $10 a month, 10,000 credits | $3.75 for 10,000 pages |
| Static page | 0.5 credit | 1 credit |
| Page with a browser | 1 credit ("real browser") | 2 credits |
| Larger plans | Builder $99 for 125,000 credits; Scale $499 for 750,000 | $74.25 for 1,000,000 pages; $350 for 5,000,000 pages |
| Unused credits | Monthly credits do not roll over; extra credits never expire | Never expire |
| Payment | Card through Stripe | BTC, USDT, USDC, ETH, SOL, LTC |
| robots.txt | Customers must honour sites' robots directives (terms) | Checked by the service on every read |

Crawl4AI Cloud is run by the Crawl4AI maintainers and marks its prices as a "soft launch" that may change. Its credit is worth $0.001, so a static page costs $0.50 per 1,000 on Starter.

## When should I self-host?

Self-hosting pays off when three things are true: your volume is high and steady, you already run servers and browsers, and you need control that an API cannot give.

- **Custom extraction.** Crawl4AI's CSS, XPath and LLM schemas, or your own rules on top of Readability or Trafilatura.
- **Data residency.** Pages never leave your infrastructure.
- **Inputs beyond web pages.** MarkItDown converts Word, Excel, PowerPoint, EPub and more, which a web reader API does not.
- **Licence fit.** Apache-2.0 and MIT tools allow closed products; Firecrawl's AGPL-3.0 core asks you to share changes if you offer it as a network service.

## When is a hosted API the better choice?

Buy pages instead of running servers when your workload is uneven, your team is small, or the reader is a tool inside an agent rather than a product of its own.

- **Spiky or seasonal volume.** A crawl twice a year does not justify a browser fleet. With WebReaderAPI, Prepaid credits never expire: unused pages stay on your account until you use them.
- **Agents.** A hosted endpoint plugs into MCP, LangChain or OpenAI tools in minutes; see [our MCP server](https://webreaderapi.com/docs/mcp/).
- **A policy you do not have to build.** We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged. If a site declares Content-Signal: ai-input=no in its robots.txt, we do not read its pages for you.
- **Payment without a card.** WebReaderAPI takes BTC, USDT, USDC, ETH, SOL, LTC with no KYC.

## Can I combine both?

Yes, and many teams do. Run an open-source converter for files you already hold, such as PDFs and Office documents with MarkItDown, and call a hosted reader for live web pages, especially JavaScript pages that would otherwise need your own browsers.

Keep the output format the same on both sides, Markdown with a source URL and a title, so your chunking and indexing code does not care where a document came from. Our [RAG guide](https://webreaderapi.com/use-cases/rag/) shows that format in practice.

## What does a hosted reader cost with WebReaderAPI?

Per-page credits paid in crypto, as of 22 October 2026: 1 credit per static page, 2 with a browser, nothing for failed or refused pages. Credits never expire.

| Plan | Quantity | Price | Price per unit |
| --- | --- | --- | --- |
| Test | 10,000 pages | $3.75 | $0.38/1K pages |
| Build | 100,000 pages | $21.75 | $0.22/1K pages |
| Volume | 1,000,000 pages | $74.25 | $0.07/1K pages |
| Bulk | 5,000,000 pages | $350 | $0.07/1K pages |

In every plan: Main-content extraction · JavaScript rendering when needed · Failed and refused pages not charged · Credits never expire

**Before you pay**

- Minimum top-up $25. Top up once in crypto, then buy any plan from your balance.
- Your balance has no end date and can buy page credits at any time. If a pack costs less than the minimum top-up, the amount you added only to reach that minimum can be sent back on request within 30 days of the payment, in USDT or USDC, as long as you have not spent it. Any other balance is not refunded unless the law says otherwise.
- Credited once your payment is checked on the blockchain. Typical network confirmation time: BTC ≈ 20 min · USDT TRC-20 ≈ 1 min · USDT ERC-20 ≈ 5 min · USDC ERC-20 ≈ 5 min · ETH ≈ 5 min · SOL ≈ 1 min · LTC ≈ 15 min

## Self-hosting a web reader: common questions

### Is Crawl4AI free for commercial use?

Its README says it is licensed under Apache-2.0, which allows commercial use, and adds that users must include an attribution, either a badge or a text line. Read the licence and README yourself before you ship.

### Does Crawl4AI respect robots.txt?

Only if you ask. Its `check_robots_txt` parameter defaults to False; set it to True to check and cache robots.txt rules before crawling. You can also set or randomise the user agent, so your own policy decides how it behaves.

### Can I self-host Firecrawl instead of Crawl4AI?

Yes. Firecrawl's core is open source under AGPL-3.0 and respects robots.txt by default. The AGPL asks you to publish changes when you offer the software as a network service, which matters for commercial products.

### Which tool converts PDFs and Office files?

Microsoft MarkItDown, under the MIT licence, converts PDF, Word, PowerPoint, Excel, HTML, EPub and more into Markdown. It does not render JavaScript pages. WebReaderAPI reads PDFs from a URL, up to 100 pages.

### How much does Crawl4AI Cloud cost?

On 22 October 2026: Starter $10 a month for 10,000 credits, Builder $99 for 125,000 and Scale $499 for 750,000, with a static page at 0.5 credit and a browser page at 1. The site marks these prices as a soft launch that may change.

## Sources

1. [unclecode/crawl4ai repository](https://github.com/unclecode/crawl4ai) — GitHub (read 2026-10-22)
2. [Crawl4AI parameters](https://docs.crawl4ai.com/api/parameters/) — Crawl4AI docs (read 2026-10-22)
3. [Crawl4AI self-hosting](https://docs.crawl4ai.com/core/self-hosting/) — Crawl4AI docs (read 2026-10-22)
4. [crawl4ai on PyPI](https://pypi.org/project/crawl4ai/) — PyPI (read 2026-10-22)
5. [Crawl4AI Cloud](https://crawl4ai.com/) — Crawl4AI (read 2026-10-22)
6. [microsoft/markitdown repository](https://github.com/microsoft/markitdown) — GitHub (read 2026-10-22)
7. [markitdown on PyPI](https://pypi.org/project/markitdown/) — PyPI (read 2026-10-22)
8. [adbar/trafilatura repository](https://github.com/adbar/trafilatura) — GitHub (read 2026-10-22)
9. [mozilla/readability repository](https://github.com/mozilla/readability) — GitHub (read 2026-10-22)
10. [firecrawl/firecrawl repository](https://github.com/firecrawl/firecrawl) — GitHub (read 2026-10-22)
11. [jina-ai/reader repository](https://github.com/jina-ai/reader) — GitHub (read 2026-10-22)

---

WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto.