# Web pages for RAG: turn sites into clean, chunkable Markdown

> Feed RAG pipelines with web pages: crawl docs and blogs into clean Markdown, chunk by heading, keep source URLs. From $0.07/1K pages, no KYC, crypto.

- URL: https://webreaderapi.com/use-cases/rag/
- Last updated: 2026-10-22
- Publisher: WebReaderAPI (https://webreaderapi.com)

Retrieval works on what you put in the index. Clean Markdown with headings, metadata and a source URL gives better chunks, fewer wasted tokens and answers that can cite their page.

**Quick answer — How do I ingest web pages into a RAG pipeline?**

Read each page as Markdown of its main content, split it along its headings, attach the source URL and title to every chunk, then embed and store. **WebReaderAPI** handles the reading: one request per page, or one crawl job for a whole site of up to 10,000 pages, at one credit per page from $0.07/1K pages as of October 2026. robots.txt is respected. No KYC; pay in crypto.

## RAG ingestion with WebReaderAPI

| Attribute | Value |
| --- | --- |
| Whole site | `POST https://webreaderapi.com/api/v1/crawl`: Crawl one website per job, up to 10,000 pages, following its sitemap, its robots.txt and its Crawl-delay |
| List of URLs | `POST https://webreaderapi.com/api/v1/batch`: Up to 100 URLs per batch job |
| Per page | Markdown of the main content, title, author, date, language, word count |
| PDFs | PDF files up to 20 MB and 100 pages are converted to Markdown, at 1 credit per started block of 10 pages |
| Billing | 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged |
| Price | $14.25 for 50,000 pages; credits never expire |

## Why does clean Markdown give better retrieval?

Chunks made of boilerplate retrieve badly. When a page is split with its menus, cookie text and footer still in it, many chunks contain the same navigation words, and the vector search returns them for unrelated questions. Main-content extraction removes that noise before you embed.

Headings matter just as much. A chunk that starts with `## Refund policy` carries its topic with it, so both the embedding and the model reading it later know what it is about. Plain text loses that signal; raw HTML hides it under markup.

## What does an ingestion pipeline look like?

Five steps, whatever vector store you use:

1. **Collect URLs.** Crawl the site, or send the list you already have as a batch.
2. **Read as Markdown.** Keep `main_content=true` for articles and docs; use `false` for listing pages.
3. **Chunk by heading.** Split on `#`, `##` and `###`, then cut long sections to your size limit with a small overlap.
4. **Attach metadata.** Store `url`, `title`, `published_at` and the heading path with each chunk, so answers can cite their source.
5. **Refresh.** Re-read pages whose sitemap `lastmod` changed, and delete chunks of pages that now return an error.

*Python: read pages and split them into chunks with their source*

```python
import os
import re

import requests

API = "https://webreaderapi.com/api/v1/read"
HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}
URLS = [
    "https://docs.example.com/getting-started",
    "https://docs.example.com/billing",
]


def chunks_for(url, max_len=2000):
    page = requests.get(API, headers=HEADERS, params={"url": url}, timeout=90).json()
    if "markdown" not in page:
        return []
    out = []
    for section in re.split(r"\n(?=#{1,3} )", page["markdown"]):
        section = section.strip()
        heading = section.splitlines()[0] if section else ""
        for start in range(0, len(section), max_len):
            out.append({
                "text": section[start:start + max_len],
                "url": page.get("final_url", url),
                "title": page.get("title"),
                "heading": heading,
            })
    return out


all_chunks = [c for u in URLS for c in chunks_for(u)]
print(len(all_chunks), "chunks ready to embed")

```

## How much does it cost to build and refresh an index?

Building costs one credit per page, two for pages that need a browser. Refreshing costs only the pages you re-read. Three examples, at the prices of October 2026:

- **A product's help center** of 800 pages, refreshed monthly for changed pages (about 10%): 800 credits once, then about 80 a month. 10,000 pages ($3.75) last for years.
- **Ten documentation sites** of 2,000 pages each, refreshed weekly at 5%: 20,000 credits, then about 4,000 a month. 50,000 pages ($14.25) cover the build and seven months of refreshes.
- **A news archive** of 300,000 articles that never change: 300,000 credits once, which fit in 500,000 pages ($50).

Embedding and storage are billed by your own providers and are not included.

## Can I index sites that do not want to be indexed?

Not through WebReaderAPI. Pages disallowed by robots.txt for our crawler, or for all robots, are skipped and not charged; sites that declare `Content-Signal: ai-input=no` are not read for you. For your own sites or partner sites that block robots, ask the owner to allow `WebReaderAPI` in robots.txt; our [crawler page](https://webreaderapi.com/bot/) shows the lines to add.

This matters for your users too. A RAG assistant that cites sources is more useful when those sources agreed to be read.

## Which pack fits a RAG project?

Count the pages to build the index, add the monthly refresh, and pick the pack that covers a few months. Credits never expire.

| Plan | Quantity | Price | Price per unit |
| --- | --- | --- | --- |
| Test | 10,000 pages | $3.75 | $0.38/1K pages |
| Starter | 50,000 pages | $14.25 | $0.28/1K pages |
| Build | 100,000 pages | $21.75 | $0.22/1K pages |
| Scale | 500,000 pages | $50 | $0.10/1K pages |

In every plan: Main-content extraction · JavaScript rendering when needed · Failed and refused pages not charged · Credits never expire

**Before you pay**

- Minimum top-up $25. Top up once in crypto, then buy any plan from your balance.
- Your balance has no end date and can buy page credits at any time. If a pack costs less than the minimum top-up, the amount you added only to reach that minimum can be sent back on request within 30 days of the payment, in USDT or USDC, as long as you have not spent it. Any other balance is not refunded unless the law says otherwise.
- Credited once your payment is checked on the blockchain. Typical network confirmation time: BTC ≈ 20 min · USDT TRC-20 ≈ 1 min · USDT ERC-20 ≈ 5 min · USDC ERC-20 ≈ 5 min · ETH ≈ 5 min · SOL ≈ 1 min · LTC ≈ 15 min

## RAG ingestion: common questions

### What chunk size should I use?

Split on headings first, then cut sections longer than about 500 to 1,000 tokens with a small overlap. Heading-based chunks keep one topic each, which usually retrieves better than fixed-size windows over the whole page.

### Does WebReaderAPI create embeddings or store vectors?

No. WebReaderAPI reads pages and returns Markdown with metadata. You embed with the model of your choice and store in your own vector database; the [LangChain and LlamaIndex page](https://webreaderapi.com/docs/langchain/) shows both.

### How do I keep the index up to date?

Re-read only the pages that changed. Most sitemaps carry a `lastmod` date per URL; compare it with your last read and send the changed URLs as a batch. Pages that now fail should be removed from the index.

### Can I include PDFs from the same site?

Yes. A crawl reads PDF links on the same domain with the same robots.txt checks. PDF files up to 20 MB and 100 pages are converted to Markdown, at 1 credit per started block of 10 pages.

---

WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto.