No KYC — no ID, no selfie, no documents. Pay in crypto: BTC, USDT, USDC, ETH, SOL, LTC
Language: English
Get an API key

Use case · retrieval-augmented generation

Web pages for RAG: turn sites into clean, chunkable Markdown

Retrieval works on what you put in the index. Clean Markdown with headings, metadata and a source URL gives better chunks, fewer wasted tokens and answers that can cite their page.

Last updated: · 5 min read

Quick answer

How do I ingest web pages into a RAG pipeline?

Read each page as Markdown of its main content, split it along its headings, attach the source URL and title to every chunk, then embed and store. WebReaderAPI handles the reading: one request per page, or one crawl job for a whole site of up to 10,000 pages, at one credit per page from $0.07/1K pages as of October 2026. robots.txt is respected. No KYC; pay in crypto.

RAG ingestion with WebReaderAPI

Whole sitePOST https://webreaderapi.com/api/v1/crawl: Crawl one website per job, up to 10,000 pages, following its sitemap, its robots.txt and its Crawl-delay
List of URLsPOST https://webreaderapi.com/api/v1/batch: Up to 100 URLs per batch job
Per pageMarkdown of the main content, title, author, date, language, word count
PDFsPDF files up to 20 MB and 100 pages are converted to Markdown, at 1 credit per started block of 10 pages
Billing1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged
Price$14.25 for 50,000 pages; credits never expire

Why does clean Markdown give better retrieval?

Chunks made of boilerplate retrieve badly. When a page is split with its menus, cookie text and footer still in it, many chunks contain the same navigation words, and the vector search returns them for unrelated questions. Main-content extraction removes that noise before you embed.

Headings matter just as much. A chunk that starts with ## Refund policy carries its topic with it, so both the embedding and the model reading it later know what it is about. Plain text loses that signal; raw HTML hides it under markup.

What does an ingestion pipeline look like?

Five steps, whatever vector store you use:

  1. Collect URLs. Crawl the site, or send the list you already have as a batch.
  2. Read as Markdown. Keep main_content=true for articles and docs; use false for listing pages.
  3. Chunk by heading. Split on #, ## and ###, then cut long sections to your size limit with a small overlap.
  4. Attach metadata. Store url, title, published_at and the heading path with each chunk, so answers can cite their source.
  5. Refresh. Re-read pages whose sitemap lastmod changed, and delete chunks of pages that now return an error.
Python: read pages and split them into chunks with their source
import os
import re

import requests

API = "https://webreaderapi.com/api/v1/read"
HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}
URLS = [
    "https://docs.example.com/getting-started",
    "https://docs.example.com/billing",
]


def chunks_for(url, max_len=2000):
    page = requests.get(API, headers=HEADERS, params={"url": url}, timeout=90).json()
    if "markdown" not in page:
        return []
    out = []
    for section in re.split(r"\n(?=#{1,3} )", page["markdown"]):
        section = section.strip()
        heading = section.splitlines()[0] if section else ""
        for start in range(0, len(section), max_len):
            out.append({
                "text": section[start:start + max_len],
                "url": page.get("final_url", url),
                "title": page.get("title"),
                "heading": heading,
            })
    return out


all_chunks = [c for u in URLS for c in chunks_for(u)]
print(len(all_chunks), "chunks ready to embed")

How much does it cost to build and refresh an index?

Building costs one credit per page, two for pages that need a browser. Refreshing costs only the pages you re-read. Three examples, at the prices of October 2026:

  • A product's help center of 800 pages, refreshed monthly for changed pages (about 10%): 800 credits once, then about 80 a month. 10,000 pages ($3.75) last for years.
  • Ten documentation sites of 2,000 pages each, refreshed weekly at 5%: 20,000 credits, then about 4,000 a month. 50,000 pages ($14.25) cover the build and seven months of refreshes.
  • A news archive of 300,000 articles that never change: 300,000 credits once, which fit in 500,000 pages ($50).

Embedding and storage are billed by your own providers and are not included.

Can I index sites that do not want to be indexed?

Not through WebReaderAPI. Pages disallowed by robots.txt for our crawler, or for all robots, are skipped and not charged; sites that declare Content-Signal: ai-input=no are not read for you. For your own sites or partner sites that block robots, ask the owner to allow WebReaderAPI in robots.txt; our crawler page shows the lines to add.

This matters for your users too. A RAG assistant that cites sources is more useful when those sources agreed to be read.

Which pack fits a RAG project?

Count the pages to build the index, add the monthly refresh, and pick the pack that covers a few months. Credits never expire.

Test

10,000 pages

$3.75

$0.38/1K pages

  • Try it on your real URLs
  • Same API as the biggest pack
  • Credits never expire
Choose 10,000 pages

Starter

50,000 pages

$14.25

$0.28/1K pages

  • An AI agent or a RAG prototype
  • Crawl a documentation site
  • Credits never expire
Choose 50,000 pages

Build

100,000 pages

$21.75

$0.22/1K pages

  • Production agents and RAG pipelines
  • Batch and crawl jobs
  • Credits never expire
Choose 100,000 pages

Scale

500,000 pages

$50

$0.10/1K pages

  • Large knowledge bases
  • Higher concurrency on request
  • Credits never expire
Choose 500,000 pages

In every plan: Main-content extraction · JavaScript rendering when needed · Failed and refused pages not charged · Credits never expire

RAG ingestion: common questions

What chunk size should I use?

Split on headings first, then cut sections longer than about 500 to 1,000 tokens with a small overlap. Heading-based chunks keep one topic each, which usually retrieves better than fixed-size windows over the whole page.

Does WebReaderAPI create embeddings or store vectors?

No. WebReaderAPI reads pages and returns Markdown with metadata. You embed with the model of your choice and store in your own vector database; the LangChain and LlamaIndex page shows both.

How do I keep the index up to date?

Re-read only the pages that changed. Most sitemaps carry a lastmod date per URL; compare it with your last read and send the changed URLs as a batch. Pages that now fail should be removed from the index.

Can I include PDFs from the same site?

Yes. A crawl reads PDF links on the same domain with the same robots.txt checks. PDF files up to 20 MB and 100 pages are converted to Markdown, at 1 credit per started block of 10 pages.

From $0.07/1K pages · no KYC

Get an API key