Quick answer
How do I load web pages into LangChain or LlamaIndex as clean Markdown?
Call WebReaderAPI's /read endpoint and wrap each answer in a Document: the Markdown as content, final_url and the title as metadata. For agents, expose the same call as a read_url tool. Pages cost 1 credit each, 2 with a browser, from $0.07/1K pages as of October 2026. No package of ours is needed. No KYC; pay in crypto.
Why Markdown documents rather than raw HTML?
Because retrieval works on text, and Markdown keeps the structure that matters while dropping the rest. Headings survive, so a splitter can cut on topic changes. Lists and tables survive, so steps and specifications stay together. Navigation, cookie banners and ads are gone, so they do not pollute your embeddings or waste tokens.
The metadata comes from the same call: final_url as the source to cite, title, published_at and language for filtering.
import os
import requests
API = "https://webreaderapi.com/api/v1/read"
HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}
class PageError(Exception):
pass
def read_page(url, **options):
resp = requests.get(API, headers=HEADERS, params={"url": url, **options}, timeout=75)
body = resp.json()
if resp.status_code != 200:
raise PageError(body["error"]["code"])
return body
How do I build a LangChain document loader?
Turn each page into a langchain_core.documents.Document, then split it by headings with MarkdownHeaderTextSplitter from the langchain-text-splitters package. Each chunk keeps the heading path in its metadata, and you add the source URL so every answer can point back to its page.
Skipped pages, such as those refused by robots.txt, raise PageError with the code; log them and continue.
from langchain_core.documents import Document
from langchain_text_splitters import MarkdownHeaderTextSplitter
from reader import PageError, read_page
splitter = MarkdownHeaderTextSplitter(headers_to_split_on=[("#", "h1"), ("##", "h2"), ("###", "h3")])
def load_pages(urls):
chunks = []
for url in urls:
try:
page = read_page(url)
except PageError as exc:
print("skipped", url, exc)
continue
for part in splitter.split_text(page["markdown"]):
part.metadata.update({"source": page["final_url"], "title": page.get("title", "")})
chunks.append(part)
return chunks
docs = load_pages(["https://example.com/docs/install", "https://example.com/docs/configure"])
print(len(docs), "chunks", isinstance(docs[0], Document) if docs else "")
How do I give a LangChain agent a read_url tool?
Decorate a function with @tool from langchain_core.tools. The docstring becomes the tool description the model reads, so say what the tool returns and when to use it. Return plain text: a title line, the source and the Markdown, cut with max_chars to suit your model.
Bind the tool to any chat model that supports tool calling, or pass it to your agent constructor. The model then calls read_url with a URL from the conversation or from a search result.
import os
from langchain.chat_models import init_chat_model
from langchain_core.tools import tool
from reader import PageError, read_page
@tool
def read_url(url: str, max_chars: int = 20000) -> str:
"""Read a public web page and return its main content as Markdown, with its title and source URL."""
try:
page = read_page(url, max_chars=max_chars)
except PageError as exc:
return "Could not read the page: " + str(exc)
return "# " + (page.get("title") or url) + "\nSource: " + page["final_url"] + "\n\n" + page["markdown"]
model = init_chat_model(os.environ.get("CHAT_MODEL", "openai:gpt-4.1-mini")).bind_tools([read_url])
reply = model.invoke("Summarise https://example.com/blog/first-post in three bullet points.")
print(reply.tool_calls)
How do I do the same in LlamaIndex?
LlamaIndex uses its own Document class with text and metadata, and FunctionTool.from_defaults to turn a function into a tool. The helper stays the same. Build an index from the documents with VectorStoreIndex.from_documents, which uses whatever embedding model your LlamaIndex settings name.
import os
import time
import requests
from llama_index.core import Document, VectorStoreIndex
from llama_index.core.tools import FunctionTool
from reader import HEADERS, PageError, read_page
def crawl_pages(start_url, limit=200):
job = requests.post("https://webreaderapi.com/api/v1/crawl", headers=HEADERS, json={"url": start_url, "limit": limit}, timeout=30).json()
pages, cursor = [], None
while True:
params = {"cursor": cursor} if cursor else {}
res = requests.get("https://webreaderapi.com/api/v1/jobs/" + job["job_id"], headers=HEADERS, params=params, timeout=30).json()
pages += [r for r in res["results"] if "error" not in r]
cursor = res.get("next_cursor")
if not cursor and res["status"] == "done":
return pages
if not cursor:
time.sleep(5)
pages = crawl_pages("https://docs.example.com/")
docs = [Document(text=p["markdown"], metadata={"source": p["final_url"], "title": p.get("title", "")}) for p in pages]
index = VectorStoreIndex.from_documents(docs)
print(index.as_query_engine().query("How do I install the product?"))
def read_url(url: str, max_chars: int = 20000) -> str:
"""Read a public web page and return its main content as Markdown."""
try:
return read_page(url, max_chars=max_chars)["markdown"]
except PageError as exc:
return "Could not read the page: " + str(exc)
read_tool = FunctionTool.from_defaults(fn=read_url, name="read_url")
print(os.environ.get("CHAT_MODEL", "set CHAT_MODEL for your agent"), read_tool.metadata.name)
How large a crawl does a small RAG index need?
Less than you might think. A product's documentation is often a few hundred pages, and a crawl with include_paths set to the docs section reads only those. At 1 credit a page, a 300-page docs site costs about 300 credits; 10,000 pages ($3.75) refreshes it many times.
Re-crawl on a schedule that matches how often the source changes, and compare final_url and word_count with the previous run to re-embed only the pages that changed. Prepaid credits never expire: unused pages stay on your account until you use them.
LangChain and LlamaIndex: common questions
Is there a WebReaderAPI package for LangChain or LlamaIndex?
No. The helper and tool on this page are the whole integration, so you can adapt them freely. WebReaderAPI is a plain HTTPS API that any Python HTTP client can call.
Which LangChain version does this need?
Any recent one with langchain_core, langchain-text-splitters and init_chat_model. The tool decorator and the Document class have been stable across recent releases.
How do I avoid re-reading pages that did not change?
Store final_url, word_count and the date of each read. Re-read on a schedule, and re-embed a page only when its word count or content hash changes. Each re-read costs a credit, so pick a rhythm that matches the source.
What if some pages are refused?
They raise PageError with a code such as blocked_by_robots, and they cost nothing. If a site refuses our request, we return its answer: we never solve CAPTCHAs, log in to accounts, or try to get around a block or a paywall.