# Web reader API glossary

> Glossary of 36 terms for reading web pages with an API: Markdown, main content, JavaScript rendering, robots.txt, Content Signals, RAG, MCP, credits and more.

- URL: https://webreaderapi.com/glossary/
- Last updated: 2026-10-22
- Publisher: WebReaderAPI (https://webreaderapi.com)

The words you meet when you turn web pages into text for language models, from the HTML side to the AI side, with links to where each one matters in the docs.

**Quick answer — What do the main web reader API terms mean?**

A web reader API fetches a page, keeps its **main content** and returns it as **Markdown** a model can read. **WebReaderAPI** adds **JavaScript rendering** when a page needs a browser, follows **robots.txt** and **Content Signals**, and bills one **credit** per page, from $0.07/1K pages as of October 2026. The 36 terms below explain the rest.

## From the web page side

**HTML** — The markup language of web pages. Raw HTML mixes the text you want with tags, scripts and styling, which is why it reads poorly for a language model.

**DOM** — The Document Object Model: the tree of elements a browser builds from HTML and then changes with JavaScript. A rendered page is read from the DOM, not from the raw HTML.

**JavaScript rendering** — Running a page's scripts in a browser so that content loaded after the first response appears. WebReaderAPI does it when needed (`render=auto`), for two credits instead of one.

**Headless browser** — A real browser engine that runs without a screen, driven by software. It is how JavaScript-heavy pages are rendered on a server.

**Main content extraction** — Finding the part of a page a reader came for, such as the article or the documentation, and leaving out everything around it. In our API this is `main_content=true`, the default.

**Readability** — Both a quality of text and the name of a well-known family of algorithms that score page blocks to find the main content, as reading modes in browsers do.

**Boilerplate** — Text repeated on every page of a site: menus, cookie notices, sidebars, footers, share buttons. Removing it is most of the work of a web reader.

**Markdown** — A plain-text format where `#` marks a heading, `-` a list item and `[text](url)` a link. Models read it well and it uses far fewer tokens than HTML.

**Metadata** — Facts about a page rather than its text: title, description, author, publication date, language. WebReaderAPI returns them next to the Markdown when the page states them.

**PDF extraction** — Turning the text of a PDF file into Markdown. PDF files up to 20 MB and 100 pages are converted to Markdown, at 1 credit per started block of 10 pages.

**Sitemap** — An XML file where a site lists its pages. A crawl job starts from it when it exists, which finds pages faster and with fewer requests.

**Paywall** — A barrier that shows content only to subscribers. WebReaderAPI never reads behind one.

## From the crawler side

**Crawler** — A program that fetches web pages for a purpose. Ours fetches only the pages customers ask for, and the [bot page](https://webreaderapi.com/bot/) describes it.

**User agent** — The name a client sends with each request. Ours is `WebReaderAPI/1.0 (+https://webreaderapi.com/bot/)`, so site owners always know who is asking.

**robots.txt** — A file at the root of a site that tells crawlers which paths they may fetch, standardised as RFC 9309. We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged.

**Crawl-delay** — A robots.txt line asking crawlers to wait a number of seconds between requests. Our crawl jobs honour it.

**Content Signals** — A policy for robots.txt lines such as `Content-Signal: search=yes, ai-input=no`, which state how a site's content may be used, including by AI.

**ai-input** — The Content Signals setting that covers using a page as input for an AI answer. If a site declares Content-Signal: ai-input=no in its robots.txt, we do not read its pages for you.

**llms.txt** — A Markdown file some sites publish at `/llms.txt` to give language models a short guide to their content and their key pages.

**Crawl** — Reading many pages of one site by following its links and sitemap. Crawl one website per job, up to 10,000 pages, following its sitemap, its robots.txt and its Crawl-delay.

**Batch job** — A list of URLs sent in one request and processed in the background. Up to 100 URLs per batch job.

**HTTP 403** — The status a server sends to refuse a request. When a site answers 403 to our crawler, the API returns `target_refused` and charges nothing.

**HTTP 429** — The status for too many requests. From a site, it means slow down; from our API (`concurrency_limit`), it means your account has reached its limit of simultaneous reads.

**Exponential backoff** — Waiting longer after each failed attempt (for example 1, 2, then 4 seconds) before retrying, so that a busy server is not flooded.

## From the AI side

**Token** — The unit a language model reads and is billed by, roughly a short word or part of one. Clean Markdown uses far fewer tokens than the same page in HTML.

**Context window** — The amount of text a model can take in at once. The `max_chars` parameter keeps each page within the budget you set.

**RAG** — Retrieval-augmented generation: the model answers from documents fetched for the question. A web reader supplies those documents from live pages.

**Chunking** — Splitting a long document into passages before indexing it. Markdown headings give natural split points.

**Embedding** — A list of numbers that represents the meaning of a passage, used to find passages similar to a question.

**Vector store** — A database that stores embeddings and returns the closest passages to a query, the usual home of chunked web pages in a RAG system.

**Tool calling** — A model's ability to ask the application to run a function, such as reading a URL, and to use the result. See [OpenAI function calling](https://webreaderapi.com/docs/openai/).

**MCP** — The Model Context Protocol, a standard way to plug tools into AI clients such as Claude Desktop or Cursor. Our [MCP page](https://webreaderapi.com/docs/mcp/) shows a fetch tool built on the API.

## From the billing side

**Credit** — The unit of use of the API. 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged.

**Concurrency** — How many reads your account can run at the same moment. Every account has a limit on how many pages it reads at once; your dashboard shows it, and support can raise it.

**KYC** — Know your customer: identity checks some services require, such as an ID document or a selfie. WebReaderAPI asks for none.

**Stablecoin** — A cryptocurrency that tracks a currency such as the US dollar. USDT and USDC are the two accepted for top-ups; the full list of coins is BTC, USDT, USDC, ETH, SOL, LTC.

---

WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto.