Web reader API glossary

The markup language of web pages. Raw HTML mixes the text you want with tags, scripts and styling, which is why it reads poorly for a language model.

The Document Object Model: the tree of elements a browser builds from HTML and then changes with JavaScript. A rendered page is read from the DOM, not from the raw HTML.

Running a page's scripts in a browser so that content loaded after the first response appears. WebReaderAPI does it when needed (render=auto), for two credits instead of one.

A real browser engine that runs without a screen, driven by software. It is how JavaScript-heavy pages are rendered on a server.

Finding the part of a page a reader came for, such as the article or the documentation, and leaving out everything around it. In our API this is main_content=true, the default.

Both a quality of text and the name of a well-known family of algorithms that score page blocks to find the main content, as reading modes in browsers do.

Text repeated on every page of a site: menus, cookie notices, sidebars, footers, share buttons. Removing it is most of the work of a web reader.

A plain-text format where # marks a heading, - a list item and [text](url) a link. Models read it well and it uses far fewer tokens than HTML.

Facts about a page rather than its text: title, description, author, publication date, language. WebReaderAPI returns them next to the Markdown when the page states them.

Turning the text of a PDF file into Markdown. PDF files up to 20 MB and 100 pages are converted to Markdown, at 1 credit per started block of 10 pages.

An XML file where a site lists its pages. A crawl job starts from it when it exists, which finds pages faster and with fewer requests.

A barrier that shows content only to subscribers. WebReaderAPI never reads behind one.

A program that fetches web pages for a purpose. Ours fetches only the pages customers ask for, and the bot page describes it.

The name a client sends with each request. Ours is WebReaderAPI/1.0 (+https://webreaderapi.com/bot/), so site owners always know who is asking.

A file at the root of a site that tells crawlers which paths they may fetch, standardised as RFC 9309. We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged.

A robots.txt line asking crawlers to wait a number of seconds between requests. Our crawl jobs honour it.

A policy for robots.txt lines such as Content-Signal: search=yes, ai-input=no, which state how a site's content may be used, including by AI.

The Content Signals setting that covers using a page as input for an AI answer. If a site declares Content-Signal: ai-input=no in its robots.txt, we do not read its pages for you.

A Markdown file some sites publish at /llms.txt to give language models a short guide to their content and their key pages.

Reading many pages of one site by following its links and sitemap. Crawl one website per job, up to 10,000 pages, following its sitemap, its robots.txt and its Crawl-delay.

A list of URLs sent in one request and processed in the background. Up to 100 URLs per batch job.

The status a server sends to refuse a request. When a site answers 403 to our crawler, the API returns target_refused and charges nothing.

The status for too many requests. From a site, it means slow down; from our API (concurrency_limit), it means your account has reached its limit of simultaneous reads.

Waiting longer after each failed attempt (for example 1, 2, then 4 seconds) before retrying, so that a busy server is not flooded.

The unit a language model reads and is billed by, roughly a short word or part of one. Clean Markdown uses far fewer tokens than the same page in HTML.

The amount of text a model can take in at once. The max_chars parameter keeps each page within the budget you set.

Retrieval-augmented generation: the model answers from documents fetched for the question. A web reader supplies those documents from live pages.

Splitting a long document into passages before indexing it. Markdown headings give natural split points.

A list of numbers that represents the meaning of a passage, used to find passages similar to a question.

A database that stores embeddings and returns the closest passages to a query, the usual home of chunked web pages in a RAG system.

A model's ability to ask the application to run a function, such as reading a URL, and to use the result. See OpenAI function calling.

The Model Context Protocol, a standard way to plug tools into AI clients such as Claude Desktop or Cursor. Our MCP page shows a fetch tool built on the API.

The unit of use of the API. 1 credit = 1 page read without a browser; 2 credits when the page needs a browser to render JavaScript. A request that fails, or that a site's robots.txt refuses, is not charged.

How many reads your account can run at the same moment. Every account has a limit on how many pages it reads at once; your dashboard shows it, and support can raise it.

Know your customer: identity checks some services require, such as an ID document or a selfie. WebReaderAPI asks for none.

A cryptocurrency that tracks a currency such as the US dollar. USDT and USDC are the two accepted for top-ups; the full list of coins is BTC, USDT, USDC, ETH, SOL, LTC.