Quick answer
How do I convert a web page to Markdown in Node.js?
Use the built-in fetch of Node.js 18+: call https://webreaderapi.com/api/v1/read?url=… with your key in the Authorization header and read markdown from the JSON answer. WebReaderAPI returns the page's main content with its title, author and date. One page costs 1 credit, 2 with a browser, from $0.07/1K pages as of October 2026. No KYC; pay in crypto.
What do I need?
Node.js 18 or later, for the global fetch, and an API key in the WEBREADER_API_KEY environment variable. The examples are ES modules: save them with the .mjs extension, or set "type": "module" in your package.json, so that top-level await works.
The JSDoc comments describe the response shape, so editors such as VS Code give you completion without a TypeScript build step.
/**
* @typedef {Object} Page
* @property {string} url
* @property {string} final_url
* @property {number} status_code
* @property {boolean} rendered
* @property {number} credits_used
* @property {string} [title]
* @property {string} [author]
* @property {string} [published_at]
* @property {string} [language]
* @property {string} [markdown]
* @property {number} word_count
*/
/** @returns {Promise<Page>} */
export async function readPage(url, options = {}) {
const params = new URLSearchParams({ url, ...options });
const res = await fetch(`https://webreaderapi.com/api/v1/read?${params}`, {
headers: { Authorization: `Bearer ${process.env.WEBREADER_API_KEY}` },
signal: AbortSignal.timeout(75_000),
});
const body = await res.json();
if (!res.ok) throw Object.assign(new Error(body.error.message), { code: body.error.code });
return body;
}
const page = await readPage('https://example.com/docs/getting-started', { max_chars: '20000' });
console.log(page.title, page.word_count, 'words,', page.credits_used, 'credit(s)');
How do I read a list of URLs without hitting the limit?
Run a fixed number of workers that take URLs from a shared queue. It is simpler than Promise.all over the whole list, which would start every request at once and hit 429 concurrency_limit. Every account has a limit on how many pages it reads at once; your dashboard shows it, and support can raise it.
The pool below keeps errors next to the results, so one refused page does not stop the run. Refusals such as blocked_by_robots are final and free; count them and move on.
import { readPage } from './read.mjs';
export async function readAll(urls, workers = 8) {
const queue = [...urls];
const results = [];
async function worker() {
while (queue.length) {
const url = queue.shift();
try {
const page = await readPage(url, { max_chars: '50000' });
results.push({ url, page });
} catch (err) {
results.push({ url, error: err.code ?? err.name });
}
}
}
await Promise.all(Array.from({ length: workers }, worker));
return results;
}
const urls = Array.from({ length: 30 }, (_, i) => `https://example.com/blog/post-${i + 1}`);
for (const r of await readAll(urls)) console.log(r.url, '->', r.error ?? r.page.title);
How do I follow a crawl job in Node.js?
Start the job with POST https://webreaderapi.com/api/v1/crawl, then read https://webreaderapi.com/api/v1/jobs/{job_id} in a loop. Pass cursor each time next_cursor comes back; when there is no cursor and the status is done, the job is finished. Wait between empty polls and double the wait up to 30 seconds, since a crawl that respects a site's Crawl-delay can be slow.
The batch and crawl page describes how the crawl chooses its pages and how limit caps what it reads.
const headers = {
Authorization: `Bearer ${process.env.WEBREADER_API_KEY}`,
'Content-Type': 'application/json',
};
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
const start = await fetch('https://webreaderapi.com/api/v1/crawl', {
method: 'POST',
headers,
body: JSON.stringify({ url: 'https://docs.example.com/', limit: 300, include_paths: ['/docs/*'] }),
});
const { job_id } = await start.json();
const pages = [];
let cursor = '';
let wait = 2000;
for (;;) {
const url = `https://webreaderapi.com/api/v1/jobs/${job_id}${cursor ? `?cursor=${encodeURIComponent(cursor)}` : ''}`;
const job = await (await fetch(url, { headers })).json();
pages.push(...job.results.filter((r) => !r.error));
if (job.next_cursor) {
cursor = job.next_cursor;
wait = 2000;
continue;
}
if (job.status === 'done') break;
await sleep(wait);
wait = Math.min(wait * 2, 30000);
}
console.log(pages.length, 'pages read, job', job_id);
How do I pass the result to a model?
Send the Markdown, not the whole JSON. The markdown field is what a model should read; the metadata is better kept for citations. Prefix it with the title and final_url, so the model can name its source, and keep max_chars at a size your model's context can hold, such as 20,000 characters per page.
The OpenAI function-calling page shows the full tool loop in Python; the same readPage function above can serve as the tool body in a Node.js agent.
Node.js: common questions
Does this work in Deno, Bun or the browser?
Deno and Bun support the same fetch and ES modules. Do not call the API from a browser page: the key would be visible to every visitor. Call it from your server instead.
Is there an npm package?
No. The functions on this page are the whole integration. WebReaderAPI is plain HTTPS and JSON, so you keep control of retries, timeouts and logging.
How do I handle 429 concurrency_limit?
Lower the number of workers, or retry after a short wait that doubles each time. A 429 costs nothing, and the errors page has a ready backoff function.
What does reading 10,000 pages cost?
About 10,000 credits for plain pages, more if some need a browser. 10,000 pages cost $3.75; Prepaid credits never expire: unused pages stay on your account until you use them.