Test
10,000 pages
$3.75
$0.38/1K pages
- Try it on your real URLs
- Same API as the biggest pack
- Credits never expire
Quick answer
How do I convert my whole website to Markdown?
Crawl your site, read each page's main content as Markdown, and write one .md file per page with its title, URL and date in front matter. WebReaderAPI crawls up to 10,000 pages per job, follows your sitemap and robots.txt, and charges one credit per page from $0.07/1K pages as of October 2026. A Python script below writes the files. No KYC; pay in crypto.
Website export with WebReaderAPI
| Endpoint | POST https://webreaderapi.com/api/v1/crawl, results from GET https://webreaderapi.com/api/v1/jobs/{job_id} |
|---|---|
| Scope | Crawl one website per job, up to 10,000 pages, following its sitemap, its robots.txt and its Crawl-delay |
| Per page | Markdown of the main content, plus title, description, author, date and language |
| Whole pages | main_content=false keeps navigation and footers when you need them |
| Your robots.txt | Must allow WebReaderAPI for the pages you export |
| Price | $3.75 for 10,000 pages; credits never expire |
When is a Markdown export useful?
Whenever content has to leave the system that renders it. Common cases:
- Migrations. Moving from a CMS to a static site generator or a documentation platform that stores pages as Markdown.
- Backups. A readable archive of your published pages that does not depend on any database or theme.
- Your own assistant. Feeding your site's text into a help bot or an internal search, with each chunk linked to its page.
- Audits. Reading all your pages as plain text to find outdated claims, broken wording or duplicate content.
In each case you want the words and the structure, not the theme around them.
import os
import pathlib
import time
from urllib.parse import urlparse
import requests
API = "https://webreaderapi.com/api/v1"
HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}
SITE = "https://www.example.com/"
OUT = pathlib.Path("export")
def target_path(url):
path = urlparse(url).path.strip("/") or "index"
return OUT / (path + ".md")
def front_matter(page):
fields = {"title": page.get("title", ""), "source": page["final_url"], "date": page.get("published_at") or ""}
lines = ["---"] + [key + ": " + repr(value) for key, value in fields.items()] + ["---", ""]
return "\n".join(lines)
job = requests.post(API + "/crawl", headers=HEADERS, json={"url": SITE, "limit": 2000, "max_depth": 5}, timeout=60).json()
cursor, delay, written = None, 5, 0
while True:
params = {"cursor": cursor} if cursor else {}
batch = requests.get(API + "/jobs/" + job["job_id"], headers=HEADERS, params=params, timeout=60).json()
for page in batch.get("results", []):
if "markdown" not in page:
continue
dest = target_path(page["final_url"])
dest.parent.mkdir(parents=True, exist_ok=True)
dest.write_text(front_matter(page) + page["markdown"] + "\n", encoding="utf-8")
written += 1
cursor = batch.get("next_cursor")
if batch["status"] == "done" and not cursor:
break
if not cursor:
time.sleep(delay)
delay = min(delay * 2, 60)
print(written, "files in", OUT)
How do I keep the folder structure and the URLs?
Mirror the URL path in the file path, as the script does: /guides/setup/ becomes export/guides/setup.md. Keep the original URL in the front matter as source, so you can set up redirects after the migration and check that nothing was lost.
Links inside the Markdown still point to your current URLs. After the move, replace the domain or the path prefix in one pass, then test a sample of pages. Tables and code blocks stay as Markdown, so most static generators render them without changes.
What if my own robots.txt blocks crawlers?
Allow our token for the time of the export. WebReaderAPI reads only what robots.txt permits for WebReaderAPI or for all robots, including on your own site. Add a group such as User-agent: WebReaderAPI with Allow: /, run the crawl, then remove it if you like.
Our crawler page shows the exact lines and the User-Agent your server logs will show. Pages that stay disallowed are skipped and not charged.
How much does an export cost?
One credit per page, two for pages that need a browser to show their text. A 2,000-page site rendered on the server costs 2,000 credits once; the same site as a JavaScript app can cost up to 4,000. Both fit in 10,000 pages for $3.75, as of October 2026.
A site of 50,000 pages needs several jobs, split by section with include_paths, and fits in 100,000 pages ($21.75). Credits that remain after the export stay on your account for later refreshes.
Which pack covers my export?
Count your pages, double the count if the site needs JavaScript to render, and pick the pack above it. Credits never expire.
Starter
50,000 pages
$14.25
$0.28/1K pages
- An AI agent or a RAG prototype
- Crawl a documentation site
- Credits never expire
Build
100,000 pages
$21.75
$0.22/1K pages
- Production agents and RAG pipelines
- Batch and crawl jobs
- Credits never expire
In every plan: Main-content extraction · JavaScript rendering when needed · Failed and refused pages not charged · Credits never expire
Website to Markdown: common questions
Can I export the whole page, including menus and footer?
Yes. Set main_content=false in the crawl options and each file holds the full page converted to Markdown. Most migrations prefer the main content only, because the new theme supplies its own navigation.
Are images downloaded?
No. With include_images=true, image links stay in the Markdown with their alt text; download the files themselves with your own script if the migration needs them.
Can I export a site I do not own?
Only public pages that its robots.txt allows, and only for uses its terms and the law permit. For a migration or a backup, export your own sites or sites whose owner agreed.
How large a site can one job handle?
Up to 10,000 pages per crawl job, within one domain. Split larger sites by section with include_paths and run several jobs.