# Convert your website to Markdown: crawl it and export every page

> Convert your website to Markdown files: crawl your own site and export each page with front matter, for a migration, a backup or your own assistant. No KYC.

- URL: https://webreaderapi.com/use-cases/website-to-markdown/
- Last updated: 2026-10-22
- Publisher: WebReaderAPI (https://webreaderapi.com)

Moving to a static site generator, a docs platform or a knowledge base often starts with one question: how do I get my pages out as clean Markdown? A crawl does it in one job, with titles, dates and URLs kept in front matter.

**Quick answer — How do I convert my whole website to Markdown?**

Crawl your site, read each page's main content as Markdown, and write one `.md` file per page with its title, URL and date in front matter. **WebReaderAPI** crawls up to 10,000 pages per job, follows your sitemap and robots.txt, and charges one credit per page from $0.07/1K pages as of October 2026. A Python script below writes the files. No KYC; pay in crypto.

## Website export with WebReaderAPI

| Attribute | Value |
| --- | --- |
| Endpoint | `POST https://webreaderapi.com/api/v1/crawl`, results from `GET https://webreaderapi.com/api/v1/jobs/{job_id}` |
| Scope | Crawl one website per job, up to 10,000 pages, following its sitemap, its robots.txt and its Crawl-delay |
| Per page | Markdown of the main content, plus title, description, author, date and language |
| Whole pages | `main_content=false` keeps navigation and footers when you need them |
| Your robots.txt | Must allow `WebReaderAPI` for the pages you export |
| Price | $3.75 for 10,000 pages; credits never expire |

## When is a Markdown export useful?

Whenever content has to leave the system that renders it. Common cases:

- **Migrations.** Moving from a CMS to a static site generator or a documentation platform that stores pages as Markdown.
- **Backups.** A readable archive of your published pages that does not depend on any database or theme.
- **Your own assistant.** Feeding your site's text into a help bot or an internal search, with each chunk linked to its page.
- **Audits.** Reading all your pages as plain text to find outdated claims, broken wording or duplicate content.

In each case you want the words and the structure, not the theme around them.

*Python: crawl your site into Markdown files with front matter*

```python
import os
import pathlib
import time
from urllib.parse import urlparse

import requests

API = "https://webreaderapi.com/api/v1"
HEADERS = {"Authorization": "Bearer " + os.environ["WEBREADER_API_KEY"]}
SITE = "https://www.example.com/"
OUT = pathlib.Path("export")


def target_path(url):
    path = urlparse(url).path.strip("/") or "index"
    return OUT / (path + ".md")


def front_matter(page):
    fields = {"title": page.get("title", ""), "source": page["final_url"], "date": page.get("published_at") or ""}
    lines = ["---"] + [key + ": " + repr(value) for key, value in fields.items()] + ["---", ""]
    return "\n".join(lines)


job = requests.post(API + "/crawl", headers=HEADERS, json={"url": SITE, "limit": 2000, "max_depth": 5}, timeout=60).json()
cursor, delay, written = None, 5, 0
while True:
    params = {"cursor": cursor} if cursor else {}
    batch = requests.get(API + "/jobs/" + job["job_id"], headers=HEADERS, params=params, timeout=60).json()
    for page in batch.get("results", []):
        if "markdown" not in page:
            continue
        dest = target_path(page["final_url"])
        dest.parent.mkdir(parents=True, exist_ok=True)
        dest.write_text(front_matter(page) + page["markdown"] + "\n", encoding="utf-8")
        written += 1
    cursor = batch.get("next_cursor")
    if batch["status"] == "done" and not cursor:
        break
    if not cursor:
        time.sleep(delay)
        delay = min(delay * 2, 60)
print(written, "files in", OUT)

```

## How do I keep the folder structure and the URLs?

Mirror the URL path in the file path, as the script does: `/guides/setup/` becomes `export/guides/setup.md`. Keep the original URL in the front matter as `source`, so you can set up redirects after the migration and check that nothing was lost.

Links inside the Markdown still point to your current URLs. After the move, replace the domain or the path prefix in one pass, then test a sample of pages. Tables and code blocks stay as Markdown, so most static generators render them without changes.

## What if my own robots.txt blocks crawlers?

Allow our token for the time of the export. WebReaderAPI reads only what robots.txt permits for `WebReaderAPI` or for all robots, including on your own site. Add a group such as `User-agent: WebReaderAPI` with `Allow: /`, run the crawl, then remove it if you like.

Our [crawler page](https://webreaderapi.com/bot/) shows the exact lines and the User-Agent your server logs will show. Pages that stay disallowed are skipped and not charged.

## How much does an export cost?

One credit per page, two for pages that need a browser to show their text. A 2,000-page site rendered on the server costs 2,000 credits once; the same site as a JavaScript app can cost up to 4,000. Both fit in 10,000 pages for $3.75, as of October 2026.

A site of 50,000 pages needs several jobs, split by section with `include_paths`, and fits in 100,000 pages ($21.75). Credits that remain after the export stay on your account for later refreshes.

## Which pack covers my export?

Count your pages, double the count if the site needs JavaScript to render, and pick the pack above it. Credits never expire.

| Plan | Quantity | Price | Price per unit |
| --- | --- | --- | --- |
| Test | 10,000 pages | $3.75 | $0.38/1K pages |
| Starter | 50,000 pages | $14.25 | $0.28/1K pages |
| Build | 100,000 pages | $21.75 | $0.22/1K pages |

In every plan: Main-content extraction · JavaScript rendering when needed · Failed and refused pages not charged · Credits never expire

**Before you pay**

- Minimum top-up $25. Top up once in crypto, then buy any plan from your balance.
- Your balance has no end date and can buy page credits at any time. If a pack costs less than the minimum top-up, the amount you added only to reach that minimum can be sent back on request within 30 days of the payment, in USDT or USDC, as long as you have not spent it. Any other balance is not refunded unless the law says otherwise.
- Credited once your payment is checked on the blockchain. Typical network confirmation time: BTC ≈ 20 min · USDT TRC-20 ≈ 1 min · USDT ERC-20 ≈ 5 min · USDC ERC-20 ≈ 5 min · ETH ≈ 5 min · SOL ≈ 1 min · LTC ≈ 15 min

## Website to Markdown: common questions

### Can I export the whole page, including menus and footer?

Yes. Set `main_content=false` in the crawl options and each file holds the full page converted to Markdown. Most migrations prefer the main content only, because the new theme supplies its own navigation.

### Are images downloaded?

No. With `include_images=true`, image links stay in the Markdown with their alt text; download the files themselves with your own script if the migration needs them.

### Can I export a site I do not own?

Only public pages that its robots.txt allows, and only for uses its terms and the law permit. For a migration or a backup, export your own sites or sites whose owner agreed.

### How large a site can one job handle?

Up to 10,000 pages per crawl job, within one domain. Split larger sites by section with `include_paths` and run several jobs.

---

WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto.