Why AI agents get 403 errors when fetching web pages

What changed on the web in 2025 and 2026?

Large infrastructure providers turned AI blocking into a default. Three dated changes explain most new 403s:

• 1 July 2025: Cloudflare said it now blocks AI crawlers by default, asking every new domain at sign-up whether to allow them. The same day it launched pay per crawl in private beta, which answers with HTTP 402 and a price.
• 1 July 2026: Cloudflare introduced three categories of AI traffic, Search, Agent and Training. "Agent" explicitly includes chat fetchers such as ChatGPT-User.
• 15 September 2026: for new domains, Cloudflare's presets block Training and Agent traffic on pages that display ads; Search stays allowed.

An agent that fetches a page for a user falls in the Agent category. On ad-supported sites onboarded after September 2026, expect refusals.

How do I tell a block from a bug?

Read the status code and the body before changing anything.

Only the 429 and the empty JavaScript page are technical problems you can solve on your side. The others are decisions made by the site owner.

What can an agent builder legitimately do?

Work with the site, not around it. These steps fix many failures and keep your agent on the right side of site owners:

1. Use the site's own channel. Many sites offer an API, an RSS or Atom feed, or a sitemap with the content you need.
2. Ask for access. Site owners can allow a named bot in robots.txt or in their CDN settings. A clear User-Agent and a contact page make that easy.
3. Render pages that are allowed but empty. For JavaScript-only pages, use a reader that runs a headless browser.
4. Slow down and cache. Read the same page once, store it, and respect Crawl-delay and 429 responses.
5. Pay where the site sells access. A 402 is an offer; take it through the site's channel or leave the page out.

Each of these leaves the site owner in control, which is also what keeps your agent useful over time.

What should I never do to get past a refusal?

Never disguise the agent or defeat a protection. Faking a browser's User-Agent, rotating IP addresses to escape a block, or solving challenge pages turns a polite client into a tool built to defeat access controls. In 2026, two US courts treated such techniques as circumvention in lawsuits over automated access to search results. The cases are still open and nothing is decided, but the risk is not theoretical.

It also breaks trust with the sites your users depend on. A refusal is information: record it, tell the user the page was not available, and move on to another source.

What do built-in fetch tools do when a site refuses?

Most return the error to the model, which then tells the user it could not read the page. OpenClaw's documentation, read on 22 October 2026, says its built-in web_fetch extracts content with Readability, does not run JavaScript and caps output at 20,000 characters; it can fall back to a fetch provider, and plugins can register new ones. The MCP reference fetch server obeys robots.txt for model-initiated fetches by default.

When these tools fail on pages that allow robots but need JavaScript, a reader that renders pages helps. When they fail because the site refused, no tool should change the answer.

What does WebReaderAPI return when a site refuses?

It returns the refusal clearly, and you are not charged. A page disallowed by robots.txt gives 403 blocked_by_robots. A site declaring ai-input=no gives 403 blocked_by_site_policy. A site that refuses the request in any other way gives 502 target_refused, with the site's own status code in target_status.

If a site refuses our request, we return its answer: we never solve CAPTCHAs, log in to accounts, or try to get around a block or a paywall. Your agent can show the user exactly why a page was skipped. The errors page lists every code with the retry rule for each.