# robots.txt for AI agents and fetchers: what applies in 2026

> robots.txt for AI agents in 2026: crawlers vs user-triggered fetchers, what OpenAI, Anthropic, Google and Perplexity say, Content Signals, aipref and RSL.

- URL: https://webreaderapi.com/guides/robots-txt-for-ai-agents/
- Last updated: 2026-10-22
- Publisher: WebReaderAPI (https://webreaderapi.com)

robots.txt was written for crawlers, then AI agents arrived that fetch a page because a person asked. Vendors now draw that line differently. This guide sets out the standard, each vendor's stated policy, the newer signals, and how our own crawler behaves.

**Quick answer — Do AI agents have to follow robots.txt?**

It depends on the vendor. robots.txt is a voluntary standard (RFC 9309, September 2022). Crawlers that build indexes generally follow it; for fetches a user triggers, OpenAI says its rules may not apply, Google and Perplexity say their fetchers generally ignore them, and Anthropic says its bots honour them. **WebReaderAPI** follows robots.txt on every read, with no KYC and payment in crypto.

## robots.txt and AI in October 2026

| Attribute | Value |
| --- | --- |
| Standard | RFC 9309, Robots Exclusion Protocol, September 2022 |
| OpenAI ChatGPT-User | robots.txt rules "may not apply" to user-initiated actions |
| Anthropic (incl. Claude-User) | Says its bots honour robots.txt directives, including Crawl-delay |
| Google user-triggered fetchers | "Generally ignore robots.txt rules" (page updated 2026-08-19) |
| Perplexity-User | "Generally ignores robots.txt rules" |
| Content Signals | Cloudflare policy of 24 September 2025: `search`, `ai-input`, `ai-train` |
| IETF AI preferences | Working-group drafts, not yet an RFC |
| WebReaderAPI | Follows robots.txt and `ai-input=no` on every read; token `WebReaderAPI` |

## What does robots.txt actually do?

It tells automated clients which paths a site would rather they skip. The file sits at the root of a host, groups rules by user-agent token, and uses `Allow` and `Disallow` lines. RFC 9309, published in September 2022, made the long-standing convention a standard.

The RFC is explicit that the file does not lock anything: "These rules are not a form of access authorization." Compliance is voluntary. That is why each vendor's stated policy matters, and why sites that want hard limits also use firewalls and bot management.

## Why do crawlers and user-triggered fetchers differ?

A crawler decides by itself which pages to visit; a user-triggered fetcher visits one page because a person asked. Many vendors treat the second case like a browser acting for its user, and say robots.txt does not fully apply.

| Fetcher | Stated policy (read 2026-10-22) |
|---|---|
| OpenAI ChatGPT-User | robots.txt rules "may not apply" to user-initiated actions; sites are pointed to OAI-SearchBot for search opt-out |
| Anthropic Claude-User | Anthropic says its bots respect robots.txt directives; blocking Claude-User stops retrieval for user queries |
| Google user-triggered fetchers | "Generally ignore robots.txt rules" because a user requested the fetch |
| Perplexity-User | "Generally ignores robots.txt rules" for the same reason |
| MCP reference fetch server | Obeys robots.txt for model-initiated fetches by default, not for user-initiated ones |

The split is real and documented. If you build an agent, decide which side you are on and say so in your user agent and docs.

## What are Content Signals, aipref and RSL?

They are newer ways to say what AI systems may do with content, not just whether a robot may fetch it.

- **Content Signals.** Cloudflare published this policy on 24 September 2025. A robots.txt can state `search`, `ai-input` (use as input to an AI answer, such as grounding) and `ai-train`. Cloudflare's managed robots.txt serves `search=yes, ai-train=no` by default.
- **IETF AI preferences (aipref).** The working group's vocabulary and attachment drafts were still Internet-Drafts in October 2026, intended as Proposed Standards.
- **Web Bot Auth.** A way for bots to sign their requests; the IETF working group adopted its protocol draft on 1 September 2026.
- **RSL (Really Simple Licensing).** Its site says the RSL 1.0 specification is available; it lets publishers state licensing terms, including pay-per-crawl, through robots.txt, headers or RSS.

## Is llms.txt the same thing as robots.txt?

No. robots.txt says what not to fetch; llms.txt is a proposed file that lists a site's key pages for language models. Adoption is growing but measured use is low. Rankability counted llms.txt on 9.3% of the top 1,000 sites on 18 September 2026. An Ahrefs study of 137,000 domains, reported by PPC Land on 2 July 2026, found that 97% of llms.txt files received no requests at all.

Our [llms.txt guide](https://webreaderapi.com/guides/llms-txt-generator/) shows how to build one from a crawl of your own site, if you want one anyway.

## How does WebReaderAPI treat robots.txt?

It follows it on every read, including reads a customer triggers. We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged. If a site declares Content-Signal: ai-input=no in its robots.txt, we do not read its pages for you. For crawl jobs, it also waits the `Crawl-delay` a site sets.

Every request carries the same User-Agent, `WebReaderAPI/1.0 (+https://webreaderapi.com/bot/)`, so site owners can see us in their logs and write rules for us. Our [crawler page](https://webreaderapi.com/bot/) gives the details and a contact for problems.

*robots.txt examples for WebReaderAPI*

```text
# Block WebReaderAPI from the whole site
User-agent: WebReaderAPI
Disallow: /

# Or: allow it, but keep it out of one folder and slow it down
User-agent: WebReaderAPI
Disallow: /account/
Crawl-delay: 5

# Or: allow reading for search, but not as AI input (Content Signals)
User-agent: *
Content-Signal: search=yes, ai-input=no
Allow: /

```

## robots.txt for AI: common questions

### Does blocking GPTBot stop ChatGPT from reading my page for a user?

Not necessarily. OpenAI documents separate tokens: GPTBot for training, OAI-SearchBot for search and ChatGPT-User for user actions, and says robots.txt rules may not apply to user-initiated actions. Check each vendor's current documentation.

### Is robots.txt legally binding?

Not by itself; RFC 9309 says it is not access authorization. In the EU, machine-readable opt-outs matter for text and data mining under the DSM Directive. See our [legal overview](https://webreaderapi.com/guides/is-reading-web-pages-for-ai-legal/); it is not legal advice.

### How do I block WebReaderAPI?

Add a `User-agent: WebReaderAPI` group with `Disallow: /` to your robots.txt. Our reads check it before every page; disallowed pages are never read.

### Does WebReaderAPI read pages that say ai-input=no?

No. If a site declares Content-Signal: ai-input=no in its robots.txt, we do not read its pages for you. The API returns `403 blocked_by_site_policy` to the customer, without charge.

## Sources

1. [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) — RFC Editor (September 2022)
2. [Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) — OpenAI (read 2026-10-22)
3. [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) — Anthropic (2026-04-07)
4. [Google user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers) — Google (updated 2026-08-19)
5. [Perplexity crawlers](https://docs.perplexity.ai/guides/bots) — Perplexity (read 2026-10-22)
6. [MCP fetch server](https://github.com/modelcontextprotocol/servers/tree/main/src/fetch) — Model Context Protocol (read 2026-10-22)
7. [Content Signals Policy](https://blog.cloudflare.com/content-signals-policy/) — Cloudflare (2025-09-24)
8. [AI Preferences working group documents](https://datatracker.ietf.org/wg/aipref/documents/) — IETF (read 2026-10-22)
9. [Web Bot Auth working group documents](https://datatracker.ietf.org/wg/webbotauth/documents/) — IETF (read 2026-10-22)
10. [Really Simple Licensing](https://rslstandard.org/) — rslstandard.org (read 2026-10-22)
11. [llms.txt adoption data](https://www.rankability.com/data/llms-txt-adoption/) — Rankability (2026-09-18)
12. [llms.txt adoption rises 8.8x but 97% of files get zero AI requests](https://ppc.land/llms-txt-adoption-rises-8-8x-but-97-of-files-get-zero-ai-requests/) — PPC Land (2026-07-02)

---

WebReaderAPI: WebReaderAPI is a web reader API for AI agents that turns any public web page into clean Markdown, paid in crypto with no ID upload. Prepaid credits from $0.07/1K pages never expire, and robots.txt is respected. No KYC — no ID, no selfie, no documents. Pay in crypto.