Quick answer
Do AI agents have to follow robots.txt?
It depends on the vendor. robots.txt is a voluntary standard (RFC 9309, September 2022). Crawlers that build indexes generally follow it; for fetches a user triggers, OpenAI says its rules may not apply, Google and Perplexity say their fetchers generally ignore them, and Anthropic says its bots honour them. WebReaderAPI follows robots.txt on every read, with no KYC and payment in crypto.
robots.txt and AI in October 2026
| Standard | RFC 9309, Robots Exclusion Protocol, September 2022 |
|---|---|
| OpenAI ChatGPT-User | robots.txt rules "may not apply" to user-initiated actions |
| Anthropic (incl. Claude-User) | Says its bots honour robots.txt directives, including Crawl-delay |
| Google user-triggered fetchers | "Generally ignore robots.txt rules" (page updated 2026-08-19) |
| Perplexity-User | "Generally ignores robots.txt rules" |
| Content Signals | Cloudflare policy of 24 September 2025: search, ai-input, ai-train |
| IETF AI preferences | Working-group drafts, not yet an RFC |
| WebReaderAPI | Follows robots.txt and ai-input=no on every read; token WebReaderAPI |
What does robots.txt actually do?
It tells automated clients which paths a site would rather they skip. The file sits at the root of a host, groups rules by user-agent token, and uses Allow and Disallow lines. RFC 9309, published in September 2022, made the long-standing convention a standard.
The RFC is explicit that the file does not lock anything: "These rules are not a form of access authorization." Compliance is voluntary. That is why each vendor's stated policy matters, and why sites that want hard limits also use firewalls and bot management.
Why do crawlers and user-triggered fetchers differ?
A crawler decides by itself which pages to visit; a user-triggered fetcher visits one page because a person asked. Many vendors treat the second case like a browser acting for its user, and say robots.txt does not fully apply.
| Fetcher | Stated policy (read 2026-10-22) |
|---|---|
| OpenAI ChatGPT-User | robots.txt rules "may not apply" to user-initiated actions; sites are pointed to OAI-SearchBot for search opt-out |
| Anthropic Claude-User | Anthropic says its bots respect robots.txt directives; blocking Claude-User stops retrieval for user queries |
| Google user-triggered fetchers | "Generally ignore robots.txt rules" because a user requested the fetch |
| Perplexity-User | "Generally ignores robots.txt rules" for the same reason |
| MCP reference fetch server | Obeys robots.txt for model-initiated fetches by default, not for user-initiated ones |
The split is real and documented. If you build an agent, decide which side you are on and say so in your user agent and docs.
What are Content Signals, aipref and RSL?
They are newer ways to say what AI systems may do with content, not just whether a robot may fetch it.
- Content Signals. Cloudflare published this policy on 24 September 2025. A robots.txt can state
search,ai-input(use as input to an AI answer, such as grounding) andai-train. Cloudflare's managed robots.txt servessearch=yes, ai-train=noby default. - IETF AI preferences (aipref). The working group's vocabulary and attachment drafts were still Internet-Drafts in October 2026, intended as Proposed Standards.
- Web Bot Auth. A way for bots to sign their requests; the IETF working group adopted its protocol draft on 1 September 2026.
- RSL (Really Simple Licensing). Its site says the RSL 1.0 specification is available; it lets publishers state licensing terms, including pay-per-crawl, through robots.txt, headers or RSS.
Is llms.txt the same thing as robots.txt?
No. robots.txt says what not to fetch; llms.txt is a proposed file that lists a site's key pages for language models. Adoption is growing but measured use is low. Rankability counted llms.txt on 9.3% of the top 1,000 sites on 18 September 2026. An Ahrefs study of 137,000 domains, reported by PPC Land on 2 July 2026, found that 97% of llms.txt files received no requests at all.
Our llms.txt guide shows how to build one from a crawl of your own site, if you want one anyway.
How does WebReaderAPI treat robots.txt?
It follows it on every read, including reads a customer triggers. We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged. If a site declares Content-Signal: ai-input=no in its robots.txt, we do not read its pages for you. For crawl jobs, it also waits the Crawl-delay a site sets.
Every request carries the same User-Agent, WebReaderAPI/1.0 (+https://webreaderapi.com/bot/), so site owners can see us in their logs and write rules for us. Our crawler page gives the details and a contact for problems.
# Block WebReaderAPI from the whole site
User-agent: WebReaderAPI
Disallow: /
# Or: allow it, but keep it out of one folder and slow it down
User-agent: WebReaderAPI
Disallow: /account/
Crawl-delay: 5
# Or: allow reading for search, but not as AI input (Content Signals)
User-agent: *
Content-Signal: search=yes, ai-input=no
Allow: /
robots.txt for AI: common questions
Does blocking GPTBot stop ChatGPT from reading my page for a user?
Not necessarily. OpenAI documents separate tokens: GPTBot for training, OAI-SearchBot for search and ChatGPT-User for user actions, and says robots.txt rules may not apply to user-initiated actions. Check each vendor's current documentation.
Is robots.txt legally binding?
Not by itself; RFC 9309 says it is not access authorization. In the EU, machine-readable opt-outs matter for text and data mining under the DSM Directive. See our legal overview; it is not legal advice.
How do I block WebReaderAPI?
Add a User-agent: WebReaderAPI group with Disallow: / to your robots.txt. Our reads check it before every page; disallowed pages are never read.
Does WebReaderAPI read pages that say ai-input=no?
No. If a site declares Content-Signal: ai-input=no in its robots.txt, we do not read its pages for you. The API returns 403 blocked_by_site_policy to the customer, without charge.
Sources (12)
- RFC 9309: Robots Exclusion Protocol — RFC Editor (September 2022)
- Overview of OpenAI crawlers — OpenAI (read 2026-10-22)
- Does Anthropic crawl data from the web, and how can site owners block the crawler? — Anthropic (2026-04-07)
- Google user-triggered fetchers — Google (updated 2026-08-19)
- Perplexity crawlers — Perplexity (read 2026-10-22)
- MCP fetch server — Model Context Protocol (read 2026-10-22)
- Content Signals Policy — Cloudflare (2025-09-24)
- AI Preferences working group documents — IETF (read 2026-10-22)
- Web Bot Auth working group documents — IETF (read 2026-10-22)
- Really Simple Licensing — rslstandard.org (read 2026-10-22)
- llms.txt adoption data — Rankability (2026-09-18)
- llms.txt adoption rises 8.8x but 97% of files get zero AI requests — PPC Land (2026-07-02)