Is reading web pages for AI legal? The dated facts in 2026

What does EU law say about reading web content?

EU law allows text and data mining of lawfully accessible content, but lets rightholders reserve it. Article 4(3) of the DSM Directive (2019/790) says the commercial exception applies only where use has not been expressly reserved "in an appropriate manner, such as machine-readable means" for content published online.

The EU AI Act's Code of Practice for general-purpose AI models, published on 10 July 2025, goes further for its signatories. They commit to crawlers that read and follow robots.txt, and to other appropriate machine-readable protocols. That is why honouring robots.txt and declared AI preferences has become a baseline for European use.

Which US cases matter, and where do they stand?

None has produced a final judgment on reading public pages for AI. The facts so far, all allegations unless a court decided them:

• Reddit v. SerpApi, Perplexity and others (S.D.N.Y.). On 31 July 2026 the court let claims under DMCA §1201(a) proceed at the pleading stage, treating Google's challenge-based protections as measures that control access. No defendant has been found liable.
• The New York Times v. Perplexity. Filed on 5 December 2025. The Times alleges retrieval of its content at query time after it had blocked Perplexity in robots.txt.
• Amazon v. Perplexity. As reported on 5 August 2026, the Ninth Circuit vacated an injunction against Perplexity's Comet shopping agent, noting the assistant acted on its user's instructions. The panel said it set no broad rule for agentic AI.

The direction is mixed, and appeals continue. Treat any summary, including this one, as dated.

What lowers the risk when an AI tool reads pages?

A few practices come up in every rule and case above:

• Read public pages only. No logins, no members areas, no paywalls.
• Honour machine-readable opt-outs. robots.txt and declared AI preferences, such as ai-input=no.
• Never defeat a protection. No disguised user agents, no rotating addresses to escape blocks, no challenge solving.
• Avoid personal data. Do not collect profiles, contact details or other data about private people.
• Credit your sources. Keep the URL with each page and show it when an answer relies on it.

These are good practice, not a guarantee. Ask a lawyer about your own use case.

What does WebReaderAPI do on its side?

We built the service around those practices. We respect robots.txt: our crawler identifies itself as WebReaderAPI, follows the rules written for it or for all robots, and does not read a page that a site disallows. Refused pages are not charged. If a site declares Content-Signal: ai-input=no in its robots.txt, we do not read its pages for you. If a site refuses our request, we return its answer: we never solve CAPTCHAs, log in to accounts, or try to get around a block or a paywall.

Our Acceptable Use Policy forbids collecting personal data about private people and reading content behind a login or a paywall. Every request identifies itself as WebReaderAPI/1.0 (+https://webreaderapi.com/bot/), and our crawler page tells site owners how to reach us.