Web scraping is the automated collection of content or data from websites and APIs for use elsewhere. It can be harmless, such as search indexing, or harmful, such as copying a price list or harvesting personal data. Defenses are rate limits, authentication, monitoring and clear terms. robots.txt is a request, not a lock.
How does OWASP describe scraping?
The OWASP Automated Threats to Web Applications project lists scraping as OAT-011. It defines it as collecting application content or other data for use elsewhere, and notes that the collector may read every accessible path and parameter value of pages and APIs, extracting data from the responses. It can be ongoing or periodic, may use fake or compromised accounts, and may need no authentication at all if the information is public.
OWASP lists many names for it, including content scraping, price scraping, screen scraping, data aggregation, and, more recently, AI scrapers and bots that collect data to train or feed language models. It maps scraping to CWE-799, Improper Control of Interaction Frequency.
Not all scraping is hostile. Search engines, price comparison services and researchers all collect pages automatically. The question for a site owner is which automated clients you want, and at what rate.
What does robots.txt do?
RFC 9309 specifies the Robots Exclusion Protocol, which lets service owners say how crawlers may access content. The RFC says plainly that these rules are not a form of access authorization. Well-behaved crawlers honor them. Others do not.
So robots.txt helps with courteous automation and does nothing against a determined scraper. Anything that must be private needs a real control, such as login and authorization.
How do sites defend themselves?
Match the defense to the harm.
- Put sensitive data behind authentication and check authorization on every object, not just on the page.
- Rate limit per account, per address range and per route. Watch for sequential walking of IDs or pagination far beyond human speed.
- Return only the fields a client needs. APIs that over-share make scraping cheap.
- Monitor for patterns: unusual user agents, missing browser behavior, identical request timing, large sessions without page assets.
- Offer an official API with clear limits for legitimate bulk users, so they have an alternative.
- State your rules in terms of service and, where useful, a published crawler policy.
- Consider challenges for suspicious clients, remembering that they also cost real users time.
No single measure stops a determined actor. The aim is to make bulk copying slow, noisy and costly, and to detect it.
A constructed example
A small marketplace lists suppliers with phone numbers and prices on public pages. One night, requests walk through supplier pages one after another from rotating addresses, at a steady rhythm, with no images or scripts loaded. Next morning a competitor's catalog shows matching prices.
The fixes: require login to see phone numbers, rate limit page views per session, add a short delay on rapid pagination, and publish a price feed API for partners. Public price pages remain public, but the harvesting gets harder and the contact data is protected.
What about legality?
Rules differ by country and by what is collected. Personal data, copyrighted material, contract terms and computer misuse laws can all come into play. This article does not give legal advice. If scraping creates legal exposure for you or your business, ask counsel who knows your jurisdiction.
How to check or apply this
- Decide what is public on purpose and what is not. Move the rest behind authentication.
- Review APIs for over-sharing and for sequential IDs that invite walking.
- Set per-client and per-route rate limits, and alert on spikes.
- Log enough to reconstruct a scraping run: client, path, timing, account.
- Publish robots.txt and terms, and offer an API for approved uses.
- Test your defenses by writing a polite scraper against your own staging site and seeing what you detect.
- Review quarterly, because tools and patterns change.
Quick check
Does robots.txt stop scrapers?
No. RFC 9309 states these rules are not a form of access authorization. Only compliant crawlers follow them.
Is scraping the same as hacking?
Not necessarily. Collecting public pages is different from bypassing access controls, but where the line falls depends on law and context. Ask counsel for specifics.
Sources
- OWASP Automated Threats: OAT-011 Scraping
- RFC 9309: Robots Exclusion Protocol
- OWASP Automated Threats to Web Applications
Last reviewed: 18 October 2026.
