Web Scraping Ethics Beyond the Robots File
Four ethical scraping decisions a data team can test without a lawyer: request load, data minimization, attribution and the effect on the source.
BotBrowser Team
Want the structured docs for Getting Started?
This article lives in the editorial library. For step-by-step setup, reference material, and ongoing updates, jump into the docs section.
A robots.txt check can pass while a collection still takes more from a service than its owner would accept. BotBrowser documents browser automation for authorized work and requires operation on systems the team owns or has written permission to test; it does not decide whether a collection is fair to the source, and it does not make an ethical choice on the team's behalf.
TL;DR
- Ethics here means four decisions a team can test without a lawyer: how much load, how little data, how much credit, and what effect the collection has on the source.
- A robots rule is a coordination instruction. It does not answer any of the four questions.
- Each test needs an observable signal and a named owner, or it becomes a slogan that no one applies.
- The legal and contractual questions stay in the separate pre-collection decision record.
- Write the four answers before the first request and review them when the job or the source changes.
Contents
- Four tests, not four principles
- Load: what the source actually has to serve
- Minimization: collect the smallest useful field set
- Attribution: make the source visible
- Effect on the source: the test teams skip
- Practical conclusion
Four tests, not four principles
An ethics page that lists values is easy to agree with and hard to use. A team needs a test it can run, a result it can record and an owner who can stop the job. The four below follow the order a job usually touches the source: first the request, then the data, then the use of the data, then the effect on the service.
| Test | Question | Observable signal | Owner |
|---|---|---|---|
| Load | Does the job take more capacity than the source can spare? | Requests per second, error rate, retry count, published quota | Engineering |
| Minimization | Is every collected field used? | Field list compared with the report or model that consumes it | Data lead |
| Attribution | Can a reader trace the data to its source? | Source name, URL, retrieval date in the output | Product |
| Source effect | Does the job change what other users see or cost? | Latency, error rate or availability during the window | Operations |
The tests are deliberately narrow. They do not decide legality, and they do not replace the terms review, which stays in the scraping legal and terms decision record. They answer a different question: given that the team may collect, is this collection considerate of the service and the people behind it? A team can fail a test and still be within its legal rights, and it can pass all four and still have a legal question. Keeping the two conversations separate is what makes either one useful. The robots.txt decision record shows why the rule check is a separate input rather than an ethical answer.
Load: what the source actually has to serve
Load is the easiest test to make observable and the easiest to ignore. A job that retries aggressively during an incident adds exactly the load the source cannot spare. The test is not "did we stay under a number"; it is "did the source's behavior during our window stay within the range it showed when we were not running".
Measure three things during the window and compare them with a quiet baseline: request rate from the job, the source's error rate, and the time other requests wait. If the job's rate is within the published quota and the source's error rate and latency do not move, the load test passes. If the error rate rises or latency grows while the job runs, the job is part of the cause until proven otherwise, and the ethical response is to slow down rather than retry harder.
window=02:00-04:00 UTC
job_rate=1.0 req/s
source_error_rate_baseline=two tenths of a percent
source_error_rate_window=three tenths of a percent
source_p95_baseline=210 milliseconds
source_p95_window=225 milliseconds
retry_budget=3 per URL
result=pass (within noise, no retry storm)
The numbers above are an example of the record shape, not a measurement of any site. The decision rule matters more than the values: a small movement inside normal variation is not a failure, and a large movement with a clear start time is not excused by a low average rate. A retry storm shows up as a rising retry count with a flat request count, which is why the retry budget belongs in the same record.
Load ethics also covers the window. A job that runs at the source's busiest hour takes capacity from real users even at a modest rate. A quiet window, a smaller concurrency and a hard stop when the source signals pressure are all part of the same decision. The rate limiting and concurrency article covers the mechanics; this test decides when the mechanics are being considerate.
Minimization: collect the smallest useful field set
Minimization asks a blunt question about each field: who uses it, and what happens if it is missing? A field that no report, model or support case consumes is a liability with no benefit. It costs storage, it widens the privacy surface, and it makes the dataset harder to justify later.
Run the test as a comparison, not a review meeting. Take the field list from the collection job and the field list from the consumer, and mark each field as used, planned or unused. A field marked planned for more than one review cycle is effectively unused; either a consumer appears or the field leaves the job. The test passes when every collected field has a named consumer and a stated reason.
collected=sku,list_price,currency,observed_at,seller_name,review_text
consumed=sku,list_price,currency,observed_at
unused=seller_name,review_text
decision=drop seller_name and review_text from the collection
next_review=2027-02-01
The example drops two fields that a price report does not need. A review text field can be especially tempting because it is visible on the page, but visibility is not a use case. If a later feature needs it, the team can add it with a new decision, a new retention rule and a new privacy check. Collecting it now "in case" reverses that order and turns an easy decision into a cleanup project.
Minimization also applies to precision. A full timestamp, a full URL with query parameters, or an exact device string may be more than the consumer needs. A coarse category, a date instead of a second-level time, or a normalized value can answer the same question with less exposure. The test is to ask the consumer whether the coarser value changes the decision; if it does not, collect the coarser value.
Attribution: make the source visible
Attribution is the test that most affects the people who read the output. A dataset that cannot be traced back to its source is hard to verify, hard to correct and easy to misrepresent. The test is simple: pick a row at random and try to find the source page and the retrieval date from the output alone. If that takes more than a minute or depends on someone's memory, the test fails.
Store the source name, the page URL and the retrieval date with each record, and carry them into the report or model that consumes the data. For a public page, a link is usually enough. For a page behind an authorized account, record the account context and the permission that allowed access rather than the credential. The point is traceability, not exposure of private material.
record=price-4412
source_name=shop.example.test
source_url=https://shop.example.test/item/4412
retrieved_at=2026-11-13T02:14:07Z
method=rendered page, authorized test account
Attribution has a second half: do not present collected data as the source's own statement. A price scraped from a page is an observation at a time, not an official feed, and a report that presents it as authoritative misleads the reader even when the number is correct. Label the observation, the time and the method, and let the reader judge. The data collection pipeline shows how raw and normalized records can be kept separate so a repost or a missing card does not become an invented fact.
Effect on the source: the test teams skip
The fourth test asks what the collection does to the service itself. A job can stay within a rate limit and still change the experience for other users if it runs at peak time, holds long-lived sessions, or repeatedly triggers an expensive page path. The test is to compare the source's behavior with and without the job, at the same time of day, and to look for a difference that a user would notice.
Choose a small set of signals: page latency, p95, error rate, and whether a known page still returns HTTP 200 and renders correctly. Run the comparison in a quiet window and again in the window the job actually uses. If the job's window shows a consistent difference, the job has an effect; reduce concurrency, move the window or stop. A difference that appears only when the job runs is the signal the test is looking for, and it is not answered by pointing at a published rate limit.
The effect test also covers the source's business, not only its servers. A collection that republishes a source's entire catalog can reduce the reason for anyone to visit the source, even if the requests are light. That is a judgment the team must make and record: what part of the source's value is being copied, and is the collection adding something back? A price history built from repeated observations adds something a single page view does not. A verbatim copy of the catalog adds little.
Record the answer with the same discipline as the other three tests. The record can be short, but it must name the effect considered and the reason the team accepts or rejects it. "We only request a few pages" answers the load test; it does not answer the effect test.
Put the four results in one record
The tests are only useful if a reviewer can read them together. Keep one record per job with four blocks, one per test, and a stop rule for each. The record can live next to the collection configuration, as long as it is versioned and has a review date. A record that lives only in a chat thread is not a record.
| Block | Fields | Stop rule |
|---|---|---|
| Load | window, job rate, baseline error rate, window error rate, retry budget | Stop when the window error rate exceeds baseline by a margin the team set in advance |
| Minimization | collected fields, consumed fields, unused fields, next review | Stop when an unused field has no consumer after one review cycle |
| Attribution | source name, source URL, retrieved at, method | Stop when a sampled row cannot be traced to its source |
| Source effect | compared signals, quiet result, job-window result, accepted reason | Stop when the job window shows a consistent difference a user would notice |
The stop rule is the part teams leave out. Without a pre-agreed margin, every result is arguable after the fact, and the person running the job has no clear instruction. Write the rule when the job is quiet, not during an incident, and make it specific enough that two people reading the same numbers reach the same decision. Review the record when the job changes. A new field, a new source, a new window or a higher concurrency can each invalidate one of the four results. The review is not a formality: it is the moment the team decides whether the previous answer still holds. If the answer is unclear, the safer default is the lower load and the smaller field set until the record is updated.
Two failure cases worth recognizing
The first case is a job that passes every individual test and still feels wrong. It runs at a low rate, collects four fields, attributes every row, and shows no measurable effect on the source. Then someone notices that the four fields reproduce the source's paid report exactly, and the collection is republishing a product rather than observing a market. None of the four tests catches that, because the question is about the source's business model rather than its capacity. The remedy is a scope decision: collect a narrower slice, add analysis the source does not provide, or stop. Record that decision as part of the source-effect test, even though the measurable signals were clean.
The second case is a job that fails the load test only during an unrelated incident. The source slows down for its own reasons, the job's retries add pressure, and the team sees a spike it did not cause alone. The ethical response is still to back off, because the job is now part of the problem, and the record should say so. Blaming the incident is accurate about the cause and irrelevant to the decision: a considerate job yields when the source is under pressure, then resumes on a schedule rather than immediately.
Both cases share a habit: the team writes down what it observed and what it decided, including the uncomfortable part, in the same record as job_rate and retry_budget. A record that only lists passing numbers teaches the next team nothing.
Questions outside these four tests
The four tests do not decide whether a source permits automated access, whether a field is personal data under a particular law, or whether a collection is lawful in a jurisdiction. Those questions belong to the separate pre-collection decision record. Mixing them into an ethics checklist makes the checklist look like a legal approval, which it is not. The tests also do not rank sources or tools. A source that publishes a generous API and a source that publishes none are not morally different by that fact alone; the team's obligations depend on what it collects and how it uses it. The four tests apply the same way to both.
Practical conclusion
Before the first request, run the four tests and record the result: load against a quiet baseline, fields against their consumers, rows against their sources, and the job's effect against the same source without it. Give each test an owner and a review date. Keep these ethical decisions separate from the legal and contractual record; a passing test does not create permission, and permission does not make a careless collection considerate.
FAQ
Is this article legal advice?
No. It covers four operational tests a team can run without a lawyer. Questions about terms, personal data, jurisdiction or authorization belong in the pre-collection decision record.
Does passing a robots.txt check satisfy these tests?
No. A robots rule is a crawler-coordination instruction. It does not measure load, field use, attribution or the effect on the source. The four tests need their own evidence.
How often should the tests run?
At the start of a job, whenever the field list, source, window or concurrency changes, and on the review date in the record. A job that ran considerately last quarter can become careless after a source change.
What should a team do when a test fails?
Reduce the pressure and re-run the test: lower concurrency, move the window, drop unused fields, add attribution or stop the job. A failed test is a reason to change the job, not a reason to reword the result.
Sources
This article is part of the guide: Browser Automation Getting Started: A First Run Map
Related Articles
Scraping Legal and Terms: What a Team Must Decide First
Deploymentrobots.txt Scraping: Rules, Blocks, and Fetch Failures
Getting StartedWeb Scraping Browser: Which Kind Do You Need?
Getting StartedScraping with Playwright: Browser and Profile Setup
Getting StartedWeb Crawling: Crawl Budget and Politeness Basics
DeploymentJob Board Scraping: Browser Boundaries
Run the same check in BotBrowser
Load a profile, keep the signals you just read about consistent inside the browser, and confirm the result on your own machine before a rollout.