Scraping Legal and Terms: What a Team Must Decide First

Most scraping projects start with a technical question and discover the permission question later, usually after someone asks why the source blocked the job. BotBrowser documents browser automation for authorized work and states that operators must use systems they own or have explicit written permission to test. It does not decide whether a collection is lawful, does not grant permission to collect, and does not replace a terms review or legal advice.

TL;DR

• Write four decisions before the first request: which source and terms, which data, how much load, and how long you keep it.
• A robots rule, a rate limit, a login and an authorization decision are four different things. Passing one says nothing about the others.
• A technical control is evidence about behavior. It is not a license, a contract term, or a legal basis.
• Record the source's terms version and the date you read it, so a later change is visible instead of remembered.
• Escalate to counsel when the source restricts automated access, the data includes personal information, or the answer depends on jurisdiction. Do not resolve those questions with a browser setting.

Contents

• Four decisions before the first request
• Read the terms at the clause level
• Write the decision record
• Separate a technical control from permission
• Know the escalation point
• Practical conclusion

Four decisions before the first request

The first decision is which source you are collecting from and which terms apply to it. A site's terms of service, an API agreement, a developer policy and a public data license can each say something different, and they can change without notice. Record the document you read, its version or last-updated date, and the clause that matters. "The site looked public" is not a record of a decision.

The second is which data the job actually needs. A page can contain a product price, a review, a profile photo, a user handle and a timestamp. Each has a different sensitivity and a different retention need. Decide the fields before the crawl, not after the export, because a broad export is hard to narrow once it exists. Keep the smallest field set that answers the business question.

The third is request load. Concurrency, retry policy, crawl window and per-origin rate belong in the record, together with the source's published limits if any. A 429 response is a signal about load, and a technical ability to retry does not make a higher rate acceptable. The article on rate limiting and concurrency for scrapers covers the mechanics; this record decides the budget the mechanics must respect.

The fourth is retention and deletion. Decide where the data lives, who can read it, how long it stays and what triggers deletion. A collection with no retention decision tends to become permanent by accident. Record the owner and the review date, because an unreviewed dataset outlives the project that justified it.

Decision	Record	Wrong shortcut
Source and terms	Document name, version date, relevant clause, reviewer	Assuming public access means unrestricted use
Data scope	Field list, personal-data flag, purpose, minimization note	Exporting the whole page and deciding later
Request load	Concurrency, rate, window, retry cap, source limit	Retrying until it works
Retention	Storage location, access list, deletion trigger, review date	Keeping everything indefinitely

Read the terms at the clause level

A terms page is not one sentence. It usually contains a grant of access, a list of prohibited conduct, an intellectual-property clause, a limitation of liability and a change clause. The clause that governs automated collection may sit in a prohibited-conduct list, in an API section or in a separate developer agreement. Reading the headline is not the same as reading the clause, and a summary written by someone else is not the document.

Read for four things. First, does the document address automated access at all, and does it use words such as crawl, scrape, robot, automated means or rate? Second, does it restrict reuse of content separately from access, for example by prohibiting redistribution or creating a derivative database? Third, does it require an account, an API key or a written agreement for the data you want? Fourth, does it reserve the right to change the terms, and does it say how a change is notified? Each answer belongs in the record with the clause number and the version date.

A public data license is a different instrument from a terms page. An open license can grant reuse rights that the terms page does not, and a terms page can restrict conduct that the license does not mention. When both exist, record both and note which one the team believes governs the intended use. That belief is a question for counsel when the documents conflict; it is not something a crawler can resolve by choosing the more permissive reading.

The same discipline applies to an API. An API agreement may permit the calls it documents and prohibit other collection, and a rate limit in the documentation is an operational limit rather than a statement about legal reuse. Record the API version, the endpoint, the quota and the clause that permits the use. If the job needs a field the API does not return, that is a scope question, not a reason to fall back to page scraping without a new decision.

Robots rules deserve a precise reading as well. RFC 9309 defines a coordination mechanism for crawlers that choose to participate. It does not create a license, a contract or a legal basis, and it does not override a terms page. A Disallow line can tell a declared crawler to stay out of a path; an Allow line can tell it that the path is not disallowed. Neither answer settles whether the collection is permitted, which is why the robots decision and the permission decision sit in separate rows.

Write the decision record

A useful record is short enough to review and specific enough to audit. One row per decision, one named owner, one date. The record should show what was decided, what evidence supported it, and what remains open. An open item is not a failure; an unrecorded assumption is.

project=price-watch
source=https://shop.example.test
terms_version=2026-11-02
terms_clause=sections 4 and 9
fields=sku,list_price,currency,observed_at
personal_data=false
load=2 concurrent, 1 request/second, window 02:00-04:00 UTC
retention=90 days, owner=data-platform, review=2027-02-01
counsel_review=not_required (no personal data, no access restriction)

The record is a statement about a decision, not proof that the decision is correct. If the source later adds a clause, or the job starts collecting a field that was not approved, the record is out of date and the job should pause until it is revised.

A record with an open item is more useful than a record with a confident guess. Mark the open item, name the person who will answer it, and set the date by which the job either proceeds or stops. A job that starts while an item is open should be limited to a synthetic fixture or to data the team already owns, so the unresolved question does not become a production collection.

Keep the record machine-readable where the team already has a schema. The field names below are enough for a small project and can be extended without changing the meaning of the existing ones.

Field	Type	Purpose
source	URL	The exact origin and path scope the job may request
termsVersion	date	The version or last-updated date of the document read
termsClause	string	The clause that governs automated access or reuse
fields	list	The approved field set, not the full page
personalData	boolean	Whether any approved field can relate to a person
load	object	Concurrency, rate, window and retry cap
retentionDays	integer	How long the collected data may stay
owner	string	The accountable role, not a shared mailbox
reviewDate	date	When the record must be re-read
counselStatus	enum	not-required, requested, answered or stop

The counselStatus field is the one teams skip. Without it, a job can drift from "we asked someone once" to "we assumed it was fine" without anyone noticing. Record the question that was asked and the answer that came back, including a stop answer. A stop answer is a valid outcome and should end the job rather than trigger another technical workaround.

Separate a technical control from permission

Engineers often reach for a technical signal as if it settled the permission question. It does not. A successful request shows that the server delivered a response. A passing robots evaluation shows what the published crawler rules say for one declared token. A login shows that an account authenticated. None of those is a license, a contractual right or a legal basis, and each can be true while collection is still inappropriate.

The reverse is also true. A blocked request does not prove that collection is prohibited, and a rate limit does not establish a legal limit. The technical outcome and the permission question need separate owners and separate evidence. The robots.txt decision record covers the crawler-rule side, and the job board scraping boundaries show the same split for a specific source; it explicitly keeps permission and HTTP delivery as separate decisions, and that separation is the point here too.

BotBrowser's responsible-use policy requires operation on systems the operator owns or has written permission to test, and the legal disclaimer places compliance with applicable law and service terms on the user. Those are product-policy boundaries. They are not a legal opinion about any particular source, and reading them does not create permission for a job.

A technical control can also create a false sense of diligence. Adding a proxy, a delay or a rotating profile changes how the request looks; it does not change what the team is allowed to do. If a job needs those controls to avoid a source's stated restriction, the restriction is the question to resolve, not the control to tune. The proxy configuration overview explains what a route can and cannot change; none of those changes is a permission.

Keep three evidence sets apart in the record. The technical set holds response codes, rate observations and fixture results. The contractual set holds the terms version, the clause and any written approval. The legal set holds counsel's answer, if one was requested. A reviewer should be able to see which set supports which decision, and no set should be used to fill a gap in another. A 200 response cannot fill a missing clause, and a clause cannot prove that a request succeeded.

When a source publishes a machine-readable policy, use it as an input rather than as a substitute. A robots file, an API quota header or a documented crawl window tells the team what the source expects operationally. Those inputs belong in the load plan and in the robots decision. They do not remove the need to read the terms, and they do not answer the personal-data question.

Know the escalation point

Some questions belong to counsel, and pretending otherwise wastes engineering time and creates risk. Escalate when the source's terms restrict automated access or reuse, when the data includes personal information, when the collection crosses a login or an access control, or when the answer depends on which jurisdiction applies. Those are not decisions a browser profile, a proxy or a retry policy can make.

Bring counsel a narrow question and the evidence already gathered: the source, the terms version and clause, the field list, the load plan, the retention plan and the business purpose. A precise question gets a useful answer; "can we scrape this" invites a general warning. Record the answer and its date in the same decision record.

Escalation is not only for the first request. A new field, a new source, a longer retention period or a change in the terms can each reopen a decision that was previously closed. Treat the record as a living document with a review date, and pause collection when a material input changes.

The escalation question is easier to answer with examples. The table below shows the trigger, the evidence to bring and the decision that stays with counsel. It is a routing aid, not a legal test, and a team should add its own cases as they arise.

Trigger	Evidence to bring	Who decides
Terms prohibit automated access	Clause, version date, intended use	Counsel, with the product owner
Approved fields can identify a person	Field list, purpose, retention plan	Counsel and the privacy owner
Job needs a login or an access control	Account role, permission scope, source clause	Counsel and the account owner
Source is in more than one jurisdiction	Source location, team location, data subjects	Counsel
Source offers an API for the same data	API terms, quota, missing fields	Product owner, then counsel if terms conflict

A team can prepare all of this without a lawyer and still reach the point where only a lawyer can answer. That is the intended boundary. The engineering work is to make the question precise, gather the evidence and stop the job while the answer is open. The legal work is to decide the applicable basis. Mixing the two produces either an unauthorized collection or a stalled project that never gets a clear answer.

There is a second escalation trigger that teams miss: a change in the source. A new clause, a new login wall, a new API version or a change in the data returned can invalidate a decision that was correct last quarter. Put the review date in the record and treat the record as expired on that date until someone re-reads the source. An expired record is not a reason to keep running; it is a reason to pause.

Practical conclusion

Before the first request, write down the source and its terms version, the fields and their sensitivity, the request budget and the retention rule. Assign an owner to each. Keep technical evidence and permission evidence separate, and send the questions that depend on law or contract to counsel with the record attached. That practice will not make a collection lawful by itself, and it is not legal advice, but it turns an implicit assumption into a decision a team can review.

FAQ

Does a public page mean the data is free to collect?

No. Public availability describes who can see a page, not what a team may do with its contents. Terms of service, database rights, personal-data rules and contractual limits can each apply independently. Read the applicable documents and record the clause that governs the intended use.

Can a robots.txt Allow rule serve as permission?

No. A robots rule is a crawler-coordination instruction for a declared product token. It does not grant a license, a data-use right or a legal basis, and it says nothing about personal data. Keep the rule decision and the permission decision in separate fields of the record.

Is it enough to stop when the site returns 403?

Stopping is the right operational response, but it is not a permission analysis. A 403 shows that the server refused a request; it does not establish why, and it does not tell the team whether another route would be appropriate. Diagnose the refusal, then revisit the permission decision rather than trying variations until one succeeds.

When should a team involve a lawyer?

When the terms restrict automated access or reuse, when personal data is involved, when the collection crosses a login or access control, or when the applicable jurisdiction is unclear. Bring the decision record and a specific question; a general request for approval rarely produces a usable answer.

Sources

• RFC 9309: Robots Exclusion Protocol
• Google Search Central: robots.txt introduction
• BotBrowser responsible use
• BotBrowser legal disclaimer