Financial Data Scraping: Browser Requirements

Financial data scraping is not a harder version of an ordinary page scrape; it is a different problem, because the source usually has an account, a session, a step-up challenge and a data agreement behind it. BotBrowser documents per-context fingerprint bundles and profile-backed browser values, so one collection session can stay bound to one declared identity. It does not authenticate to an institution, store credentials, satisfy a multi-factor challenge, or replace a licensed feed. Confirm an authorized route before designing the job at all.

TL;DR

• Regulated financial data usually has a licensed route. Identify the vendor feed, official API or agreed export first, and treat a browser job as the fallback that needs its own justification.
• Bind one authenticated session to one profile. A run that resumes the same session under a second profile has already lost the ability to say who it was.
• A step-up challenge is a state transition. In-flight requests from before the challenge are void, not retryable.
• Record the TLS expectation you are relying on and the channel properties you actually observed. "It connected" is not an audit statement.
• Keep a per-request audit record that names the route, the outcome and the time, and never stores the credential value itself.

Contents

• Four requirements a general scrape does not have
• Bind one session to one profile
• Treat a step-up challenge as a state transition
• Record the TLS expectation and the observed channel
• Build the per-request audit record
• When the licensed feed is the answer
• Practical conclusion

Four requirements a general scrape does not have

An ordinary page scrape can be described by a URL, a parser and a schedule. A regulated-data job also has to answer who it was, what it was allowed to do, and what happened when the source interrupted it. Those questions do not appear in a general scraping checklist, and leaving them out produces a dataset nobody can defend later.

Requirement	Failure it prevents	Record field
Session bound to a profile	One session silently resumed under a second browser identity	session_id, profile_id
Step-up as a state change	In-flight requests replayed after the source demanded a challenge	auth_state, voided_at
Recorded TLS expectation	An unexamined assumption that the channel was authenticated	expected_channel, observed_channel
Per-request audit record	Collection that cannot be attributed to an authorized run	request_id, outcome, observed_at

Each row has a specific failure behind it rather than a general principle. The first prevents a run from splitting its identity halfway through. The second prevents a client from treating a challenge as a transient error worth retrying. The third prevents "the request succeeded" from standing in for "the channel was what I assumed". The fourth prevents a dataset whose provenance exists only in someone's memory of the run.

Bind one session to one profile

MDN describes the Authorization request header as a way for a user agent to provide credentials that authenticate it with a server, usually after the server answered an initial request with 401 Unauthorized and a WWW-Authenticate header. That description is about a single request-and-response exchange. A collection session is a longer object: it spans many requests, it may be tied to a cookie or a token, and the server decides whether it is still valid.

The consequence for a browser job is that the session and the browser identity have to be bound together and recorded together. If a run starts under one profile, hits a problem and resumes under another, the resulting data came from a session whose declared identity changed mid-flight. Even if every individual request succeeded, the run can no longer answer who it was.

The session continuity analysis separates client retention from server acceptance: keeping a cookie is not the same as the server honouring it. That distinction matters more here, because a financial source usually ties the session to more than a cookie. Record the binding explicitly, as a pair, and refuse to continue when either half changes.

BotBrowser's per-context documentation describes assigning a fingerprint bundle to a BrowserContext, with each context carrying its own profile file, user agent, timezone, locale and proxy configuration. That is the mechanism a collection job can use to keep one context's declared identity stable for the length of a session. It is a browser-side input; it does not make the session authorized, and it does not tell you whether the server still considers it valid.

Two operational habits make the binding survive contact with a real run. The first is to allocate the session identifier before the context is created, and to refuse to create a second context for the same session. That ordering matters because a context created first will happily receive a session that already exists elsewhere, and the duplicate is only discovered when the source invalidates one of them. Allocating first turns a late, confusing invalidation into an immediate, local error.

A rotating proxy profile record covers the same allocation discipline for a route change, and a passkey user journey shows why the credential and the session it opens must stay separate objects. The second habit is to record the binding at the moment it is made rather than reconstructing it from logs afterwards. A record written at allocation time can include the run identifier, the profile identifier and the context's launch arguments, which are exactly the facts a reviewer needs and exactly the facts that are hardest to recover later. Reconstructing them from a log means matching timestamps across two systems, and the match fails precisely when the interesting case occurs.

Session binding also has a termination side. When a run finishes, the context should be closed and the session marked closed in the same record. A session left open in the bookkeeping while its browser is gone will eventually be resumed by a later run that believes it is continuing something, and the resulting data will span two different identities without saying so. Closing both halves together is a small discipline that removes an entire class of provenance questions.

Finally, distinguish the binding from the credential. The binding is a statement about which profile carried a session. The credential is the material that proves the session. A run record can carry the first safely and should never carry the second, because the first is an identifier and the second is a secret. Keeping that line explicit in the schema prevents a later feature, such as a debugging export, from accidentally shipping credentials into a dataset.

Treat a step-up challenge as a state transition

A step-up challenge is the point where the source asks for more evidence than the session currently carries. It is common in financial interfaces, and it is frequently mishandled by collection code because the challenge arrives as an ordinary-looking HTTP response.

RFC 8446 describes the TLS channel and its authentication properties, including that the server side is always authenticated while the client side is optionally authenticated. A step-up challenge is a different layer: it is an application-level demand that the client prove something again. The two are easy to conflate when both appear during a single failing request, and the mistake leads to code that retries the request instead of changing the session's state.

The state model below is constructed, not observed at any institution. It exists to make the transition explicit so a client cannot silently continue.

Event	Session state after the event	What happens to in-flight work
Request accepted	authenticated	Continues
Challenge returned	step_up_required	Voided; each request is reclassified individually
Re-authentication completed	authenticated again	New work only; voided work is not replayed
Request refused by policy	refused	Recorded as an outcome, not retried

The third row is the one that collection code most often gets wrong. After re-authentication the session is usable again, which makes it tempting to replay everything that failed during the challenge. Those requests were made under a state the source had already rejected, so replaying them produces observations that cannot be attributed to the current authorization. Void them, record the void, and start new work.

The fourth row covers the case where the source refuses on policy grounds rather than asking for more evidence. That is not a technical failure and retrying will not change it. Record the refusal with its reason, stop the run, and return to the authorized-route question rather than treating it as a transient error.

A challenge also has a timing dimension that the state table alone does not capture. Requests issued before the challenge may already have been accepted by the source, and their responses may still be in flight when the challenge arrives. Those responses are not invalid because they arrived late; they are invalid as evidence about the current state, because the state they belong to no longer exists. The client should classify each one by the state it was issued under, not by the state that prevails when it completes.

That rule has a practical consequence for batching. A job that dispatches a hundred requests and then processes the results in completion order will interleave observations from two states without any marker. A job that dispatches in bounded groups and checks the state between groups keeps the boundary visible. The group size is a throughput decision, but the boundary check is a correctness decision, and it should not be relaxed to save a round trip.

Retry policy belongs to the same model. A network timeout under an authenticated state can reasonably be retried once with a new request identifier, because the state did not change. A timeout during step_up_required cannot, because the retry would carry a request into a state the source has already rejected. Writing that distinction into the retry function, rather than into the surrounding prose, is what keeps the model true after the next refactor.

Record the TLS expectation and the observed channel

TLS expectations in a regulated-data job are usually stated by the counterparty, not by the client. The client's job is to record what it was relying on and what it actually saw, so that a later reviewer can tell whether the two matched.

The connection trust analysis covers certificate validation and the signals a browser exposes about a connection. What a collection job needs on top of that is much smaller: a declared expectation, an observed value, and a decision rule for when they disagree. The expectation might be a protocol version, a requirement that the server be authenticated, or a specific counterparty policy. The observed value is whatever the client can actually read for that request.

Two failure modes are worth naming because both look like success. The first is a silent downgrade: the connection completes, the data arrives, and nobody checks whether the negotiated channel was the one the agreement assumed. The second is a scope error: an operator verifies the channel once at the start of the run and then assumes it held for every later request. Record per request, or at minimum per connection with an explicit boundary, and state which of those two you did.

// Constructed fixture: four requests in one run, one channel expectation.
const EXPECTED_CHANNEL = { protocol: 'TLSv1.3', serverAuthenticated: true };
const RUN = { runId: 'run-7', sessionId: 's-41', profileId: 'p-17' };
const requests = [
  {
    requestId: 'r-1',
    route: 'positions',
    protocol: 'TLSv1.3',
    serverAuthenticated: true,
    outcome: 'accepted',
    observedAt: '2026-11-10T09:00:00Z',
  },
  {
    requestId: 'r-2',
    route: 'balances',
    protocol: 'TLSv1.3',
    serverAuthenticated: true,
    outcome: 'step_up_required',
    observedAt: '2026-11-10T09:00:04Z',
  },
  {
    requestId: 'r-3',
    route: 'balances',
    protocol: 'TLSv1.3',
    serverAuthenticated: true,
    outcome: 'accepted',
    observedAt: '2026-11-10T09:03:10Z',
  },
  {
    requestId: 'r-4',
    route: 'statements',
    protocol: 'TLSv1.2',
    serverAuthenticated: true,
    outcome: 'refused',
    observedAt: '2026-11-10T09:03:12Z',
  },
];

const rows = requests.map(r => ({
  request_id: r.requestId,
  run_id: RUN.runId,
  session_id: RUN.sessionId,
  profile_id: RUN.profileId,
  route: r.route,
  outcome: r.outcome,
  observed_at: r.observedAt,
  channel_matches:
    r.protocol === EXPECTED_CHANNEL.protocol && r.serverAuthenticated === EXPECTED_CHANNEL.serverAuthenticated,
}));

const lines = rows.map(
  r =>
    `request_id=${r.request_id} run_id=${r.run_id} session_id=${r.session_id} profile_id=${r.profile_id} ` +
    `route=${r.route} outcome=${r.outcome} channel_matches=${r.channel_matches} observed_at=${r.observed_at}`
);
console.log(lines.join('\n'));

The constructed output below records one accepted request, one step-up transition, one accepted continuation and one refused request, and it shows r-4 diverging on the channel:

request_id=r-1 run_id=run-7 session_id=s-41 profile_id=p-17 route=positions outcome=accepted channel_matches=true observed_at=2026-11-10T09:00:00Z
request_id=r-2 run_id=run-7 session_id=s-41 profile_id=p-17 route=balances outcome=step_up_required channel_matches=true observed_at=2026-11-10T09:00:04Z
request_id=r-3 run_id=run-7 session_id=s-41 profile_id=p-17 route=balances outcome=accepted channel_matches=true observed_at=2026-11-10T09:03:10Z
request_id=r-4 run_id=run-7 session_id=s-41 profile_id=p-17 route=statements outcome=refused channel_matches=false observed_at=2026-11-10T09:03:12Z

That row is the reason the record exists. An accepted request over a channel the job did not expect is not a clean observation, and the dataset should be able to show it without anyone re-reading logs.

Deciding what to do with a mismatched row is a policy question, and it should be answered before the first run rather than during an incident. Three options are defensible: reject the row and halt, accept the row but mark it so downstream analysis can exclude it, or accept it and record the mismatch as an advisory. The right choice depends on what the counterparty agreement says, which is why the decision belongs in the run's written basis rather than in the code that noticed the mismatch.

The observation mechanism also needs a stated limitation. A page-level script cannot read the negotiated protocol of its own connection; the value has to come from the browser's network instrumentation or from the client that made the request. Recording a channel field that was populated by an assumption is worse than leaving it empty, because an empty field prompts a question while an assumed one closes it. Where the value is genuinely unavailable, record that it was unavailable and why.

Version drift is the last piece. A run that spans weeks may see the counterparty change its supported protocols, and a channel expectation written at the start will slowly become wrong. Tie the expectation to a date and re-check it when the run is extended. An expectation without a date reads as though it were still current, and that is the same silent-downgrade failure in a slower form.

Build the per-request audit record

An audit record answers four questions per request: which run made it, under which identity, what the source returned, and when. Everything else is supporting detail. The record below is an application artefact, and it deliberately carries no credential value.

Field	Content	Why it is in the record
request_id	Opaque identifier for this request	Joins the record to a log line without exposing content
run_id	Identifier of the authorized collection run	Attributes the request to a permission
session_id	Session identifier, never a token value	Shows the identity that made the request
profile_id	Profile the context was launched with	Shows the declared browser identity
route	Endpoint class, not a full query string	Describes the request without storing parameters
outcome	accepted, step_up_required, refused	Makes the state transition explicit
observed_at	Time in UTC	Orders requests and bounds the run
channel	Protocol and server-authentication result	Records what the TLS expectation resolved to

Two fields deserve a note. session_id is not the token. Storing a bearer token in an audit table converts the audit trail into a credential store, which is a worse problem than the one the record was built to solve. Record an opaque identifier and a hash if you need to compare two records for the same session. The same reasoning applies to route: record the endpoint class rather than a full URL with query parameters, because those parameters frequently carry account or instrument identifiers that the audit record does not need.

The record also needs a retention decision that is separate from the run. An audit trail that is deleted with the working data cannot answer a question asked a quarter later, and one that is kept forever accumulates personal data the job never needed. Decide the retention period from the agreement and the question the record is meant to answer, write it down, and make deletion an explicit scheduled action rather than an accident of cleanup.

Access to the audit record is a second decision. It frequently contains account identifiers and endpoint classes, which together describe a customer's activity in more detail than the dataset itself. Restrict it to the people who need to answer provenance questions, and log reads of the audit trail as well as writes. A provenance system that cannot say who inspected it has the same weakness it was built to remove.

Finally, make the record verifiable. A record with a hash chain, or with a periodic digest written somewhere independent of the writer, can show that it was not edited after the fact. That property matters most in exactly the situation the record exists for: a dispute about what a run did. A plain table is better than nothing, but a table the writer can silently rewrite is weaker evidence than its contents suggest. Verifiability is not authorization: an audit record shows that a run happened and was not edited afterwards, and it does not prove that the run was permitted.

The shared-workstation session lifecycle applies a related rule to reuse across users: the record of who held a session matters as much as the session itself. A collection job with one profile per run is simpler, but it still needs the same attribution.

When the licensed feed is the answer

Most regulated financial data has a route that does not involve a browser at all. A market-data vendor, an official API, an aggregated export or a contractual feed gives you the same values with a defined schema, a stated update cadence and a licence that says what you may do with them. When one of those exists, a browser job is usually the wrong instrument: it produces a less stable dataset, with more legal exposure, for a source that already offered a supported path.

The honest decision rule is therefore not "can a browser reach it" but "is there an authorized route, and if not, what specifically authorizes this one". If the answer to the second half is a person's memory of an email, that is not an authorization. Write the route and the basis into the run record so the next person inherits the answer rather than the assumption.

schema.org's FinancialProduct type is a useful illustration of the boundary. It defines a vocabulary for describing a financial product, with properties such as annualPercentageRate and interestRate, and notes that the annual percentage rate differs from the calculated interest rate. That vocabulary tells you how to name fields; it says nothing about whether a source publishes them, whether a published value is current, or whether you may collect it.

FAQ

Does a successful login mean the collection is authorized?

No. Authentication establishes who the session is, not what the operator may do with the data. Authorization comes from the institution's terms or a separate agreement, and a working session does not substitute for either.

Can the same session continue after a step-up challenge?

It can, once re-authentication completes, but the work performed during the challenge is not part of that continuation. Void those requests, record the void, and begin new work under the new state.

Should the audit record store the access token?

No. Store an opaque session identifier and, if needed, a hash. A token in an audit table turns the audit trail into a credential store.

Does BotBrowser authenticate to a financial institution?

No. It provides documented profile-backed browser inputs, including per-context fingerprint bundles, so a session can stay bound to one declared identity. Credentials, challenges and data licences remain outside its scope.

Sources

• MDN Authorization header
• RFC 8446 TLS 1.3
• schema.org FinancialProduct
• BotBrowser per-context fingerprint
• BotBrowser fingerprint overview

Practical conclusion

Start from the authorized route. If a licensed feed or official API exists, use it and keep the browser job out of the design. Where a browser job is the supported path, bind one session to one profile, model a step-up challenge as a state transition that voids in-flight work, record the TLS expectation beside the observed channel for each request, and keep an audit record that names the run, the identity, the outcome and the time without storing credentials. Those four requirements are what make a regulated-data dataset defensible after the run has ended.