Back to Knowledge Hub
Getting Started

Web Scraping Browser: Which Kind Do You Need?

Choose between a browser you drive, a hosted result API, and a browser extension by page control, state ownership, permissions, and failure responsibility.

BotBrowser Team

Documentation

Want the structured docs for Getting Started?

This article lives in the editorial library. For step-by-step setup, reference material, and ongoing updates, jump into the docs section.

A web scraping browser can mean a browser your code drives, a service that returns a requested result, or an extension extracting from a document in your browser. Those are different interfaces, even when all three use Chromium somewhere. BotBrowser documents an executable controlled through Playwright with profile-first configuration; that does not make it a hosted extraction endpoint, an extension, or a guarantee that a site permits collection. The first useful choice is the control your task needs, not the browser name on a comparison list. Validating a returned artifact is separate from choosing the browser interface; see what a scraping API response establishes.

TL;DR

  • Choose browser control when your application must keep interacting with a page and decide the next action from its state.
  • Choose a result endpoint when its documented response is sufficient; an HTTP API can still run a browser and return rendered HTML.
  • Choose an extension when extraction belongs inside a permitted browsing workflow and its document access, permissions and output fit the task.
  • A hosted browser can remain under your automation code's control. Hosting the process does not automatically transfer extraction logic to the provider.
  • Keep browser access, application authorization and permission to use collected data separate. None follows from a successful connection or export.

Contents

A branching choice separates continued page control, a returned result and permitted extension access; browser control then splits into self-operated and hosted execution.
Choose the required interface first. Hosting location is a second decision, not another name for an extraction result.

Three meanings, four operating arrangements

The phrase mixes a capability with its packaging. A browser executes a document and exposes interaction with that document. An automation library controls a browser. A service can expose browser control or a completed result. An extension can read a permitted document and provide an export. The same vendor may offer more than one arrangement, so the product homepage is not enough to identify the contract. Inspect the actual interface the integration will call. Playwright's browser connection reference describes control over an existing browser; an endpoint returning text/html has a different output boundary.

The practical distinction is what happens after the first response. Can your code inspect a page and continue clicking or navigating in the same controlled session? Does it receive a finished HTML document or records and then make another request for more work? Or does a feature operate on the document already open in a permitted browser profile? These questions separate interaction, result delivery and document access without assigning a quality rank. A task may compose them, but each transition needs an explicit owner.

ArrangementWhat your application receivesWho runs the browser or taskFirst ownership question
Self-operated browser controlA controllable browser, context and pageYour runtime teamWho maintains the executable, context and shutdown?
Hosted browser controlA remote connection to a browser sessionService provider; your code still directs the pageWhich connection and session contract does the client use?
Hosted result APIThe endpoint's documented result or errorService provider behind that endpointIs the returned artifact sufficient, and who validates its fields?
Browser extensionThe extension's supported document access and exportBrowser/profile operator plus extension implementationWhich documents and actions have granted access?

This matrix precedes the six browser-selection criteria. That scorecard compares candidate configurations once browser control is the chosen interface. It does not replace deciding whether a browser session is required. Likewise, the headless-mode comparison concerns the executable and launch mode after choosing a browser path. A browser running without a visible window is not, by that fact, a hosted service or an extraction API.

The rows are not exclusive product categories. An extension can call a remote service, and a hosted browser can support a separate result endpoint. A self-operated program can wrap its own browser in an internal HTTP service. Describe the boundary your application consumes rather than assuming the entire system belongs to one row. That also keeps responsibility visible when one successful component passes an incomplete artifact to the next.

Browser control means owning the next action

Browser control is useful when the extraction depends on successive page states. An owned application might reveal a detail panel only after a user action, navigate through a workflow, or update a table after a setting changes. With a browser-control interface, your code chooses the action, observes the resulting document and decides whether another action is needed. The Playwright library exposes browser contexts and pages for such workflows. It does not choose the required business record or decide which missing field is acceptable.

For a self-operated process, the runtime team owns browser availability. The executable must exist, have the required runtime dependencies and start under the intended account. Its lifecycle also needs a deliberate end: closing a tab is not a substitute for defining how the process, contexts and temporary state are managed. BotBrowser's Playwright setup describes selecting its executable and profile inputs. Those configuration steps leave application navigation, extraction, record validation and job scheduling with the caller.

State needs its own decision. Playwright distinguishes clean browser contexts from a persistent context launched with a userDataDir. Its documentation says the persistent launch returns the only context and closing that context closes the browser. It also warns against automating the regular Chrome user profile and recommends a separate automation directory. An operator should therefore not expect a newly launched scraping process to inherit a person's everyday sign-in state. Use the documented persistent-context contract rather than treating every folder called a profile as interchangeable.

A clean context is equally deliberate. Playwright's isolation guide describes separate cookies and storage for its test contexts. Playwright Test creates those contexts automatically, while library users can create them explicitly. This storage boundary improves reproducibility, but it does not make contexts separate physical machines or establish anonymity. If an owned application needs a selected language or a permitted test-account state, record whether the context is supposed to begin empty or retain that state. A missing preference in an intentionally clean context is not necessarily a launch defect.

Hosting changes the runtime operator without necessarily changing the control interface. Playwright can connect to a remotely running browser, and its connectOverCDP method attaches through Chrome DevTools Protocol. The documentation limits that CDP connection to Chromium-based browsers and notes lower fidelity than the Playwright protocol connection. This matters when a remote attachment succeeds but a particular automation feature behaves differently. Check the supported connection method and versions before attributing an extraction problem to the page. A remote endpoint is still a control channel, not a completed data record.

The caller's logic remains important after connection. If your program chooses the selectors, waits for document state and constructs the record, the provider has not taken over those responsibilities simply by running the process. Conversely, a provider may offer additional managed tasks under another contract. Keep those products separate in the integration design. For this initial choice, ask whether your code needs continued page control and whether your team wants to operate the runtime. Detailed provider state, regional networking and adoption questions belong to the next deployment decision, not a claim that all hosted environments are outside customer control.

A result API is a different contract

A result endpoint accepts a documented request and returns a documented artifact. That artifact might be source HTML, rendered HTML, an image or structured fields; these are not interchangeable. Browserless's Content API documentation gives a concrete example: its /content endpoint describes rendering a URL or supplied HTML in a browser and returning text/html, including JavaScript-rendered content. The fact that a caller uses an HTTP request therefore does not establish that the service skipped browser execution.

The useful question is whether that response contains everything the next stage needs. If a downstream parser only requires the rendered document at a defined state, a result interface may be sufficient. If the caller must inspect an intermediate dialog and decide the next action, a simple HTML response does not by itself supply continued page control. Some services expose configurable waits or additional tasks, but those must be established from the selected endpoint's current contract. Do not infer them from the provider's use of the word browser.

Rendering and extraction are also different services. Returning HTML does not mean the service selected the correct item identifiers, interpreted a currency or checked that every required record exists. A structured extraction endpoint may accept some of those duties, but only as specified. The integration owner still needs to validate its output against the application's schema. For a hypothetical catalogue record, a delivered document and a valid price value are separate outcomes. Missing currency information cannot be repaired by labelling the HTTP request successful.

An endpoint error is not the only incomplete result. A response can arrive with the expected media type while containing a sign-in document, a not-found message, an application error or a page whose required content has not appeared. Treat the returned artifact according to its meaning before parsing it as the requested business data. The rendering and pagination diagnosis addresses content readiness; changing the packaging from a browser library to an API does not remove the need to define what a completed document means.

State and retention are contract questions here too. An HTTP interface is not proof of stateless execution, automatic persistence or deletion after a response. Ask which session facilities the endpoint actually offers and which data its processing terms cover. The same caution applies to retries: a second request may repeat the task rather than resume an earlier page. Do not rely on either behavior without documentation. This is an interface distinction, not a statement that result APIs provide less control or privacy in every implementation.

Result APIs can reduce the amount of browser-lifecycle code in the consuming application, but they do not remove responsibility for input scope, allowed data handling or downstream failure classification. A team choosing this path should be able to name the required response type, the point at which the task is considered finished and the owner who handles malformed or incomplete output. Those requirements form the acceptance boundary; vendor names and endpoint convenience do not determine them.

An extension works inside a permission boundary

A scraping extension is software operating through browser-extension capabilities. It may read an open document, expose a selection interface, organize records or export data, depending on its implementation. Chrome's content-script documentation describes scripts that can read and modify a page's DOM and communicate with the extension. That makes an extension relevant when the task belongs inside an existing, permitted browsing workflow rather than in a separately launched automation process.

Document access is bounded. Chrome requires appropriate host permissions for programmatic content-script injection; temporary access through activeTab is one documented option. Matching rules, frames and injection timing also determine where scripts run. Browser permission grants an extension a capability within the browser. It does not establish permission from the website to automate collection or a lawful purpose for reusing the data. A user clicking an extension action cannot collapse these separate decisions into one approval.

An extension is not necessarily manual-only. Chrome supports content scripts that run automatically on matching pages, as well as dynamic and programmatic injection. Whether a particular extension supports scheduled work, navigation or unattended operation depends on its implementation and operating environment. Do not select or reject the whole category based on the assumption that every record needs a separate click. Instead, inspect the feature's supported trigger, document scope and output lifecycle.

The execution environment is not identical to the page's JavaScript environment. Content scripts normally run in an isolated world: they share access to the document but do not automatically share the page's variables. A visible table may be available through DOM access while an application-internal object is not directly accessible from that isolated context. This distinction helps explain why an extension's supported extraction contract matters. It is not a recipe for reaching into private page state or broadening access beyond granted permissions.

Background work has another lifecycle. Chrome's extension service-worker guide describes normal worker termination after inactivity and revival by later events. It warns that global variables are lost when a worker shuts down and recommends persistent storage for durable values. This does not mean an extension stops functioning whenever its worker becomes dormant. It means a multi-step job must not assume one in-memory background process will stay alive throughout the workflow. A restart and an uninstalled extension are different diagnoses.

The browser profile is also part of the privacy boundary. An extension can operate alongside a person's permitted browsing state, but its requested access and data destinations need review. Determine whether records remain local or are sent elsewhere from the actual extension's documentation and policy; neither the extension label nor a local export button answers that question. Limit the document and data scope to the task. If a workflow requires a separately isolated unattended environment, compare that requirement with browser control instead of expanding permissions until an extension resembles another operating model.

Route an empty result to its first failed boundary

An empty export is a symptom shared by all arrangements. The missing records do not distinguish a startup problem from denied document access. A browser process might never start; a remote connection might use an unsupported protocol combination; an API might return the wrong document; an extension might lack access to the relevant frame. Only after those boundaries are established does a selector or schema investigation become meaningful. Playwright's connection reference and Chrome's permission rules describe different contracts, so they require different first checks.

Arrangement and hypothetical symptomFirst boundary to establishResponsible next check
Self-operated job has no pageExecutable startup and context creationRuntime owner checks the configured binary and startup result
Hosted browser connects, then an action failsSupported protocol, session and actual pageIntegration owner distinguishes connection capability from document readiness
Result endpoint returns HTML but no recordsRequested artifact, received document and required fieldsAPI owner checks the response contract; data owner validates its meaning
Extension export is empty on one documentGranted access, match/frame and script timingBrowser or extension owner checks access before changing extraction rules
Extension loses an unfinished job after dormancyDurable job state versus worker globalsExtension owner checks storage and event-resumption behavior

These are diagnostic examples, not observed incidents or statements about specific vendors. The table assigns the first check rather than the final cause. A successful browser launch can still lead to an application-level failure, and a valid extension permission does not guarantee that a page contains the expected fields. Keep stage outcomes separate enough that one success does not silently certify the next stage.

The same applies to resource cost. A completed navigation, a returned API response and an exported valid record have different denominators. Compare costs only after defining the unit your application accepts. The existing browser-selection scorecard provides workload measures once the interface is chosen. Reusing that scorecard as a universal ranking across unlike outputs would obscure who performed the work and what the result actually contained.

Choose the smallest sufficient interface

Start with the required artifact and the next decision. For a hypothetical owned catalogue that requires only a completed document for a separate parser, a documented result endpoint may suffice. For an owned workflow where code must choose among page states and continue interacting, browser control is the relevant contract. For an authorized user reviewing a small document set in an existing browser workflow, an extension may fit if its granted access and export support the needed data. These examples choose interfaces, not products or access strategies.

Then choose where state belongs. A clean context, an explicitly persistent automation directory, a provider-managed context and extension storage have different lifecycles. Record which state must survive and which must be isolated. Avoid importing unrelated everyday profile data to make a job appear complete. In the BotBrowser Playwright integration, executable and profile configuration remain distinct from application records; the automation program still determines how its job begins and ends.

Finally, verify a small authorized fixture at the boundary you selected. Check that the required document or record arrives, that intended state behaves as declared and that an incomplete result reaches the responsible owner. No benchmark or service test is reported here. The outcome of that future fixture would establish compatibility for the tested setup, not anonymity, permission or a general promise that a remote site will accept automated traffic.

FAQ

Is a web scraping browser the same as a browser extension?

No. An extension provides its implemented features within browser permissions and document scope. Browser-control software directs a browser session. Both may inspect a document, but the available interaction, lifecycle and state responsibilities differ.

Does an HTTP scraping API mean JavaScript is not executed?

No. A documented result endpoint can run a browser and return rendered HTML. Check the particular endpoint's response and readiness contract rather than inferring its execution model from HTTP transport.

Does hosting a browser transfer extraction logic to the provider?

Not automatically. When your client drives pages through a remote connection, it still owns its navigation and extraction logic unless a separate managed-task contract assigns that work to the service.

Can an extension run without clicking for every page?

Some can. Chrome supports automatically matched content scripts, but a particular extension's unattended behavior depends on its implementation, permissions, background lifecycle and browser environment.

Does BotBrowser replace a scraping API or scheduler?

Its cited documentation describes browser execution and Playwright integration. That does not supply an extraction schema, queue or hosted-service contract. The application remains responsible for those components and for authorized use.

After selecting hosted browser control, the Browserbase session and state boundaries distinguish provider termination from a valid record, saved Context writeback from application state, and browser region from proxy exit. Those provider-specific contracts determine which evidence an integration owner needs next.

Sources

Practical conclusion

Choose the required interface before the browser brand: continued page control, a documented result or permitted extension access. If browser control is needed, decide separately who operates the runtime. Keep state, validation and failure ownership explicit so that a successful component does not stand in for a valid record or permission to collect it.

#Web Scraping#Browser Automation#Browser Extensions

This article is part of the guide: Browser Automation Getting Started: A First Run Map

Run the same check in BotBrowser

Load a profile, keep the signals you just read about consistent inside the browser, and confirm the result on your own machine before a rollout.