Web Crawler vs Web Scraper: Outputs and Failure Boundaries
Separate URL discovery from field extraction, define their handoff, and diagnose missing pages versus incomplete records before choosing a browser.
BotBrowser Team
Want the structured docs for Getting Started?
This article lives in the editorial library. For step-by-step setup, reference material, and ongoing updates, jump into the docs section.
Web crawler vs web scraper is a decision about two outputs: a bounded set of retrievals, and records extracted from retrieved content. A page can be fetched successfully while its price field is missing. BotBrowser provides a Chromium browser for an authorized rendering step; it does not discover a site's URLs, schedule a crawl frontier, or define the fields a scraper must return.
TL;DR
- Crawling decides which URLs to visit and manages their retrieval; scraping extracts selected information from content.
- The same program can perform both jobs. Framework names do not establish which output failed.
- A URL frontier contains pending work. An empty frontier does not prove that every relevant page was discovered.
- Browser rendering is a separate choice; neither task always requires JavaScript execution.
- Join fetch outcomes to record-validation outcomes before calling a collection complete.
Contents
- What the two jobs produce
- One pipeline can perform both jobs
- Why request counts do not describe the cost
- A handoff ledger with two acceptance checks
- URL identity is different from record identity
- A record needs more than a matching selector
- Assign the failure before replacing the tool
- Define completion and replay separately
- Set the collection boundary before fetching
- Practical conclusion
What the two jobs produce
A crawler selects URLs, schedules requests and records what happened to each retrieval. Seeds may come from an explicit list, a sitemap or links found in fetched pages. Google describes a crawler as a program that automatically discovers and scans websites. A useful crawl artifact is a ledger containing source_url, discovery origin, fetch status and retrieval time. Keeping response bodies is optional: an availability crawl may need only statuses and links. The step-by-step crawler trace follows a seed through fetching, parsing, URL resolution and scheduling, including why a discovered link can remain unfetched.
A scraper selects information from a response or rendered document and turns it into a record. Its contract might require sku, price and currency. The input can be one saved page or thousands of fetched documents; following links is not required to extract fields. A response with HTTP 200 can still contain an empty price, an unexpected layout or a value that fails validation.
| Decision | Crawling owns | Scraping owns |
|---|---|---|
| Input | Authorized seeds and URL scope | Retrieved content and field schema |
| Working state | Pending URLs and retrieval outcomes | Parse outcomes and candidate records |
| Output check | What was visited, skipped or unresolved | Which required fields and record keys are valid |
| Typical defect | A permitted catalogue page was never queued | A fetched item page has no valid price |
The frontier is the queue of work still to be considered, not an inventory of everything on the site. A run can exhaust that queue after missing a pagination link. Coverage needs a declared scope and an independent completion rule, such as reconciling against an owner-provided URL list.
One pipeline can perform both jobs
Scrapy's architecture provides a concrete counterexample to a rigid crawler/scraper split. A spider can return both extracted items and new requests. Its engine sends requests to the scheduler and items to item pipelines, where validation and persistence can happen. Two responsibilities share one framework without becoming the same acceptance check.
For an authorized catalogue, one component may discover /item/42 from /catalog, retrieve its document and hand that document to an extractor. The extractor then emits a price record. If it also emits the next catalogue URL, it is participating in discovery as well. The useful question is which component owns the URL decision and which owns the field decision.
Rendering fits between retrieval and interpretation when the required content depends on JavaScript. A crawler may render a page to discover navigation links; a scraper may render it to obtain a price. Either can also work from server-delivered HTML. BotBrowser can supply that browser step, while the surrounding application retains responsibility for its frontier, extraction rules and stored records. The Playwright browser setup covers connecting the browser stage when the recorded input requires it.
Why request counts do not describe the cost
A crawl that explores navigation can issue requests without producing any business records. A scraper fed saved documents may produce records without contacting the source again. If both stages independently retrieve /item/42, the source receives duplicate work even though the team sees one final row.
Track source requests separately from extraction attempts and browser work. Useful counters include fetch_attempts, render_attempts, parse_attempts and accepted record keys. Retries belong to the stage that retried: a parser repair against a retained body should not automatically trigger another network request. Retention itself needs a storage and data-handling policy; keeping every body is not a prerequisite for calling a process a crawler.
The rate and concurrency diagnosis separates request frequency from in-flight work. That distinction applies across the combined pipeline: two modules sharing an origin still contribute to the source's aggregate load. A browser adds rendering work and may fetch page subresources, so “one page” is not necessarily one HTTP request.
A handoff ledger with two acceptance checks
Use a stable retrieval identifier to connect the fetch ledger and extraction result. Keep the original requested URL alongside a final URL after redirects; a redirect must not silently change what the team thinks it collected. The following catalogue rows are hypothetical contract examples, not results from a test or a live site.
fetch_id | source_url | Fetch result | Extraction result | Decision |
|---|---|---|---|---|
f-41 | /item/41 | HTTP 200 | sku=41; price=12.90; currency=USD | Accept after schema validation |
f-42 | /item/42 | HTTP 200 | sku=42; price=missing | Reject the record; inspect extraction |
f-43 | /item/43 | HTTP 503 | Not attempted | Keep retrieval unresolved |
Add fetched_at, final URL, schema_version and parse status in the actual ledger. If bodies are retained, a content hash or storage reference identifies the exact input to reprocess. Store a record key separately from a URL: one document can contain several records, and several URLs can describe one item. Count unique records only after applying the declared key and validation rules.
The first acceptance check reconciles the authorized URL scope with visited, intentionally skipped and unresolved entries. The second checks required fields in the retrieved inputs that should yield records. A job that fetched 3 URLs but accepted 1 record has not demonstrated either “one third complete” or “fully complete” without those two denominators. An index page might intentionally yield no records; a failed item retrieval might hide several.
URL identity is different from record identity
The crawler's duplicate check answers whether a retrieval is already scheduled or sufficiently current. The scraper's duplicate check answers whether a business entity or observation is already represented. Those keys need not be equal. /item/41 and /offers/41 might describe the same item, while one /catalog document might contain several items. Deduplicating accepted records by URL alone would preserve duplicates in the first case and discard valid rows in the second.
Define a URL identity policy with the source owner instead of removing query parameters indiscriminately. A parameter named page may select another catalogue segment. A language or variant parameter may change the values a consumer needs. Conversely, an approved tracking-only parameter might be irrelevant to the retrieval contract. These are source-specific decisions; alphabetically sorting every query or dropping every parameter is not a generally valid equivalence rule.
| Identity decision | Retrieval consequence | Record consequence |
|---|---|---|
Keep a meaningful page parameter | Different catalogue segments remain reachable | Records may overlap and still need their own keys |
| Follow an authorized redirect | Retain requested and final URL in the ledger | Associate extracted values with the actual source document |
| Keep distinct observation times | A later retrieval is a new observation | A history consumer may require sku plus observation time |
| Reprocess a retained document | No new source request is needed | A new schema_version may change validation outcomes |
For a current-price table, sku might be an appropriate record key within a declared source and variant. For a price-history table, overwriting every row with the same sku would erase observations. Put the consumer's key into the handoff contract, and carry fetch_id as provenance rather than treating it as the entity identifier. This lets an operator explain whether a repeated row is an accidental duplicate, a distinct variant or an intentional later observation.
A record needs more than a matching selector
Finding an element named .price establishes only that the selector matched something in that document. The extraction contract still needs a numeric interpretation, a unit or currency, and a rule for absent values. The string 0.00 and an empty string cannot both become a generic false value: zero can be a legitimate amount, while absence may mean that the document does not contain the required field.
Use explicit validation cases for the actual source format. In the hypothetical catalogue, a consumer might require a nonempty sku, a decimal price and a recognized currency. A promotional message, a range or a member-only price should not silently become the standard price merely because it occupies the same visual box. Decide which representation the schema permits and reject unsupported shapes with a parse status that an operator can inspect.
| Candidate value | Question the extractor must answer | Result to preserve |
|---|---|---|
price=0.00 | Is zero permitted for this field? | Valid zero or a named validation failure |
price=missing | Is the field required for this record type? | Missing field, never an invented zero |
price=12.90; currency=USD | Does the source format support this interpretation? | Typed amount and currency with source provenance |
Several matching .price elements | Which one corresponds to the intended item and price type? | A disambiguated value or an ambiguity failure |
These are extraction checks even when the values came from a browser. Rendering can make an element available without explaining its business meaning. Store the schema version and the rejected input reference so a corrected parser can be evaluated against the same document. If a required field is absent from the retained input, a selector adjustment cannot create it; that finding returns the investigation to the input contract, not to a claim that rendering always fixes scraping.
Assign the failure before replacing the tool
If /item/42 never entered the frontier, inspect seed coverage, link discovery and scope rules. Changing a price selector cannot recover an undiscovered page. If it was queued but returned HTTP 503, investigate retrieval and the source's permitted retry policy before counting it as an extraction failure.
If the ledger contains the intended page with HTTP 200 but its required price is absent, inspect the content actually handed to the extractor. It may be the wrong document, a page that needs rendering, or a layout the parser no longer recognizes. A successful navigation does not establish a valid record. Preserve that distinction in alerts: fetch_failed and required_field_missing need different evidence and often different owners.
A run can also produce valid duplicate records. Re-fetching everything or adding a browser does not repair an incorrect record key. Compare the normalized key, source URL and retrieval identifier first, then decide whether the defect belongs to discovery deduplication or record deduplication.
Define completion and replay separately
An empty frontier is a stopping condition for the admitted work, not a coverage certificate. For a finite owner-provided item list, reconcile each expected URL with an explicit terminal outcome: retrieved, intentionally excluded under the approved scope, or unresolved with a reason. Keep those categories visible. Removing a failed URL from the queue without recording it would make the job appear finished by deleting evidence of incomplete retrieval.
For open-ended link discovery, an exhaustive denominator may not exist. State the actual boundary instead: permitted origins and paths, the seed set and the discovery rule used for the run. A depth limit, a time window or a deliberate exclusion affects what “done” means. Do not convert that bounded result into “the whole site was scraped.” The permitted scope belongs to the collection owner, and discovering another link does not automatically admit it.
Record acceptance separately for documents expected to yield items. A catalogue index may be a successful crawl input that only generates more requests. An item document may yield multiple offer records. A consumer that needs one current price per sku must reconcile expected keys, not compare the number of output rows with the number of HTTP responses. Preserve HTTP 503 as unresolved retrieval rather than hiding it among pages that legitimately yield no records.
Replay is another contract. If the team retains a body and changes only its parser, rerun extraction against that identified input and record the new schema_version; do not claim the output is a fresh observation from the source. If a consumer needs the current value, the team needs a new authorized retrieval with its own time and fetch_id. Parser freshness and source freshness are different properties.
For delivery, supply both ledgers and their reconciliation summary. The crawl owner signs off on admitted scope and unresolved retrievals; the data owner signs off on required fields, entity keys and rejected records. This makes a partial result usable without presenting it as complete: a consumer can decide whether unresolved items are acceptable for its purpose while retaining the evidence needed for a later retry or parser repair.
Set the collection boundary before fetching
Evaluate applicable robots rules before scheduling automatic requests. RFC 9309 explicitly says those rules are not access authorization; the robots.txt policy and permission distinction keeps that check separate from technical delivery. Review the source's terms and the team's authorization independently, including the intended data use. Neither a crawler label nor a scraper label changes that decision.
Write the permitted origins and paths, stop conditions and required fields before the first fetch. BotBrowser does not grant collection rights or raise a site rate limit. A completed queue is a statement about the work admitted by the team's policy; it is not permission to widen that policy or a guarantee of exhaustive site coverage.
FAQ
Is a crawler always a scraper?
No. A link or availability audit can retrieve pages without extracting business fields. A combined pipeline can do both, but its retrieval ledger and record schema still answer different questions.
Does a scraper need a crawler?
Not when it receives a known authorized URL list or saved content. Discovery becomes a separate requirement when the team must find the inputs, rather than being inherent in field extraction.
Does either job require a headless browser?
Only when its required content or interaction needs a browser. Check whether the needed links or values exist in the delivered response before adding rendering. The presence of JavaScript somewhere on a page does not by itself establish that requirement.
Sources
- Google: overview of crawlers and fetchers
- Scrapy: architecture and data flow, project source
- RFC 9309: Robots Exclusion Protocol
- BotBrowser: getting started
- BotBrowser: responsible use
Practical conclusion
Declare the URL scope and the required record fields as separate contracts. Join them through fetch_id, retain unresolved retrievals and invalid records as distinct outcomes, and reconcile both before accepting the run. When the URL set is too large to fetch in one pass, Crawl budget and politeness basics explains how to rank the queue and pace the requests. Add a browser only where the recorded input shows rendering is needed; keep discovery policy and record validation with their named application owners.
This article is part of the guide: Browser Automation Getting Started: A First Run Map
Related Articles
What Is a Web Crawler? Five Stages of a Crawl
Getting StartedHeadless Browser Scraping: What Actually Changes
Getting StartedWeb Crawling: Crawl Budget and Politeness Basics
Getting StartedScraping with Playwright: Browser and Profile Setup
NetworkScraper Blocked? Diagnose DNS, TLS, HTTP and Page Errors
Getting StartedWeb Scraping JavaScript: Rendering, Pagination, Scroll
Run the same check in BotBrowser
Load a profile, keep the signals you just read about consistent inside the browser, and confirm the result on your own machine before a rollout.