Job Board Scraping: Browser Boundaries

Job board scraping is mostly a pagination and record-integrity problem, and the browser layer owns only part of it. The boundary that matters is access class: a public listing page can be requested under an operator's authorization, while a candidate profile, application history, recruiter inbox, or saved-search account page is private data outside this workload. BotBrowser provides a configurable browser runtime that launches from an operator-supplied configuration and returns the visible page result, but it cannot grant permission, decide whether a site's terms allow collection, or remove an authentication, consent, or rate-limit gate. This plan covers public listings only.

TL;DR

• Treat each job board as three record sources: listing cards, detail pages, and JobPosting JSON-LD. Merge them into one record with an explicit precedence order.
• Define page readiness as a visible result container plus a stable card count, not as a fixed sleep or a global network-idle event.
• Carry job_id, canonical_url, datePosted, validThrough, hiringOrganization, and jobLocation from the first observation instead of reconstructing them later.
• Classify every URL as public listing or authenticated data before the first request; stop the run when a response crosses into the second class.
• Attribute a 429, a robots.txt rule, a consent interstitial, and a 401 to different layers. None of them is evidence that the browser needs a different profile.

Contents

• Map the listing workload to browser duties
• Draw the public and authenticated boundary
• Measure completeness and attribute failures

Map the listing workload to browser duties

A job-board run starts from a query URL such as https://jobs.example.com/jobs?q=analyst&p=1, walks the pagination links the site provides, and opens a detail page per posting. With 40 cards per page and 12 listing pages, the expected ceiling is 480 detail pages, but the browser never decides that number. The operator fixes the query, the permitted page range, and the stop rule; the browser creates a context, navigates, waits for the agreed signal, and reads the page it received.

Splitting duties prevents a common false conclusion. A scheduler owns retries, queue order, and concurrency. Storage owns normalization and deduplication. The browser owns navigation, rendering, readiness, and extraction of what the page exposes. When a detail page returns a card without a title, the missing field is an extraction or markup problem until a status code or an interstitial proves otherwise.

Job-board pages are unusually repetitive: one template serves thousands of postings, filters live in the query string, and the detail layout rarely changes between roles. The per-page cost is therefore dominated by navigation, readiness wait, and parse work rather than by learning a new layout each time. That shape rewards a fixed selector contract and a fixed field contract, and it penalizes ad hoc selectors written per posting.

The same page-count-to-detail-page arithmetic appears in SERP scraping workload planning, which separates browser stages from response wait before a collection target is chosen.

Read the structured posting before the visible card

Many boards publish Schema.org JobPosting markup as JSON-LD. The node usually sits in a <script type="application/ld+json"> block, sometimes inside an @graph array, and sometimes split across several blocks on one page. Read every block, keep parse failures as a counted field, and select nodes whose @type is JobPosting or an array containing it.

Field	Typical shape	Store as	Failure to record
@type	"JobPosting" or an array	node_type	Page has cards but no posting node
datePosted	ISO 8601 date such as 2026-11-03	date_posted	Only relative text such as "3 days ago"
validThrough	ISO 8601 date or date-time	valid_through	Field absent on evergreen roles
hiringOrganization.name	Organization.name string	employer_name	Employer shown as "Confidential"
jobLocation.address.addressLocality	Nested PostalAddress	location_city	Remote role with no address
identifier	String or PropertyValue	source_job_id	Identifier changes between renders

The same field can appear in three places with different values: the listing card, the detail DOM, and the JSON-LD node. Record which source supplied each field instead of letting a later pass silently overwrite an earlier one. A workable precedence is structured data first, embedded page state second, visible DOM last, with the chosen source stored as field_source.

const raw = await page.locator('script[type="application/ld+json"]').allTextContents();
const errors = [];
const nodes = raw.flatMap(text => {
  try {
    const doc = JSON.parse(text);
    return Array.isArray(doc['@graph']) ? doc['@graph'] : [doc];
  } catch (error) {
    errors.push(String(error));
    return [];
  }
});
const postings = nodes.filter(node => [].concat(node['@type'] ?? []).includes('JobPosting'));

This snippet returns the posting nodes and keeps parse errors visible. It does not assume the board publishes structured data; a board that renders cards only still needs the DOM contract, and a board that injects markup after first paint still needs the readiness rule below.

Define pagination from the page, not from a guessed URL

Boards express paging in at least five ways, and the difference changes how a run detects the end. Record the URL that produced each page, the index, and the next link the page itself advertises.

Pattern	Example	Detection	Main risk
Page number	?page=2	Query parameter	Site caps or ignores deep pages
Short page number	?p=2	Query parameter	Same parameter name means a different thing on another board
Offset	?start=25	Query parameter with page size	Offset derived from wrong page size
Cursor	?after=eyJpZCI6MTg0Mn0	Opaque token from the prior page	Reusing a stale cursor
Link relation	<link rel="next" href="...">	DOM attribute	Missing on some templates

Stop when the page advertises no next link, when the result container is present but empty, when the first and last card keys repeat an earlier page, or when the visited-URL set already contains the next target. Treat a page count taken from a "1 of 12" widget as a hint, not as a guarantee: filter changes, personalization, and expired postings can shift the total between requests.

Two guards belong in the loop from the first run. Keep a set of normalized page URLs and refuse a repeat. Keep a signature built from the first and last card keys on each page; identical signatures on consecutive pages usually mean the site clamped the page number instead of returning new results.

Set a readiness condition that matches the extraction

A navigation event proves that a document arrived, not that the listing is usable. DOMContentLoaded can fire before cards render, and a global network-idle wait can stall on ads, analytics, or a long-poll request. Use a condition tied to the elements the extractor reads.

Readiness signal	What it proves	What it misses
domcontentloaded event	HTML parsed	Client-rendered cards
Listing container visible	Template mounted	Cards still streaming in
Card count stable across two checks	List settled	Late JSON-LD injection
JobPosting node present	Detail structured data parsed	Boards without markup
Next control enabled	Pagination interactive	Server-side page clamp

The practical rule is a conjunction: the result container is visible, the card count equals the count the page reports, and every card has a non-empty detail link. For a detail page, add either a parsed JobPosting node or an explicit "no structured data" marker before extraction starts. Record the timestamp of the readiness event separately from the request start so a later review can compute the wait.

Session state belongs to the same contract. Some boards keep filters in the URL and return the same list to a fresh context. Others keep sort order, location radius, or a page cursor in cookies or local storage, so a new context restarts at page one while the URL still says page eight. Decide which behavior applies, then either reuse one context for a run or rebuild the filter state from the URL. A dedicated context per run also keeps storage from leaking between unrelated collection jobs; multi-account browser isolation explains how per-context storage and identity settings are separated when several authorized workloads share one host.

If a run loses its route mid-pagination, do not restart the whole crawl. The session recovery after a network interruption guide covers how to distinguish a dropped connection from an expired page and how to resume without duplicating completed pages.

Draw the public and authenticated boundary

The access class of a URL is a property of the site and the operator's authorization, not of the browser build. A public search result, a public detail page, and a company career page that is served without a session belong to one class. A candidate profile, an application status page, a recruiter message thread, a saved search tied to an account, and an apply form that accepts a resume belong to another. This workload requests only the first class.

That public-data boundary is the same one fingerprint protection for web data collection keeps in scope: browser signals can be made repeatable for an authorized workflow, but they cannot turn an authenticated page into public data.

URL or response class	Example path	Expected response	In scope	Action
Public search	/jobs?q=analyst&p=2	200 with result cards	Yes	Request within the agreed page range
Public detail	/jobs/1842	200 with posting markup	Yes	Extract and record
Public company page	/careers/acme	200 without a session	Yes, if permitted	Treat as another listing source
Login form	/login	200 with a sign-in form	No	Exclude the path
Candidate profile	/account/profile	401 or 403, or 200 only with a session	No	Stop and exclude
Application history	/account/applications	Session-scoped 200	No	Stop and exclude
Recruiter inbox	/employer/messages	Session-scoped 200	No	Stop and exclude
Resume upload	/apply/1842/upload	Form accepting personal files	No	Stop and exclude

Apply the table before the first navigation. A URL allowlist is easier to audit than a blocklist, because a new account route added by the site cannot silently enter the run. Every entry should carry the operator's approval reference, the date it was checked, and the page class it was assigned.

Recognition rules for an authentication boundary

The clearest signal is an HTTP 401 with a WWW-Authenticate header, but real boards also redirect to /login, return 403 for a protected route, or serve a 200 page whose only content is a sign-in prompt. A candidate-only navigation item, an account avatar, or an "apply" control that opens a resume form is a boundary marker even when the status is 200.

When any of those appears, stop that URL, record the class as authenticated, and leave it out of the denominator for public-listing completeness. Never supply credentials, replay a cookie copied from a person's session, complete a challenge, or continue through a page that asks for candidate data. The correct output is an exclusion with a reason, not a partially filled record.

Consent, robots, and rate limits are separate gates

A consent interstitial is a policy gate. The banner may hide the listing until a choice is made, and the available choices may differ by region and by whether a session already exists. Record consent_state as present, required, or absent, capture the URL, and route the decision to the operator. Scripting an acceptance click is not part of a public-listing extraction plan, because the choice is the visitor's, not the collector's.

The Robots Exclusion Protocol defines how a site publishes crawl rules, including groups selected by user-agent token. Fetch robots.txt, record the response status and the rule that applies to the declared client, and exclude a disallowed path. Choosing a different user-agent token to land in a more permissive group is a policy workaround, not a browser requirement.

A 429 response with Retry-After is a rate limit, defined alongside other status semantics in RFC 9110. Honor the window, reduce the requested scope, or ask the site owner for an API. Rotating routes or profiles to keep requesting during the window changes who receives the request, not whether the request is allowed. A 503 is a service condition; treat it as an availability failure and retry only under the operator's agreed policy.

Failure attribution matrix

The matrix below keeps a blocked response from being misread as a browser defect. Each row names the layer that produced the observation and the next permitted action.

Observation	Responsible layer	What it does not prove	Next action
200 but no cards in the container	Rendering, selector contract, or empty result	That the site blocked the browser	Check readiness, selector, and query state; record empty_result
200 with a consent dialog	Consent policy	That the page is unavailable	Record consent_gate and stop for an operator decision
401 plus WWW-Authenticate	Authentication	That another profile would be authorized	Exclude the URL
403 on a candidate route	Authorization	That the route is public but misconfigured	Exclude the URL
Redirect to /login	Authentication	A network or locale problem	Exclude the URL
429 plus Retry-After	Rate limiting	That the browser identity is wrong	Honor the window or reduce scope
Path listed in robots.txt	Site crawl policy	That the HTML is broken	Exclude the path
Challenge page instead of results	Access control	That a rendering option will remove it	Stop; do not automate the challenge
Navigation timeout	Network, host, or page weight	That the site refused the request	Record the stage and recheck the route
200 with fewer than one page of results	Pagination end or filter narrowing	That the run completed the query	Compare the first-page signature and query

A browser profile, a proxy route, or a different locale changes what a server observes. None of those changes creates permission, satisfies a robots rule, accepts a consent choice on a visitor's behalf, or converts a 403 into authorization. A collection is permitted by the site owner, the applicable terms, and the law, and the browser layer can only stay inside or outside that decision.

Network routing is a separate configuration surface. The browser proxy configuration guide covers route selection, credential handling, and DNS behavior; it also makes clear that a route does not validate a destination's policy. Use a route the operator is authorized to use, and keep the route fixed while comparing extraction behavior.

Measure completeness and attribute failures

A useful measurement question for this workload is: on a permitted set of public listing URLs, how many postings were discovered, how many required fields were populated, and which failures came from which layer. That question has a defensible denominator. A "success rate" measured against a login page or a rate-limited response does not.

Freeze the input before the first run. The fixture should list the query URLs, the expected page range, the readiness contract, and the selector contract. Keep an approval reference beside it. If the board changes its template, the fixture changes as a new version instead of being edited in place, so old and new results remain comparable.

Record schema

Store one record per observed posting and one record per page. The page record carries the status, timing, and count. The posting record carries the identity and the field sources, as in this example with illustrative values.

{
  "job_id": "1842",
  "canonical_url": "https://jobs.example.com/jobs/1842",
  "page_url": "https://jobs.example.com/jobs?q=analyst&p=2",
  "http_status": 200,
  "observed_at": "2026-11-07T14:30:00Z",
  "date_posted": "2026-11-03",
  "valid_through": "2026-12-01",
  "employer_name": "Example Analytics",
  "location_city": "Toronto",
  "field_source": "jsonld",
  "etag": "W/\"abc123\"",
  "content_hash": "9f2c1d7a4b8e0c31"
}

Keep job_id as the site's own identifier when it is stable. Keep canonical_url as the address the page declares, not the paginated URL that led to it. Keep page_url so a duplicate can be traced to the listing page that produced it. Store etag and content_hash to detect unchanged detail pages and to avoid rewriting identical records.

Merge duplicates with a declared key

The same posting often appears on a board's search page, a company page, a regional mirror, and an aggregator. Job identifiers are the strongest key when they exist, but a re-post can receive a new identifier while keeping the same employer, title, city, and description.

Candidate key	Stability	Collision risk	Use when
source_job_id plus board host	High while the posting lives	Same id reused after deletion	Board publishes a stable id
canonical_url	High	Two URLs for one requisition	Canonical link is present
Employer, title, city, date_posted	Medium	Same role opened twice on one day	Id and canonical URL are missing
Employer, title, city, description hash	Medium	Identical template text across teams	Description is substantive
Title and city only	Low	Many genuine openings share a title	Never, except as a review hint

Merge on the strongest available key, then keep the weaker matches as variant_urls rather than deleting them. A merged record should retain every URL that contributed to it, the first and last observation times, and the source that supplied each field.

Duplicate decision matrix

Situation	Merge	Reason
Same canonical_url, different page_url	Yes	One posting reached through two listing pages
Same employer, title, city, and date_posted; different job_id	Yes, with variants	Likely the same requisition syndicated twice
Same employer and title, different city	No	Location is part of the role
Same employer, title, and city, date_posted more than 30 days apart	No, unless description hash matches	A later opening can be a genuine new role
valid_through has passed but the page still returns 200	Keep, mark expired	The page was not removed on schedule
Aggregator copy and company career page	Merge to the company record when both are authorized	The source page is the better canonical target
Redirect chain ends at a new canonical URL	Merge and store the chain	The old URL no longer identifies the posting

Run plan and timing marks

The plan below uses a site contract object so the selectors and readiness rule are named in one place. The names are illustrative; replace them with the contract agreed for the permitted site.

const contract = {
  list: "[data-testid='job-card']",
  detailLink: "a[data-testid='job-link']",
  next: "a[rel='next']",
};

const started = performance.now();
const response = await page.goto(listUrl, { waitUntil: 'domcontentloaded' });
await page.locator(contract.list).first().waitFor({ state: 'visible' });
const cards = page.locator(contract.list);
const count = await cards.count();
await page.waitForTimeout(250);
const stabled = await cards.count();
if (count !== stabled) throw new Error('card count changed before extraction');
const links = await cards.locator(contract.detailLink).evaluateAll(nodes => nodes.map(node => node.href));

Record request_start, response_received, ready_signal, and extract_done as separate timestamps. Compute readiness wait as ready_signal - request_start and extraction time as extract_done - ready_signal. If the run reports one duration for the page, a slow selector and a slow server become indistinguishable. The performance optimization documentation covers runtime tuning for an existing deployment; the marks above describe what that tuning should be compared against.

Use three repetitions of the same fixture, and report the minimum, median, and maximum for each stage instead of the best run. Keep failed pages in the denominator. A page that timed out before readiness counts as an attempted page with a stage failure, not as an absent posting.

Completeness and field coverage

Completeness compares discovered postings with the postings the permitted query reports. If page one says 480 results at 40 per page, and the run extracts 462 unique postings after merging, report 462 / 480 plus the duplicate count and the exclusion count. Do not silently raise the numerator by merging unrelated roles or lower the denominator by dropping pages that failed.

Field coverage is a separate ratio per field. Report date_posted present in 440 of 462 records, valid_through in 301, and jobLocation in 455. A board can have complete pagination and poor field coverage, or the reverse. The two numbers answer different questions, and a single "success" figure hides which one failed.

Attribution decision matrix

Pattern in the record	Interpretation to check	Next measurement
Cards present, JSON-LD absent on every detail page	The board may not publish markup	Confirm on a second template version before changing extraction
Readiness wait grows while status stays 200	Client rendering or script weight changed	Compare ready timestamps at the same fixture version
200 pages shrink in count without a query change	Pagination cap or filter state	Recheck page one and the next-link chain
429 appears only at higher concurrency	Request rate exceeded the site's limit	Lower concurrency or request an API
403 appears only on account paths	Authentication boundary reached	Exclude the path and review the allowlist
Duplicate rate rises after a template change	New URL or id pattern	Compare merge keys across versions
Field coverage drops but card count holds	Markup or selector drift	Compare field_source by field

Pre-run checklist

• Every seed URL is a public listing or public detail page, with an approval reference.
• Account, apply, resume, message, and saved-search routes are excluded by allowlist.
• robots.txt has been fetched and the applicable rule recorded.
• The readiness condition is written as a visible element plus a stable count.
• The pagination stop rule includes visited URLs and a repeated-page signature.
• The merge key and the weaker fallback keys are declared before extraction.
• Page-level status, timing, and count are stored beside posting records.
• Failed and excluded pages stay in the denominator with a reason code.
• The run stops on a 401, 403, consent gate, challenge page, or 429.
• The run record names the browser build, the route used, and the fixture version.

FAQ

Is JobPosting JSON-LD enough to identify every listing?

No. Coverage varies by board and by template, and some pages render cards from an internal state object with no structured data block at all. Use JSON-LD as the preferred source, keep a DOM contract as the fallback, and store field_source so a coverage drop is visible. A page with cards and no posting node is a markup observation, not an access failure.

Should pagination stop at the first empty page?

Stop and then verify. An empty container can mean the query is exhausted, a filter narrowed the set, or the page number was clamped. Compare the page signature, the advertised total, and the next link before recording the run as complete. A single empty page is a stop condition for requests, not proof that the query returned nothing.

Does a browser profile decide whether collection is allowed?

No. Permission comes from the site owner, the applicable terms, and the law. A profile changes observable browser signals, and a proxy changes the route, but neither satisfies a robots rule, a consent choice, or a rate limit. The access-class table and the failure matrix are the decision surface; browser configuration is downstream of that decision.

Practical conclusion

Start by classifying every URL as a public listing or authenticated data, and remove the second class before any request. Then define the readiness signal, the pagination stop rule, and the merge key as one written contract. Measure completeness and field coverage separately, and attribute each failure to the status, policy, or rendering layer that produced it.

The next action is small: write the allowlist and the approval reference for one permitted public job-board query, then confirm that the first page returns 200 with either a JobPosting node or the agreed DOM contract. Everything else in the run depends on that boundary holding.

Sources

• Schema.org: JobPosting
• Google Search Central: Job posting structured data
• Playwright: Browser contexts
• Playwright: Page API
• RFC 9309: Robots Exclusion Protocol
• RFC 9110: HTTP Semantics
• BotBrowser: Responsible use
• BotBrowser: Playwright getting started