Scraping Indeed Jobs: Fields, Pagination and Duplicates

Scraping Indeed jobs requires an authorized source before a parser or browser can produce a useful employment dataset. A posting's title, organization, location, dates and source identity need a field contract; moving result pages also need a way to distinguish repeated observations from different jobs. The same evidence questions apply to an authorized LinkedIn source, but neither platform's visible pages grant general collection permission. BotBrowser provides declared browser locale and session inputs; it does not grant board access, supply job postings or guarantee stable result ordering. Preserve what the source actually establishes, and keep missing fields or list-page absence separate from employment facts.

TL;DR

• Check the applicable board terms and the actual authorized source. Public visibility and structured markup are not blanket collection permission.
• Define each required field, its source and the meaning of missing data before parsing cards or detail pages.
• JobPosting is a vocabulary, not proof that Indeed or LinkedIn exposes a particular field or markup in a given response.
• Moving offset pages can repeat or skip records in a constructed changing-list example. That example does not establish either board's implementation.
• Deduplicate source postings with source-scoped identity, retain observation history, and require separate evidence before calling a job closed.

Contents

• Establish the permitted access
• Define the job field contract
• Separate structured and rendered observations
• Explain page overlap with a changing-list fixture
• Deduplicate postings without merging different jobs
• Keep disappearance separate from closure
• Practical conclusion

Establish the permitted access

The current Indeed terms address automated access directly. Their automated-systems provision requires express written permission and includes a conditional crawl provision solely as outlined in the site's robots file. Read the actual applicable scope rather than converting that clause into a general data-reuse license. A permission to perform one kind of crawl does not establish permission for another collection purpose, a different interface or account information. An employment-data application should identify its approved integration, feed or expressly permitted observation before choosing a parser. The Google Maps place-data comparison applies a different provider contract: requested API fields, changeable place IDs and content-retention limits remain separate from rendered interface observations. The real-estate listing model separates source listing occurrences from physical properties and compares asking prices only when currency and transaction meaning match; a missing card does not establish a sale. The job-market ingestion model keeps raw and normalized fields, source-scoped posting occurrences and observation outcomes separate, so a repost or a missing card is not converted into an invented closure.

The LinkedIn user agreement separately prohibits software, scripts and other automated means for scraping or copying the service, and unauthorized automated access. Do not assume the conditions are identical to Indeed's or that a publicly readable listing overrides them. Where the needed permission is unavailable, use an approved source or return that limitation. This source decision does not require a login workaround, a candidate-profile collector or an attempt to change client identity after refusal.

RFC 9309 explicitly states that robots rules are not access authorization. A site's separate contractual provision may refer to robots conditions, as the Indeed text does; that requires reading the actual provision and its limits rather than claiming the standard grants rights. Preserve the permission basis and the intended data scope as separate inputs to the collection plan. Do not infer redistribution or retention rights from a successful HTTP response. Technical readability and the approved use remain different questions.

The job-board browser-layer discussion owns the broader access and browser boundaries. A field contract adds a narrower question after source approval: what does this particular permitted observation establish about a posting? That contract should say whether the application uses an approved feed, supplied fixture or separately authorized page observation. Record that choice without retaining reusable account state. If the source cannot answer the required field question, return the missing evidence instead of treating browser automation as an alternative permission mechanism.

Define the job field contract

The JobPosting vocabulary defines properties such as the posting title, hiring organization, location and publication date. Those definitions help describe a record, but they do not guarantee that a board supplies every property. State each required field and its actual source independently. A title copied from a card, an employer name from a detail page and a date from structured data are separate observations until the application establishes that they belong to the same source posting. Do not join them merely because their text looks plausible.

Field group	Evidence to preserve	Interpretation boundary
Posting identity	Source namespace and actual posting identifier when available	A title is not a source identifier or a global vacancy key
Role and organization	Attributed title and hiring organization	A similar employer name does not prove the same posting
Work location	Observed location and any explicit remote-work meaning	Browser locale is not the job location
Publication and validity	Source publication date and validity value when supplied	Collection time is not publication time or a closure date
Compensation and type	Actual amount, currency, period and employment type if supplied	Missing salary is not zero; a display range is not one fixed amount
Observation provenance	Source representation, observation time and collection outcome	A failed refresh is not a changed employment fact

The schema reference gives datePosted the meaning of publication date for an online listing, while validThrough describes when an item ceases to be valid. Keep those meanings distinct from the collector's own observation time. A record first encountered today may have an earlier publication date, and the absence of a validity field leaves that field unknown. An application should not create a validity deadline from the latest observation or from a general expectation about how long postings remain open.

Location needs the same care. The vocabulary includes jobLocation and jobLocationType, but a browser's configured region does not establish either. Preserve the actual source statement, including an explicit remote-work designation where available, rather than turning missing geography into worldwide eligibility. The observation pipeline used in price monitoring separates the source record from downstream analysis. Keep the field's meaning and unknown state visible at that boundary so salary, location and posting-age statistics do not silently incorporate invented defaults.

Each table row is an application requirement, not a claim about current Indeed or LinkedIn output. If a source lacks a required field, keep an incomplete record or reject it under the application's documented rule. Optional information can remain absent without making the posting invalid for every use. The important distinction is between a missing source value, an unsuccessful attempt to read it and a source value that genuinely changed. Those outcomes should not collapse into the same empty string in an accepted dataset.

Separate structured and rendered observations

Google's JobPosting guidance instructs publishers to put this structured data on the specific page describing a single job, together with the visible description. It does not establish that every search-results page exposes a complete JobPosting object. An authorized list observation may contain a summary card while the permitted detail source contains fuller information. Establish the actual representation supplied before assuming that a field missing from a card is absent from the posting itself.

MDN's script type reference describes non-executable data blocks in HTML when an appropriate non-JavaScript MIME type is used. This explains how structured data can exist without executing it. It does not prove that a particular Indeed or LinkedIn response includes such a block, nor that the block is complete or current. A saved authorized response can establish whether the representation was present in that observation. Source attribution must stop there unless additional evidence supports a broader claim.

When both structured data and rendered content exist in a permitted fixture, retain their source types instead of silently choosing whichever field is easier to parse. A structured title can differ from shortened card text without implying another job. A rendered compensation phrase may carry context that a separately extracted amount loses. Determine which value answers the defined field question and preserve the matching rule. If the representations cannot be reconciled for the intended posting, mark the field unresolved rather than combining fragments into a supposedly complete record.

The JobPosting vocabulary can guide interpretation of an actual property, but it is not a board-specific selector contract. Do not infer a private endpoint, hidden identifier or stable HTML structure from the vocabulary. For an authorized parser change, preserve a minimal redacted sample and the interpretation rule it exercises. Keep unrelated account or candidate information out of the fixture. A source-format change should produce a parser-version decision and validation outcome, not an unexplained change in employment statistics.

The rendered route has a bounded purpose: observing text or state that the approved question requires and the approved response does not supply. It does not repair an unavailable field by itself. If a required value remains unattributable to the intended posting, record that limitation. Running a browser can establish the permitted interface state observed in that run; it cannot establish universal board coverage, permission to expand scope or the correctness of every field in the source.

Explain page overlap with a changing-list fixture

Offset pagination can produce overlap when the underlying ordered list changes between observations. That is a property of the constructed fixture below, not a measured assertion about Indeed or LinkedIn. The fixture assumes a page size of three and an insertion before the second request. Posting labels represent source identities in an owned test list, not real listings. Its purpose is to separate page position from posting identity before choosing a deduplication rule.

Fixture step	Ordered list at that time	Observed page	Consequence
Initial request	A, B, C, D, E, F	A, B, C	First page observes three identities
Insert X at the front	X, A, B, C, D, E, F	No page requested yet	Existing records shift position
Next offset request	X, A, B, C, D, E, F	C, D, E	C repeats even though the offset advanced

The repeated C is not a newly created job, and its second occurrence need not be discarded as useless. It is another observation of the same fixture identity at a later time. An analogous removal before the next offset could instead cause an unobserved entry to be skipped. These examples illustrate why a completed sequence of page requests does not prove a stable snapshot. They do not establish that either board uses offsets, preserves a particular order or offers a supported snapshot cursor.

The pagination contract depends on the selected source. Google's JobPosting guidance describes publishing individual job details, not a universal pagination contract for job boards. Use the approved integration's actual documented navigation model if it has one. If an authorized source supplies no snapshot guarantee, state that the collection is a sequence of observations over time. Do not create a guarantee by adding browser retries, fixing a locale or treating a displayed result count as a verified population size.

In an owned pagination test, acceptance should check the preserved identities and observation times, not only the number of rows returned. The synthetic insertion must produce one repeated identity with two observations, rather than two distinct jobs. A missing record in a later list should remain unobserved there rather than automatically closed. Keep the fixture and its expected interpretation separate from production source evidence. That makes the rule testable without making unsupported claims about protected services or collecting real job-seeker data.

Deduplicate postings without merging different jobs

Choose a source-scoped posting key when the authorized source supplies a usable identifier: the source namespace plus that identifier. This is an application identity rule, not a promise that every board provides such a field. The JobPosting identifier definition is general and does not establish a globally unique employer vacancy identifier. Retain the actual source and identifier semantics instead of assigning one shared key to every posting with similar title and organization text.

Cross-board matching is a separate inference. An employer may publish similar roles, reuse titles or advertise multiple locations. A pair of similar records does not prove one vacancy syndicated to two boards. Keep independent source posting identities even if a later analytical process proposes a match. Preserve the matching evidence and uncertainty rather than deleting one source as a duplicate before establishing what duplicate means for the application. Source occurrence identity and real-world vacancy identity answer different questions.

If the approved source lacks a reliable identifier, an attributed source URL may be a candidate reference, subject to the source's actual URL semantics. Do not normalize away parameters or path components without evidence that they are irrelevant to posting identity. If neither reference establishes a stable key, retain a provisional identity and mark the uncertainty. A hash of mutable description text can help compare observations, but changing text does not necessarily create a new vacancy. Likewise, identical text does not prove two source records are one posting.

The shared observation pipeline can keep posting entities and observation history separately. Deduplication should consolidate repeated occurrences under the chosen source key while retaining observation time, source representation and collection outcome. The changing-list fixture then produces one C entity with two observations. A new description under that key is a potential revision to inspect, not automatically a new job. A failed refresh should remain an incident attached to collection, not replace the last accepted description with a blank record.

Browser inputs do not supply that identity. BotBrowser's geographic settings documentation defines profile prerequisites, automatic proxy-derived settings and manual overrides requiring ENT Tier1. Those are inputs for an authorized locale-sensitive observation. They do not establish the posting's location, stable ranking or cross-session identity. Record the intended settings and actual loading evidence separately when a runtime observation is performed. No real-board runtime experiment is claimed by the field or pagination rules.

Keep disappearance separate from closure

A posting missing from one result page can mean that the observed list changed, the relevant page was not reached, the response was incomplete or the source no longer returned that posting. List absence alone does not choose among those explanations. The Indeed terms expressly reserve the ability to exclude or remove job ads from search results, so inclusion in a list is not a posting-lifecycle guarantee. Google's JobPosting guidance gives publishers several ways to expire unavailable jobs: a past validThrough value, removal of the page with an unavailable status, or removal of the structured data. These are publisher-side options, not proof that every board applies them immediately or identically.

Preserve lifecycle evidence with its scope. A source explicitly stating that applications are closed is different from a collector failing to find a card. A past validity value has the meaning supplied by the source; a transport failure has no equivalent employment meaning. If the application needs a closure state, define the evidence it accepts and retain an unknown state where that evidence is absent. An access refusal, timeout or incomplete page traversal cannot establish that a vacancy was filled.

Drift handling should therefore distinguish representation changes, field changes, repeated list occurrences and lifecycle evidence. Compare permitted samples against the field contract before revising the parser. The schema vocabulary helps retain the meanings of publication and validity, while the collector owns its observation and processing timestamps. Reprocessing a historical fixture is a new processing event, not a new observation of a live vacancy. Preserve that distinction so a repaired parser cannot make old jobs appear newly collected.

The Indeed terms also state that the service does not guarantee the accuracy or availability of employer-submitted content. An authorized source is not a guarantee of underlying employment truth. Keep provenance so downstream users can distinguish the source's statement from the application's interpretation. Publish a comparable, sufficiently supported observation or retain its specific limitation. Browser configuration, successful rendering and page-count completion do not remove the need for that evidence.

FAQ

Does JobPosting prove either board serves that structured data?

No. It defines a vocabulary. Only the actual authorized response establishes whether that representation and its fields were present in an observation.

Is a repeated listing on another page a new job?

Not from page position alone. Apply the source-scoped identity rule and preserve the separate observation time; cross-board vacancy matching remains a different inference.

Can a missing listing be marked closed immediately?

Not on that basis alone. List absence, failed collection and explicit lifecycle evidence are different outcomes and need different states.

Can changing browser locale make job results stable?

No. It changes browser inputs, not the source's ordering contract or posting identity. Required location and lifecycle facts need source evidence.

Sources

• Schema.org: JobPosting
• Google: JobPosting structured data
• MDN: script type
• BotBrowser: Timezone, locale and language
• RFC 9309: Robots Exclusion Protocol
• LinkedIn: User agreement
• Indeed: Terms

Practical conclusion

Choose an authorized job-data source, define the required fields and keep source-scoped posting identity separate from page position and cross-board vacancy matching. Preserve structured or rendered provenance, observation time and unknown values. BotBrowser can provide declared browser inputs, while the source must establish job facts and the applicable access scope. Treat overlap as repeated observations and require lifecycle evidence before calling a missing listing closed.