Job Posting Scraping: Titles, Reposts and Closure

Job posting scraping becomes a data-model problem once more than one board supplies the same role. Equal titles do not identify one job, a new URL does not prove a new vacancy, and a row that stops appearing does not prove the employer closed it. BotBrowser provides documented profile-backed browser inputs for an authorized observation; it does not supply job data, log in for you or grant access to a board. Choose a permitted source, preserve the raw fields beside any normalization, and keep each posting occurrence and observation outcome explicit.

TL;DR

• Keep raw title, employer and location strings beside normalized values, with a mapping version.
• Key a record by the source's posting occurrence. A repost needs source evidence of continuity before it replaces the earlier occurrence.
• Treat datePosted, validThrough and observation time as different facts with different meanings.
• A failed refresh is an incident. Absence from a bounded list is an absence, not proof that a vacancy closed.
• Prefer an official feed or approved export where one exists. Vocabulary and structured-data eligibility do not authorize collection.

Contents

• Separate the three normalization problems
• Preserve raw fields and mapping versions
• Decide when a repost is a new occurrence
• Detect closure without inventing it
• Use a minimal ingestion schema
• Choose an approved source before a browser route
• Practical conclusion

Separate the three normalization problems

Cross-board job data needs three different decisions, and merging them into one cleaning step loses the evidence needed later.

Title. “Senior Engineer”, “Sr. Engineer, Platform” and “Engineer III” may describe comparable seniority or different scopes. Normalize a separate field for a declared purpose, such as a coarse role family, while retaining the source title. Do not overwrite the title with the normalized label.

Location. “Remote”, “Remote, US”, “New York, NY (Hybrid)” and a city name in a free-text field answer different questions. Keep the raw location text, then record the fields the source actually supplies: country, region, city, remote scope and work arrangement. A missing country is not permission to infer one from a browser's language or proxy region.

Employer. A brand, legal entity, staffing agency and board-specific employer label are not interchangeable. Preserve the source label and any source-provided organization reference. Similar names can belong to different entities, and one employer can post through several channels.

Record the three mapping decisions as separate fields such as title_family, location_parts and employer_ref, each with its own rule version. A single cleaned_title or city field hides which part was supplied and which was inferred. Keep the raw strings next to those fields, and keep an explicit unknown state when the source does not answer the question. This makes a later comparison reviewable without reconstructing the original response.

The job field contract establishes how one board exposes fields and observations. This pipeline begins after that source contract and adds a cross-board record that remains explicit about what was mapped.

Preserve raw fields and mapping versions

Normalization is an application decision, so the ingestion record should show both the input and the rule that transformed it. The schema.org JobPosting vocabulary describes a listing for a job opening and defines fields such as datePosted as the publication date of an online listing. It supplies meanings to compare, not a universal parser for every board.

Record field	Preserve	Boundary
Source	Board or feed identity and dataset scope	Equal values from different sources are not automatically the same posting
Source posting key	Provider-defined identifier when actually supplied	A URL or title is not automatically a stable posting key
Raw title	Exact source string within permitted retention	Do not replace it with a normalized label
Raw location	Exact source location text and supplied structured parts	Do not infer a missing country from locale or proxy
Employer label	Source-provided employer or organization text	Brand, legal entity and agency remain distinct unless evidence resolves them
Normalized fields	Value, rule identifier and mapping version	A mapping change starts a new interpretation, not a silent rewrite
Publication and expiry	Source datePosted and validThrough when supplied	Neither establishes the actual fill date or a confirmed closure
Observation	Collection time, source state and acceptance outcome	Collection time is not publication, expiry or transaction time

The Google JobPosting documentation describes structured data that can make a posting eligible for a job-search experience, and lists example fields including identifier, datePosted, validThrough, employmentType, hiringOrganization and jobLocation. Eligibility is not a guarantee of accuracy, access permission or identical markup across boards. Use the vocabulary to name fields, then map only what the approved source actually returns.

Keep the mapping rule versioned. If “Senior” moves from one role-family bucket to another, an old record should not look as though it always used the new rule. A normalized series can be rebuilt from raw values and a recorded rule; overwriting raw fields destroys that ability. Retain only what the source agreement permits, and keep the raw values inside the same approved retention boundary as the normalized record.

This matters most when two boards describe the same role in different formats. One may provide a structured city and country, another may put both in a single text field, and a third may omit the country while naming a metro area. A normalized comparison can still be useful, but it needs a mapping rule that says what was inferred and what was supplied. Record the source field path or response location for each mapped value, so a reviewer can see whether a country came from a provider field or from an application guess. When the source supplies no country and the mapping needs one, the normalized value should remain unknown. Do not substitute the browser's locale, the proxy's region or the page's interface language. Those describe the observation environment, not the job.

Employer normalization needs a similar audit trail. A board may label the same organization with a display name, a legal name or a recruitment agency. A source-provided organization identifier is stronger evidence than a name match, but it is still scoped to that source. Record the label, the identifier and the rule that connected two records. Keep a low-confidence relation visible instead of collapsing it into a single employer row. That preserves the ability to compare counts by source and to withdraw a match when new evidence shows two different organizations.

The normalized fields should also carry a confidence or resolution state when the mapping cannot be decisive. A role-family label can be mapped, a location can be partial and an employer can be unresolved. These states are more useful than an empty string, because they distinguish “not supplied” from “not yet interpreted”. Keep the same states in exports and reports so a downstream analyst does not silently treat an unresolved value as a clean one.

Decide when a repost is a new occurrence

A board can relist a role with a new URL, a new internal identifier or a changed title. None of those changes alone proves a new vacancy, and none proves continuity either. Keep the occurrence separate unless the approved source establishes a relationship.

Constructed observation	Source evidence	Accepted relation
Same key, title revised	Source posting key unchanged; source supplies the update	Same occurrence with an observed field revision
New URL, same visible employer and title	URL changed; no source continuity field	Unresolved relation; retain as a separate occurrence pending evidence
New source key, source marks a replacement	Source explicitly relates the new posting to the old one	Related occurrence; preserve both keys and the relationship
Same title and city on another board	Different source and no shared source key	Different source occurrence; do not merge by title and place

The table is constructed, not a measured board behavior. The real-estate listing model applies the same discipline to property listings: a listing occurrence, the physical property and an offer are distinct. For jobs, a source posting occurrence, the employer's role and a candidate's application are similarly distinct.

Use a source-scoped key when one exists, and record why a match was accepted or left unresolved. Fuzzy matching on title, employer and location can propose a candidate pair, but a proposal is not proof of continuity. Preserve both source keys and the relation label. When a deduplication step merges records, record the rule and the evidence so a later audit can separate two genuinely different vacancies that happened to share a title.

A relation label needs to be stable enough to survive a later refresh. Record whether the relation is same_occurrence, related_occurrence or unknown, and store the evidence behind that label: a shared source key, an explicit replacement field or a source-provided canonical reference. Do not derive the label only from the current normalized title. A title edit can happen inside one posting, while an unchanged title can accompany a genuinely new opening. The relation describes source continuity, not textual similarity.

When a board supplies both a posting identifier and a job identifier, keep them in separate fields. The posting identifier can change on a repost while the job identifier remains stable, or the reverse depending on the source model. Recording only one of them loses the distinction that the dedup decision needs. If the source provides neither, keep the URL and observation time as evidence, but mark the occurrence provisional. Provisional records can still be useful for a bounded analysis; they should not be merged into a permanent canonical role without additional evidence.

Store the source reference alongside the occurrence key rather than replacing it. URLs can change through tracking parameters, locale prefixes or a board migration, and an old URL may later redirect. Preserve the reference that was actually observed and a normalized comparison form separately. A redirect can show a relationship worth investigating, but it does not by itself prove that the target is the same posting. Treat that as a source question, not a URL-string rule.

Detect closure without inventing it

The Google expiry guidance says postings no longer open for applications must be expired, and names ways a publisher can do that: populate validThrough in the past, remove the page with a 404 or 410, or remove the JobPosting structured data. That describes a publisher's obligation to a search experience. A collector observing a missing card or a changed response has not necessarily observed any of those events.

Constructed event after the baseline	Observation	Accepted interpretation
Accepted field update	Same source key; source supplies revised title or location	Observed revision; no new employer or vacancy claim
Repost with new URL	New source reference; no continuity evidence	Separate occurrence with an unresolved relation
Refresh fails	No accepted source record	Collection incident; do not delete the prior accepted value
Card absent from a bounded list	Valid list window; target not observed	Bounded absence; closure remains unestablished

datePosted describes publication of an online listing, validThrough describes a validity boundary, and the collector's observation time records when the application looked. A passed validThrough can support an expiry label if the source meaning is known, but it does not show when a role was filled. A missing row in one list can support “not observed in this window”, not “closed”. A challenge page, timeout or parser rejection is an incident, not an empty successful result.

An owned fixture can replay the four rows without collecting a real board. Require one accepted revision, one separate repost occurrence, one refresh incident and one bounded absence. Reject any output that converts all four into closures. This checks the application's classification logic; it does not measure a particular board's update cadence or prove that a vacancy is open.

Closure detection also needs a time boundary. A posting may disappear from a search result because the query window moved, the list was truncated or the board changed its ordering. Record the query or feed scope, the requested page range and the observation time. If the collector only looked at the first page, absence from that page is not absence from the board. If a list changed order between observations, a missing item can be a pagination artifact. Keep these outcomes separate from a source-provided expiry state.

The source's own state has priority over an application inference. If the approved source returns an explicit closed or expired state, record it with the source field name and observation time. If it returns a removal response, record that response separately from a successful empty list. If it returns nothing usable because of a challenge, timeout or malformed body, record an incident. The application can then produce a lifecycle view that says “source-confirmed expired”, “not observed in this window” or “unknown”, instead of reducing every disappearance to one closure label.

Historical records should preserve the evidence that was available at the time. A later source contract may add a status field, but that does not retroactively turn an old absence into a closure. Keep the observation outcome and the source schema version together. When the schema changes, start a new interpretation series or reconcile the old records explicitly. This is the same discipline used for normalized fields: a change in the rule must not look like a change in the world.

Use a minimal ingestion schema

A small record keeps the decisions reviewable:

source_namespace, source_posting_key, raw_title, raw_location,
raw_employer_label, employer_ref, normalized_title, normalized_location,
normalization_rule_version, date_posted, valid_through,
query_scope, page_range, source_schema_version,
observed_at, observation_outcome, relation_to_prior_occurrence

The schema is an application record, not a required board format. A missing field stays null or unknown rather than receiving a guessed value. Keep the source response outcome separate from the interpreted lifecycle. If two observations share an occurrence key, their field revisions can be compared; if they do not, the relation remains unresolved until the source supplies evidence.

The job-board access boundary separates public listing pages from authenticated candidate data. An ingestion schema does not change that boundary. Candidate profiles, applications and messages are outside the collection scope even if a browser session could reach them. Prefer an official feed or approved export when one exists, and stop where permission or required evidence is unavailable.

Choose an approved source before a browser route

RFC 9309 states that robots rules are not a form of access authorization. A public page that permits crawling still may not permit building a market dataset or redistributing postings. The applicable board terms and a real data agreement decide what may be collected and retained. Do not treat a successful browser response as permission.

The BotBrowser timezone, locale and language documentation describes browser-facing locale, timezone and language inputs with automatic detection and manual overrides. Those settings can keep an authorized observation environment explicit. Automatic detection is documented; manual overrides require an ENT Tier1 license. They do not establish the location of a job, the country of an employer or the meaning of a board's location text.

The profile configuration reference documents configurable profile packages, configuration priority and field availability. A profile can make declared browser inputs reproducible. Profile packages also record a matching major-version requirement, but a profile cannot log in, grant a data licence or guarantee that a board returns the same listings between runs. Record the profile and observation conditions beside accepted records, and keep the source agreement as the authority for access and retention.

FAQ

Can two boards' identical titles be merged into one job?

Only with source evidence that they describe the same posting or role. Equal text is a candidate for review, not proof. Preserve each source occurrence and an explicit unresolved relation until continuity is established.

Does a posting that disappeared from a list mean the job closed?

No. A bounded list can omit a posting for many reasons, and a failed refresh is not an empty result. Record the absence and require source evidence, such as an expiry state or removal response, before assigning a closure.

Should datePosted be used as the collection time?

No. It is the publication date of an online listing. Keep the collector's observation time and the source's datePosted and validThrough as separate fields with their own meanings.

Does BotBrowser supply job listings or permission to collect them?

No. It provides documented browser configuration inputs. It does not provide job data, log in, grant access or establish a board's terms; use an approved source and retain only what the agreement permits.

Sources

• schema.org JobPosting
• Google JobPosting structured data
• RFC 9309
• BotBrowser timezone, locale and language
• BotBrowser profile configuration

Practical conclusion

Build a permitted job record around source-scoped posting occurrences. Keep raw and normalized values together with a mapping version, separate reposts until the source establishes continuity, and treat publication, expiry and observation time as different facts. Classify failed refreshes and bounded absences without inventing closures. Browser inputs can make an authorized observation reproducible; the approved source and its agreement determine what data is available, what it means and what history may be retained.