Twitter Scraping: X API Windows and Browser Limits
Match X Posts, mentions, search and home feeds to the right endpoint, history window and authentication. Separate embed HTML and ranked browser views from API data.
BotBrowser Team
Want the structured docs for Getting Started?
This article lives in the editorial library. For step-by-step setup, reference material, and ongoing updates, jump into the docs section.
Twitter scraping requirements often fail before the first selector: the requested history or ordering does not match the chosen X endpoint. Choose the data product, authentication context and time window first. BotBrowser supports Chromium integration with Playwright and Puppeteer for separately authorized browser checks; it cannot grant X API access, extend endpoint history, reproduce algorithmic ranking or remove platform limits. The browser setup workflow belongs after that data decision, when a permitted browser task actually remains.
TL;DR
- An authored-Post timeline, mentions feed, chronological home timeline and search result are different datasets with different documented bounds.
- Recent search covers the last 7 days; the full-archive product reaches back to March 2006, subject to its access requirements.
- A reverse-chronological home endpoint does not reproduce algorithmic ranking. An embedded timeline returns display markup, not a historical Post export.
- OAuth 2.0 access-token expiry, API request accounting and browser-session lifetime are separate questions. More browser sessions do not repair a missing data entitlement.
Contents
- Match the requirement to an X result
- Work three requests through the history boundary
- Keep authentication and lifetime attached to the route
- Separate request limits from browser observations
- Practical conclusion
Match the requirement to an X result
The X timeline documentation separates authored Posts, mentions and the authenticated user's reverse-chronological home feed. A client that needs replies mentioning its account is not asking for its own authored timeline. A product that needs what a person saw in a ranked home view is not asking for the same order as the chronological endpoint. Preserve that distinction in the request specification instead of treating every surface as “tweets.”
| Required result | Candidate official route | Documented boundary and choice rule |
|---|---|---|
| Posts authored by one account | /2/users/:id/tweets | Up to 3,200 most recent Posts; filters and time parameters do not establish a lifetime archive |
| Posts mentioning one account | /2/users/:id/mentions | Up to 800 most recent mentions; do not substitute the account's authored Posts |
| Authenticated user's chronological home feed | /2/users/:id/timelines/reverse_chronological | Most recent 3,200 Posts or 7 days; requires user authentication and excludes algorithmic ranking |
| Query matches in the recent window | /2/tweets/search/recent | Last 7 days, up to 100 per request; verify developer/app access and query support |
| Query matches in an older period | /2/tweets/search/all | Archive back to March 2006, up to 500 per request; check current pay-per-use or Enterprise access |
| Display a Post or supported user/list timeline | https://publish.x.com/oembed | JSON containing embed HTML; timeline display limit 1-20, not a structured history export |
| Exact algorithmically ranked home order | No equivalent established by these references | Stop or narrow the requirement; browser visibility does not establish a stable extraction contract |
These numbers are documented ceilings or windows, not promises that every request returns that many records. Availability, filters, authentication and the endpoint's current contract still matter. The search overview describes recent search as available to all developers, but also lists developer-account, app and token prerequisites. “All developers” is not an unauthenticated public endpoint or a guarantee that an arbitrary project is ready to run it.
Current documentation also needs a date-aware reading. The API introduction describes pay-per-usage without subscriptions. Some timeline and pagination text still says “standard tier.” Those labels do not justify resurrecting an old fixed-tier comparison. Verify the currently offered product and the app's actual access before budgeting a job; request-rate limits and usage charges answer different questions. No price or subscription plan is assumed here.
Display HTML has a different output contract
The canonical oEmbed reference describes a JSON envelope whose useful display content is HTML recognized by X's widget JavaScript. Its timeline parameters include a display count from 1 to 20. That number controls an embedded presentation; it is not a page size for collecting account history. A JSON response format alone does not make an endpoint a structured Post-data API.
The same reference says the returned markup may change and recommends caching according to the response's suggested cache lifetime. Keep cache_age attached to that display response rather than treating a cached fragment as a permanent record of Post text or availability. The endpoint's documented lack of authentication and rate limiting applies to that embed resource, not to search, timelines, unrestricted reuse or every future request. No live oEmbed request was performed for this comparison.
Displaying X content also has conditions beyond the response schema. The current Developer Policy covers public display, privacy and redistribution; its display rules address keeping content current and responding when content becomes unavailable. Embedding does not settle downstream analytics or dataset-distribution rights. The permission decision record provides the general authorization questions; the endpoint matrix resolves a narrower technical selection problem.
Work three requests through the history boundary
The following are constructed requirements, not successful API calls or observations from an X account. They expose different failure modes: selecting the wrong dataset, asking a recent endpoint for older history, and confusing a visible ranked view with an available API order. Their useful result is a route decision and an explicit remaining condition, not a fabricated response containing the requested data.
A recent query needs a bounded interval
Suppose an approved integration needs matching public Posts from a defined interval wholly inside the recent 7-day window, with text, identifiers and creation timestamps. The candidate is recent search, subject to the query's supported operators, the app's access and the permitted purpose. The requirement names a start and end in UTC, the exact query and the requested fields. It does not say “all recent social activity,” which would mix search matches with other surfaces.
The fields guide explains that a default Post lookup returns minimal properties, including id, text and edit_history_tweet_ids. Additional properties such as created_at require the appropriate fields parameter. Requesting a field does not remove its authentication requirements, and a related object's properties may require an expansion as well as that object's fields. Define the schema against the selected endpoint rather than a screenshot of a Post card.
For a search request, preserve the query and interval while following its continuation value. The recent-search pagination guide documents reading next_token from the response and sending it with the subsequent request. Changing the query midway means the collected pages no longer describe the same search. Increasing the page size reduces neither the need for permission nor the endpoint's history restriction.
An application-side specification can make those distinctions copyable without pretending to be an API response:
result_kind = recent_search_matches
interval = explicit UTC start and end within the last 7 days
query = fixed throughout this pagination sequence
requested_fields = id,text,created_at
continuation = response meta.next_token
completion = no next_token within this endpoint and query
excluded_claim = complete lifetime account history
This is a planning notation, not executable request syntax. Its completion statement is deliberately scoped to the endpoint and query. Reaching a page with no continuation does not prove that every relevant Post ever published is available, that the query captured the intended concept, or that unavailable content can be recovered. A schema check and an endpoint choice answer different questions from a coverage claim.
Historical mentions are not an authored timeline
Now suppose a client asks for every mention of its account over the previous 12 months. The /2/users/:id/mentions endpoint's documented ceiling is the 800 most recent mentions. It cannot be relabeled a complete annual archive because a loop exhausted its pages. The authored timeline is also the wrong fallback: Posts written by the account and Posts mentioning it have different membership rules, even if some records overlap.
An archive-search product may be a candidate for an appropriately expressed historical query, but it requires checking supported operators, access, the time interval and the returned fields. The search overview's archive reach to March 2006 describes a product's historical range, not a certificate that the client's annual dataset is exhaustive. If the project cannot establish a supported query and permitted access, mark the annual requirement unresolved or reduce it to a documented recent scope.
The same reasoning applies to an authored-Post request older than the most recent 3,200 records. Adding older start and end parameters does not itself remove the timeline's stated bound. Record the limit next to the proposed endpoint so that a downstream analyst cannot mistake “returned all available pages” for “returned the account's entire lifetime.” A limit that remains visible in the export description is more useful than an unqualified completeness label.
Ranked home is not chronological home
A third requirement asks for the exact algorithmic order a particular account saw on its home screen. The documented reverse-chronological route explicitly excludes algorithmic ranking. It requires user authentication and describes the most recent 3,200 Posts or 7 days from followed accounts. That is a useful official dataset, but choosing it changes the original ranking requirement rather than satisfying it.
If chronological home is sufficient, obtain the correct user authorization and specify that order in the consumer's contract. If exact ranked presentation remains essential, these sources do not establish a stable official equivalent. A separately permitted browser observation could describe what was rendered at a particular time; it cannot establish a replayable ranked history, a complete recommendation set or permission to replay internal requests. The rendering and pagination comparison explains the general response-versus-render distinction without turning a DOM into an API entitlement.
Keep authentication and lifetime attached to the route
The authentication overview distinguishes app-only access to publicly available information from user-context flows. An app-only bearer credential is not interchangeable with a user's authorization. The home-timeline reference specifically requires OAuth user context. Likewise, account control or client consent does not mean the operator owns X, may request every field, or has approval for every downstream use.
| Credential context | Documented lifetime or purpose | Operational consequence |
|---|---|---|
| OAuth 2.0 Authorization Code with PKCE | Access token defaults to 2 hours; offline.access issues a refresh token | Implement the documented renewal flow when authorized; distinguish access token and refresh token |
| OAuth 1.0a user context | The user-token guide describes obtaining a permanent access token | Do not transfer that wording to OAuth 2.0, app-only credentials or browser cookies; authorization can still change |
| App-only bearer context | Public-information access where the endpoint supports it | Confirm endpoint support; the retrieved overview does not establish a universal token lifetime |
| Signed-in browser session | A browser-side session, not an X API credential class | No universal cookie lifetime is established by these references; do not use it as an API fallback credential |
The PKCE guide gives a default 2-hour access-token lifetime and explains the refresh-token role of offline access. A refresh token permits a supported renewal flow; it does not turn authorization into an irrevocable grant. The OAuth 1.0a guide uses permanent-token terminology for its own flow. Keeping those statements separate prevents a long-running collector from assuming that a token copied from any login method behaves identically.
Attach the credential class and required scopes to the endpoint before estimating continuity. A user approving one app operation does not make an app-only request user-authenticated. An expired access token does not by itself imply exhausted rate capacity. A supported refresh operation addresses token validity, while a quota reset addresses a request-accounting window. Those events belong in separate diagnostic fields, with secrets excluded from logs and review records.
Browser persistence is another layer. The multi-account session workflow covers authorized browser-session separation and storage. It does not document X API grants or the lifetime of X cookies. A saved browser state can cease to be accepted even though a local file still exists; retaining that file is not evidence that an API token remains valid or that the account's permissions are unchanged.
Separate request limits from browser observations
The rate-limit reference assigns limits by endpoint and authentication context. User-authenticated requests can count against a user's allocation, while app-only requests can count against an application's allocation. The response headers x-rate-limit-limit, x-rate-limit-remaining and x-rate-limit-reset identify the applicable limit, remaining capacity and reset timestamp. Keep the endpoint and credential context with those values; a header observed on search is not a budget for mentions.
A useful diagnostic therefore separates four fields: which route was requested, which authentication context was used, what response and headers actually arrived, and what billing or access condition the app currently has. The published request-rate table is not a balance statement. A remaining request allowance does not establish usage credit, and access to one product does not establish another product's horizon. Check the actual response and current developer account rather than freezing a large quota table into application assumptions.
The Developer Agreement prohibits exceeding or circumventing API limitations. Switching accounts, browser identities or request routes is not a remedy for an exhausted or unavailable entitlement. Generic waiting, bounded retries and concurrency planning are covered by rate limiting and concurrency; the X-specific task is identifying which accounting scope applies before those mechanics are considered.
No live X account, API request, paid credit balance, browser timeline or embed response was exercised here. The documented matrix can reject a mismatched design, but only an appropriately authorized minimal request can establish what a specific app currently receives. Even that result establishes one response under its recorded conditions, not permanent availability, full archive coverage or a stable browser layout. Keep permission, response semantics and collection completeness as separately stated conclusions.
FAQ
Does “Twitter scraping API” mean one endpoint?
No. Authored Posts, mentions, home timelines, search and embedded display have different membership and output contracts. An endpoint returning a Post-shaped record is not automatically a substitute for another dataset.
Can pagination cross from recent search into the full archive?
No. A next_token advances the selected query within its product boundary. Older requirements need a separately supported archive route and access decision, rather than more recent-search pages.
Does an empty or shorter response prove an account has no older Posts?
No. The request describes one route, interval, filter and authorization context. Exhaustion inside that scope does not prove global absence or justify bypassing the stated boundary.
Is a browser required whenever an API requirement fails?
No. An unsupported history, field or ranking requirement may need narrowing or another permitted data source. Browser selection follows an established role; it does not supply missing platform authorization.
Sources
- X API introduction
- Timelines and documented bounds
- Recent and full-archive search
- Fields and search pagination
- Authentication overview
- OAuth 2.0 PKCE and OAuth 1.0a user tokens
- oEmbed API
- Rate limits
- Developer Policy and Developer Agreement
- BotBrowser automation consistency
Practical conclusion
Choose the X result before the browser: authored Posts, mentions, chronological home, bounded search or embedded display. Write the required interval, ordering, fields and authorization context beside the documented endpoint limit. If that combination cannot satisfy the task, narrow or stop it; if it can, make a permitted minimal request and describe its actual scope before building a larger collector.
This article is part of the guide: Browser Automation Getting Started: A First Run Map
Related Articles
Scraping with Playwright: Browser and Profile Setup
Getting StartedScraping Legal and Terms: What a Team Must Decide First
Getting StartedWeb Scraping Browser: Which Kind Do You Need?
Getting StartedInstagram Scraping: API Eligibility Before Browsers
Getting StartedWeb Scraping JavaScript: Rendering, Pagination, Scroll
Getting StartedPuppeteer Proxy: Auth, SOCKS5 and Browser Signals
Run the same check in BotBrowser
Load a profile, keep the signals you just read about consistent inside the browser, and confirm the result on your own machine before a rollout.