Twitter Scraper Pagination: Cursors, Limits and Deduplication

Twitter search results arrive in pages: the X API returns a next_token for each next page, and apidojo’s actors follow X’s page cursors for you until maxItems is reached or the search runs out. For large jobs, split the search into date windows, read datasets in pages with offset and limit, and deduplicate every result on its post id.

This guide covers pagination on both routes, why searches stop early, and how to merge pages without duplicates. The actor choice is explained in how to scrape Twitter data in 2026.

How Does Pagination Work in the X API?

The X API’s search endpoints return up to 100 posts (recent) or 500 posts (full archive) per request, plus a next_token in the meta object when more results exist. Send that token as next_token on the next request, and stop when no token comes back (X API docs).

import os, requests

url = "https://api.x.com/2/tweets/search/recent"
headers = {"Authorization": f"Bearer {os.environ['BEARER_TOKEN']}"}
params = {"query": "from:NASA -is:retweet", "max_results": 100}
posts = []

while True:
    data = requests.get(url, headers=headers, params=params).json()
    posts.extend(data.get("data", []))
    token = data.get("meta", {}).get("next_token")
    if not token:
        break
    params["next_token"] = token

Every post returned costs $0.005, so set a page limit before running an open-ended loop (X API pricing).

How Does Pagination Work in apidojo’s Actors?

apidojo’s actors page through X search internally, so you set maxItems and the actor keeps fetching pages until it reaches that number or the search has no more results. You never handle X’s cursors yourself (Apify).

{
  "searchTerms": ["#AI lang:en since:2026-09-01"],
  "sort": "Latest",
  "maxItems": 20000
}

On Twitter Scraper Unlimited, each query includes the first 2 pages, about 40 posts, in its $0.016 query fee, and later items are billed per item (Apify).

How Do You Read a Large Dataset in Pages?

Read a large Apify dataset in pages with offset and limit, or let the client iterate for you. iterate_items() in Python pages through the whole dataset without loading it into memory at once.

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
dataset = client.dataset("DATASET_ID")

page = dataset.list_items(offset=0, limit=1000)
print(page.total, "items in the dataset")

for item in dataset.iterate_items():
    pass  # process one post at a time

Unnamed datasets expire after 7 days, so download or name the ones you keep (Apify docs).

Why Does a Twitter Search Stop Before the End?

A Twitter search stops early when X stops serving further pages for a broad query, when maxItems is reached, or when the date window holds fewer posts than expected. Tweet Scraper V2’s documentation suggests switching sort to Top or removing until: when a query returns fewer results than expected (Apify). Listed below are the 3 fixes.

1. Split the period into date windows, one query per window; see how to scrape tweets by date range.
2. Narrow broad queries with OR groups and exclusions, so each one fits inside what X serves.
3. Raise maxItems, or clear it.

How Do You Deduplicate Tweets Across Pages and Runs?

Deduplicate tweets by keeping one record per post id, because the same post can appear in touching date windows, in several keyword queries, or twice with sort: Latest + Top. A dictionary or database primary key on id does this automatically.

tweets = {}
for item in dataset.iterate_items():
    tweets[item["id"]] = item  # later copies overwrite earlier ones

For recurring collection, store the IDs you have already processed and skip them on the next run, as in how to monitor a keyword on X with an API.

How Do You Resume a Collection Where It Stopped?

Resume a collection with the newest post time you already have: start the next query’s since: at that date, or use since_id: with the highest post ID. since_id: returns posts newer than the given ID, so nothing you already hold comes back (twitter-advanced-search reference). The ID operators are explained in since: and until: explained.