# Twitter Scraper Pagination: Cursors, Limits and Deduplication

Twitter search results arrive in pages: the X API returns a next\_token for each next page, and apidojo's actors follow X's page cursors for you until maxItems is reached or the search runs out.

[![apidojo](/blog/authors/apidojo.png)**Apidojo** *15+ years of experience in data retrieval.*](/blog/authors/apidojo/)

**7 October 2026** Published

**4 min** Read

![Cover card reading 'Twitter Scraper Pagination: Cursors and Limits'](/blog/covers/pagination.png)

**Twitter search results arrive in pages: the X API returns a `next_token` for each next page, and apidojo’s actors follow X’s page cursors for you until `maxItems` is reached or the search runs out.** For large jobs, split the search into date windows, read datasets in pages with `offset` and `limit`, and deduplicate every result on its post `id`.

This guide covers pagination on both routes, why searches stop early, and how to merge pages without duplicates. The actor choice is explained in [how to scrape Twitter data in 2026](/blog/twitter-scraping/how-to-scrape-twitter/).

## How Does Pagination Work in the X API?

**The X API’s search endpoints return up to 100 posts (recent) or 500 posts (full archive) per request, plus a `next_token` in the `meta` object when more results exist.** Send that token as `next_token` on the next request, and stop when no token comes back ([X API docs](https://docs.x.com/x-api/posts/search)).

```
import os, requests

url = "https://api.x.com/2/tweets/search/recent"
headers = {"Authorization": f"Bearer {os.environ['BEARER_TOKEN']}"}
params = {"query": "from:NASA -is:retweet", "max_results": 100}
posts = []

while True:
    data = requests.get(url, headers=headers, params=params).json()
    posts.extend(data.get("data", []))
    token = data.get("meta", {}).get("next_token")
    if not token:
        break
    params["next_token"] = token
```

Every post returned costs $0.005, so set a page limit before running an open-ended loop ([X API pricing](https://docs.x.com/x-api/getting-started/pricing)).

## How Does Pagination Work in apidojo’s Actors?

**apidojo’s actors page through X search internally, so you set `maxItems` and the actor keeps fetching pages until it reaches that number or the search has no more results.** You never handle X’s cursors yourself ([Apify](https://apify.com/apidojo/tweet-scraper)).

```
{
  "searchTerms": ["#AI lang:en since:2026-09-01"],
  "sort": "Latest",
  "maxItems": 20000
}
```

On Twitter Scraper Unlimited, each query includes the first 2 pages, about 40 posts, in its $0.016 query fee, and later items are billed per item ([Apify](https://apify.com/apidojo/twitter-scraper-lite)).

## How Do You Read a Large Dataset in Pages?

**Read a large Apify dataset in pages with `offset` and `limit`, or let the client iterate for you.** `iterate_items()` in Python pages through the whole dataset without loading it into memory at once.

```
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
dataset = client.dataset("DATASET_ID")

page = dataset.list_items(offset=0, limit=1000)
print(page.total, "items in the dataset")

for item in dataset.iterate_items():
    pass  # process one post at a time
```

Unnamed datasets expire after 7 days, so download or name the ones you keep ([Apify docs](https://docs.apify.com/storage/dataset)).

## Why Does a Twitter Search Stop Before the End?

**A Twitter search stops early when X stops serving further pages for a broad query, when `maxItems` is reached, or when the date window holds fewer posts than expected.** Tweet Scraper V2’s documentation suggests switching `sort` to `Top` or removing `until:` when a query returns fewer results than expected (Apify). Listed below are the 3 fixes.

1.  **Split** the period into date windows, one query per window; see [how to scrape tweets by date range](/blog/twitter-scraping/scrape-tweets-by-date/).
2.  **Narrow** broad queries with OR groups and exclusions, so each one fits inside what X serves.
3.  **Raise** `maxItems`, or clear it.

## How Do You Deduplicate Tweets Across Pages and Runs?

**Deduplicate tweets by keeping one record per post `id`, because the same post can appear in touching date windows, in several keyword queries, or twice with `sort: Latest + Top`.** A dictionary or database primary key on `id` does this automatically.

```
tweets = {}
for item in dataset.iterate_items():
    tweets[item["id"]] = item  # later copies overwrite earlier ones
```

For recurring collection, store the IDs you have already processed and skip them on the next run, as in [how to monitor a keyword on X with an API](/blog/twitter-scraping/monitor-keyword/).

## How Do You Resume a Collection Where It Stopped?

**Resume a collection with the newest post time you already have: start the next query’s `since:` at that date, or use `since_id:` with the highest post ID.** `since_id:` returns posts newer than the given ID, so nothing you already hold comes back ([twitter-advanced-search reference](https://github.com/igorbrigadir/twitter-advanced-search)). The ID operators are explained in [since: and until: explained](/blog/twitter-search/since-until-operators/).

## The scrapers behind this

-   [Tweet Scraper V2 Tweets from searches, profiles, Lists and URLs; 49–64 tweets/sec; minimum 50 per query $0.40 / 1K tweets](/scrapers/twitter-scraper/)
-   [Twitter Scraper Unlimited Single tweets, reply threads, X Articles and small queries; no minimum $0.016 / query + $0.40 / 1K](/scrapers/twitter-scraper-lite/)

## Questions people ask

### Is there a maximum number of tweets per query?

Tweet Scraper V2 is documented as returning more than 100,000 tweets from a single query when the data exists (Apify).

### Do you pay for duplicate tweets?

Each item returned is billed, so avoid overlapping windows and queries where you can.

### Where is the Node.js version?

See [how to collect tweets with JavaScript and Node.js](/blog/twitter-scraping/scrape-tweets-javascript/).

## More on scraping X data

-   [How to Scrape Twitter (X) Data in 2026 What Twitter data you can scrape, the 3 ways to collect it, which apidojo actor fits each task, prices, limits and Python code.](/blog/twitter-scraping/how-to-scrape-twitter/)
-   [How to Scrape Tweets by Keyword (2026 Guide) Scrape tweets by keyword with apidojo's Tweet Scraper V2: query rules, batching, Python code, costs and fixes for low results.](/blog/twitter-scraping/scrape-tweets-by-keyword/)
-   [How to Scrape Tweets by Date Range Scrape tweets between two dates by splitting periods into windows, running them in batches and removing duplicates, with Python code.](/blog/twitter-scraping/scrape-tweets-by-date/)
-   [How to Scrape Tweets From a User Profile Scrape any public profile's tweets with from:username, collect a full history in date windows, and compare two apidojo actors.](/blog/twitter-scraping/scrape-user-tweets/)
-   [How to Scrape Twitter Search Results Turn any X search into a JSON, CSV or Excel dataset with apidojo's Tweet Scraper V2, using search queries or search result URLs.](/blog/twitter-scraping/scrape-search-results/)
-   [How to Scrape a Twitter Thread and All Replies Scrape every reply in a thread with conversation\_id:, keep only the author's thread, filter replies and see what each thread costs.](/blog/twitter-scraping/scrape-thread-replies/)
-   [How to Scrape Images and Videos From Tweets Collect tweets with media, find the media URLs in the output, download images with Python and pick the best video file.](/blog/twitter-scraping/scrape-twitter-media/)
-   [How to Export Tweets to CSV or JSON Export tweets to CSV, Excel, JSON or JSONL: your own archive, any search through a scraper, the columns you get and the API call.](/blog/twitter-scraping/export-tweets-csv-json/)
-   [How to Collect Tweets With Python (2026) A complete Python tweet collector: setup, first run, date parsing, date-window batching, deduplication and CSV export.](/blog/twitter-scraping/scrape-tweets-python/)
-   [How to Collect Tweets With JavaScript and Node.js A complete Node.js tweet collector: setup, first run, safe tweet IDs, date-window batching, deduplication and CSV export.](/blog/twitter-scraping/scrape-tweets-javascript/)
-   [How to rebuild an X conversation tree from replies Replies arrive as a flat list. Two fields, conversationId and inReplyToId, are what turn that list back into the threaded conversation you saw on X.](/blog/twitter-scraping/x-reply-tree/)
-   [When the X replies endpoint under-returns A thread that looks busy on X can come back thin from a scrape. Here is how to tell a quiet conversation from a short run, and what the second flow fixes.](/blog/twitter-scraping/x-replies-vs-search-flow/)
-   [Reading account age in X reply data Every reply carries the responder's account creation date, follower counts and post history. Here is what those fields can and cannot tell you.](/blog/twitter-scraping/x-replier-account-age/)

![apidojo](/blog/authors/apidojo.png)

## [Apidojo](/blog/authors/apidojo/)

15+ years of experience in data retrieval.

API Dojo is always ready for you. With 15+ years of experience, data acquisition has never been that easy. Cheapest prices with high availability.

-   🏯 15+ years of experience in data retrieval.
-   💰 Cheapest prices, qualified data.
-   ⚙️ High availability, extreme maintenance.
-   🐉 Hyper communicative, excellent support.

-   [apify.com](https://apify.com/apidojo)
-   [x.com](https://x.com/apidojo_)
-   [github.com](https://github.com/apidojo-io)