# How to Build a Twitter Dataset for LLM and RAG Projects

To build a Twitter dataset for an LLM or RAG project, collect posts and full threads on your topic with apidojo's actors on Apify, save each post as one JSON Lines record with its text, author, date, engagement and URL, merge threads into single documents, then embed and index them.

[![apidojo](/blog/authors/apidojo.png)**Apidojo** *15+ years of experience in data retrieval.*](/blog/authors/apidojo/)

**7 October 2026** Published

**4 min** Read

![Cover card reading 'How to Build a Twitter Dataset for LLM and RAG'](/blog/covers/llm-rag-dataset.png)

**To build a Twitter dataset for an LLM or RAG project, collect posts and full threads on your topic with apidojo’s actors on Apify, save each post as one JSON Lines record with its text, author, date, engagement and URL, merge threads into single documents, then embed and index them.** Keeping the URL with every record lets the model cite its sources.

This guide covers collection, document format, thread handling, indexing and refresh. The terms that apply to X content are covered at the end; check them before you start. This guide extends the polling pattern in [how to monitor a keyword on X with an API](/blog/twitter-scraping/monitor-keyword/).

## What Twitter Data Works Best for RAG?

**The best Twitter data for RAG is original posts and threads from accounts with real expertise on your topic, filtered for engagement and stripped of reposts.** Listed below are the 3 sources to combine.

| Source | Query | Why |
| --- | --- | --- |
| Expert accounts | from:expert1 OR from:expert2 -filter:retweets | High-signal original posts |
| Topic discussion | "vector database" min_faves:50 -filter:retweets lang:en | What practitioners say, with an engagement floor |
| Full threads | conversation_id:ID | Long explanations split across posts |

The operators are explained in the [Twitter advanced search operators](/blog/twitter-search/advanced-search-operators/) reference.

## How Do You Collect the Posts?

**Collect topic and account posts with Tweet Scraper V2, and complete threads with Twitter Scraper Unlimited’s `conversationIds` input.** Tweet Scraper V2 doesn’t fetch conversations, so threads go to Twitter Scraper Unlimited ([Apify](https://apify.com/apidojo/tweet-scraper)).

```
import json, os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("apidojo/tweet-scraper").call(run_input={
    "searchTerms": ['"vector database" min_faves:50 -filter:retweets lang:en since:2026-01-01'],
    "sort": "Top",
    "maxItems": 2000,
})

with open("tweets.jsonl", "w", encoding="utf-8") as f:
    for t in client.dataset(run.default_dataset_id).iterate_items():
        f.write(json.dumps({
            "id": t["id"], "url": t["url"], "author": t["author"]["userName"],
            "created": t["createdAt"], "likes": t["likeCount"], "text": t["text"],
        }, ensure_ascii=False) + "\n")
```

Thread collection is shown in [how to scrape a Twitter thread and all its replies](/blog/twitter-scraping/scrape-thread-replies/).

## How Should You Format Tweets for Retrieval?

**Store one post per record in JSON Lines, with the text as the content and everything else as metadata for filtering and citation.** The table shows the fields to keep.

| Field | Use in RAG |
| --- | --- |
| text | The content you embed |
| url | The citation shown to users |
| author | Filter by trusted accounts |
| created | Filter or boost by recency |
| likes | Rank or filter by engagement |
| id | Deduplicate and update records |

Most posts are short enough to embed whole, so a single post rarely needs splitting.

## How Do You Handle Threads?

**Merge each author’s thread into one document in time order, because a thread’s posts only make sense together.** Keep the first post’s URL as the citation and list the post IDs in the metadata. Sort the thread’s posts by `createdAt`, join the texts with line breaks, and embed the result as a single record.

## How Do You Embed and Index the Dataset?

**Embed each record’s text with the embedding model your RAG stack uses, and store the vectors with the metadata in a vector database or search index.** Then retrieve with metadata filters, for example only posts from the last 6 months, or only from a list of trusted authors, so stale or low-quality posts don’t reach the model.

## How Do You Keep the Dataset Fresh?

**Keep the dataset fresh with a scheduled collection that adds new posts and skips IDs you already hold.** The pattern is in [how to monitor a keyword on X with an API](/blog/twitter-scraping/monitor-keyword/), and resuming from the newest post with `since_id:` is in [Twitter scraper pagination](/blog/twitter-scraping/pagination/).

## What Does a RAG Dataset Cost?

**A RAG dataset costs $0.40 per 1,000 posts to collect with Tweet Scraper V2, so a 20,000-post knowledge base costs $8.** Threads on Twitter Scraper Unlimited cost $0.016 per conversation, including about 40 posts, then from $0.0004 per post ([Apify](https://apify.com/apidojo/twitter-scraper-lite)).

## What Terms Apply to Using Tweets With AI?

**X’s Terms of Service and Developer Agreement restrict some uses of X content, including some AI training uses, so check them before training or fine-tuning a model on tweets.** Retrieval that cites and links the original posts is a different use from training, but data protection law such as the GDPR applies either way, because posts contain personal data.

## The scrapers behind this

-   [Tweet Scraper V2 Tweets from searches, profiles, Lists and URLs; 49–64 tweets/sec; minimum 50 per query $0.40 / 1K tweets](/scrapers/twitter-scraper/)
-   [Twitter Scraper Unlimited Single tweets, reply threads, X Articles and small queries; no minimum $0.016 / query + $0.40 / 1K](/scrapers/twitter-scraper-lite/)

## Questions people ask

### Should you include replies?

Include replies only from trusted accounts or with an engagement floor; most replies add noise.

### How do you stop outdated posts from misleading the model?

Store the date with every record and filter or down-rank older posts at retrieval time.

### Where do you start?

At [how to scrape Twitter data in 2026](/blog/twitter-scraping/how-to-scrape-twitter/) or the [apidojo Twitter Scraper](/scrapers/twitter-scraper/).

## More X data use cases

-   [How to Monitor a Keyword on X With an API Build a keyword monitor for X: the query, hourly schedule, deduplication, Slack alerts and a cost table for 3 schedules.](/blog/twitter-scraping/monitor-keyword/)
-   [How to Build a Twitter Mention Monitor Build a mention monitor for X that catches tagged and untagged mentions, sorts them into complaints, questions and praise, and alerts your team.](/blog/twitter-scraping/mention-monitor/)
-   [How to Build a Competitor Tweet Tracker Track competitors' posts, engagement, formats and follower growth on X with a daily collector and a weekly pandas report.](/blog/twitter-scraping/competitor-tweet-tracker/)
-   [How to Collect Tweets for Sentiment Analysis Collect, clean and score tweets for sentiment analysis, from the search query to VADER scoring and weekly sentiment reports.](/blog/twitter-scraping/sentiment-analysis-dataset/)

![apidojo](/blog/authors/apidojo.png)

## [Apidojo](/blog/authors/apidojo/)

15+ years of experience in data retrieval.

API Dojo is always ready for you. With 15+ years of experience, data acquisition has never been that easy. Cheapest prices with high availability.

-   🏯 15+ years of experience in data retrieval.
-   💰 Cheapest prices, qualified data.
-   ⚙️ High availability, extreme maintenance.
-   🐉 Hyper communicative, excellent support.

-   [apify.com](https://apify.com/apidojo)
-   [x.com](https://x.com/apidojo_)
-   [github.com](https://github.com/apidojo-io)