How to Build a Twitter Dataset for LLM and RAG Projects

To build a Twitter dataset for an LLM or RAG project, collect posts and full threads on your topic with apidojo’s actors on Apify, save each post as one JSON Lines record with its text, author, date, engagement and URL, merge threads into single documents, then embed and index them. Keeping the URL with every record lets the model cite its sources.

This guide covers collection, document format, thread handling, indexing and refresh. The terms that apply to X content are covered at the end; check them before you start. This guide extends the polling pattern in how to monitor a keyword on X with an API.

What Twitter Data Works Best for RAG?

The best Twitter data for RAG is original posts and threads from accounts with real expertise on your topic, filtered for engagement and stripped of reposts. Listed below are the 3 sources to combine.

Source	Query	Why
Expert accounts	from:expert1 OR from:expert2 -filter:retweets	High-signal original posts
Topic discussion	"vector database" min_faves:50 -filter:retweets lang:en	What practitioners say, with an engagement floor
Full threads	conversation_id:ID	Long explanations split across posts

The operators are explained in the Twitter advanced search operators reference.

How Do You Collect the Posts?

Collect topic and account posts with Tweet Scraper V2, and complete threads with Twitter Scraper Unlimited’s conversationIds input. Tweet Scraper V2 doesn’t fetch conversations, so threads go to Twitter Scraper Unlimited (Apify).

import json, os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("apidojo/tweet-scraper").call(run_input={
    "searchTerms": ['"vector database" min_faves:50 -filter:retweets lang:en since:2026-01-01'],
    "sort": "Top",
    "maxItems": 2000,
})

with open("tweets.jsonl", "w", encoding="utf-8") as f:
    for t in client.dataset(run.default_dataset_id).iterate_items():
        f.write(json.dumps({
            "id": t["id"], "url": t["url"], "author": t["author"]["userName"],
            "created": t["createdAt"], "likes": t["likeCount"], "text": t["text"],
        }, ensure_ascii=False) + "\n")

Thread collection is shown in how to scrape a Twitter thread and all its replies.

How Should You Format Tweets for Retrieval?

Store one post per record in JSON Lines, with the text as the content and everything else as metadata for filtering and citation. The table shows the fields to keep.

Field	Use in RAG
text	The content you embed
url	The citation shown to users
author	Filter by trusted accounts
created	Filter or boost by recency
likes	Rank or filter by engagement
id	Deduplicate and update records

Most posts are short enough to embed whole, so a single post rarely needs splitting.

How Do You Handle Threads?

Merge each author’s thread into one document in time order, because a thread’s posts only make sense together. Keep the first post’s URL as the citation and list the post IDs in the metadata. Sort the thread’s posts by createdAt, join the texts with line breaks, and embed the result as a single record.

How Do You Embed and Index the Dataset?

Embed each record’s text with the embedding model your RAG stack uses, and store the vectors with the metadata in a vector database or search index. Then retrieve with metadata filters, for example only posts from the last 6 months, or only from a list of trusted authors, so stale or low-quality posts don’t reach the model.

How Do You Keep the Dataset Fresh?

Keep the dataset fresh with a scheduled collection that adds new posts and skips IDs you already hold. The pattern is in how to monitor a keyword on X with an API, and resuming from the newest post with since_id: is in Twitter scraper pagination.

What Does a RAG Dataset Cost?

A RAG dataset costs $0.40 per 1,000 posts to collect with Tweet Scraper V2, so a 20,000-post knowledge base costs $8. Threads on Twitter Scraper Unlimited cost $0.016 per conversation, including about 40 posts, then from $0.0004 per post (Apify).

What Terms Apply to Using Tweets With AI?

X’s Terms of Service and Developer Agreement restrict some uses of X content, including some AI training uses, so check them before training or fine-tuning a model on tweets. Retrieval that cites and links the original posts is a different use from training, but data protection law such as the GDPR applies either way, because posts contain personal data.