How to Collect Tweets for Sentiment Analysis

To collect tweets for sentiment analysis, run a focused X search for your topic in one language with reposts excluded, collect it in date windows, clean the text, remove duplicates, and then score each post with a sentiment model. apidojo’s Tweet Scraper V2 on Apify collects the posts at $0.40 per 1,000 tweets, with the date and engagement fields you need to chart sentiment over time.

This guide covers the 4 stages: collection, cleaning, scoring and reporting. The collection method itself is in how to scrape tweets by keyword. This guide extends the polling pattern in how to monitor a keyword on X with an API.

How Do You Collect a Sentiment Dataset From X?

Collect a sentiment dataset with a query that matches the topic precisely, in one language, with reposts removed so each opinion counts once. Listed below are the 4 rules for the query (twitter-advanced-search reference).

1. Name the topic precisely, with OR groups for variants: ("model y" OR "model 3").
2. Fix the language, because sentiment models are language-specific: lang:en, or the actor’s tweetLanguage input.
3. Remove reposts: -filter:retweets.
4. Remove promotion: -filter:links or -giveaway -#ad, if ads would skew the results.

{
  "searchTerms": [
    "(\"model y\" OR \"model 3\") -filter:retweets since:2026-07-01 until:2026-08-01",
    "(\"model y\" OR \"model 3\") -filter:retweets since:2026-08-01 until:2026-09-01"
  ],
  "tweetLanguage": "en",
  "sort": "Latest"
}

Monthly windows keep each period comparable; window sizes are discussed in how to scrape tweets by date range.

How Do You Clean Tweets Before Scoring?

Clean tweets by removing URLs and handles, deduplicating on post ID, and dropping near-identical texts that bots or campaigns post repeatedly. Keep the original text in a separate column for review.

import re

def clean(text: str) -> str:
    text = re.sub(r"https?://\S+", "", text)
    text = re.sub(r"@\w+", "@user", text)
    return re.sub(r"\s+", " ", text).strip()

rows, seen_text = [], set()
for t in tweets.values():
    c = clean(t["text"])
    if c.lower() in seen_text or len(c) < 10:
        continue
    seen_text.add(c.lower())
    rows.append({"id": t["id"], "created": t["createdAt"], "text": c, "likes": t["likeCount"]})

Collecting tweets as a dictionary keyed by id is shown in how to collect tweets with Python.

How Do You Score Tweet Sentiment?

Score tweet sentiment with a model built for social media text: VADER is a fast rule-based option, and transformer models fine-tuned on tweets are more accurate for sarcasm and context. VADER returns a compound score from -1 to 1; scores of 0.05 and above are usually treated as positive, and -0.05 and below as negative.

from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

analyzer = SentimentIntensityAnalyzer()
for r in rows:
    score = analyzer.polarity_scores(r["text"])["compound"]
    r["sentiment"] = "positive" if score >= 0.05 else "negative" if score <= -0.05 else "neutral"

Check a random sample of 100 scored posts by hand before trusting the totals; sarcasm and product slang cause most errors.

How Do You Report Sentiment Over Time?

Group the scored posts by week, compute the share of positive and negative posts, and weight by engagement if reach matters more than volume. A post with 5,000 likes says more about public mood than one with none, so report both the raw share and the like-weighted share.

What Does a Sentiment Dataset Cost?

A sentiment dataset costs $0.40 per 1,000 tweets to collect with Tweet Scraper V2, so 50,000 posts across 6 months cost $20. Each query must return at least 50 tweets; very narrow topics belong on Twitter Scraper Unlimited (Apify).

What Should You Know About Using Tweet Data?

Tweets contain personal data, so collect only what the analysis needs, store it securely, and check X’s Terms of Service and data protection law such as the GDPR before you start. Replace handles with a placeholder, as the cleaning step does, when you publish results.