# How to Collect Tweets for Sentiment Analysis

To collect tweets for sentiment analysis, run a focused X search for your topic in one language with reposts excluded, collect it in date windows, clean the text, remove duplicates, and then score each post with a sentiment model.

[![apidojo](/blog/authors/apidojo.png)**Apidojo** *15+ years of experience in data retrieval.*](/blog/authors/apidojo/)

**7 October 2026** Published

**3 min** Read

![Cover card reading 'How to Collect Tweets for Sentiment Analysis'](/blog/covers/sentiment-analysis-dataset.png)

**To collect tweets for sentiment analysis, run a focused X search for your topic in one language with reposts excluded, collect it in date windows, clean the text, remove duplicates, and then score each post with a sentiment model.** apidojo’s Tweet Scraper V2 on Apify collects the posts at $0.40 per 1,000 tweets, with the date and engagement fields you need to chart sentiment over time.

This guide covers the 4 stages: collection, cleaning, scoring and reporting. The collection method itself is in [how to scrape tweets by keyword](/blog/twitter-scraping/scrape-tweets-by-keyword/). This guide extends the polling pattern in [how to monitor a keyword on X with an API](/blog/twitter-scraping/monitor-keyword/).

## How Do You Collect a Sentiment Dataset From X?

**Collect a sentiment dataset with a query that matches the topic precisely, in one language, with reposts removed so each opinion counts once.** Listed below are the 4 rules for the query ([twitter-advanced-search reference](https://github.com/igorbrigadir/twitter-advanced-search)).

1.  **Name the topic precisely**, with OR groups for variants: `("model y" OR "model 3")`.
2.  **Fix the language**, because sentiment models are language-specific: `lang:en`, or the actor’s `tweetLanguage` input.
3.  **Remove reposts**: `-filter:retweets`.
4.  **Remove promotion**: `-filter:links` or `-giveaway -#ad`, if ads would skew the results.

```
{
  "searchTerms": [
    "(\"model y\" OR \"model 3\") -filter:retweets since:2026-07-01 until:2026-08-01",
    "(\"model y\" OR \"model 3\") -filter:retweets since:2026-08-01 until:2026-09-01"
  ],
  "tweetLanguage": "en",
  "sort": "Latest"
}
```

Monthly windows keep each period comparable; window sizes are discussed in [how to scrape tweets by date range](/blog/twitter-scraping/scrape-tweets-by-date/).

## How Do You Clean Tweets Before Scoring?

**Clean tweets by removing URLs and handles, deduplicating on post ID, and dropping near-identical texts that bots or campaigns post repeatedly.** Keep the original text in a separate column for review.

```
import re

def clean(text: str) -> str:
    text = re.sub(r"https?://\S+", "", text)
    text = re.sub(r"@\w+", "@user", text)
    return re.sub(r"\s+", " ", text).strip()

rows, seen_text = [], set()
for t in tweets.values():
    c = clean(t["text"])
    if c.lower() in seen_text or len(c) < 10:
        continue
    seen_text.add(c.lower())
    rows.append({"id": t["id"], "created": t["createdAt"], "text": c, "likes": t["likeCount"]})
```

Collecting `tweets` as a dictionary keyed by `id` is shown in [how to collect tweets with Python](/blog/twitter-scraping/scrape-tweets-python/).

## How Do You Score Tweet Sentiment?

**Score tweet sentiment with a model built for social media text: VADER is a fast rule-based option, and transformer models fine-tuned on tweets are more accurate for sarcasm and context.** VADER returns a compound score from -1 to 1; scores of 0.05 and above are usually treated as positive, and -0.05 and below as negative.

```
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

analyzer = SentimentIntensityAnalyzer()
for r in rows:
    score = analyzer.polarity_scores(r["text"])["compound"]
    r["sentiment"] = "positive" if score >= 0.05 else "negative" if score <= -0.05 else "neutral"
```

Check a random sample of 100 scored posts by hand before trusting the totals; sarcasm and product slang cause most errors.

## How Do You Report Sentiment Over Time?

**Group the scored posts by week, compute the share of positive and negative posts, and weight by engagement if reach matters more than volume.** A post with 5,000 likes says more about public mood than one with none, so report both the raw share and the like-weighted share.

## What Does a Sentiment Dataset Cost?

**A sentiment dataset costs $0.40 per 1,000 tweets to collect with Tweet Scraper V2, so 50,000 posts across 6 months cost $20.** Each query must return at least 50 tweets; very narrow topics belong on [Twitter Scraper Unlimited](https://apify.com/apidojo/twitter-scraper-lite) ([Apify](https://apify.com/apidojo/tweet-scraper)).

## What Should You Know About Using Tweet Data?

**Tweets contain personal data, so collect only what the analysis needs, store it securely, and check X’s Terms of Service and data protection law such as the GDPR before you start.** Replace handles with a placeholder, as the cleaning step does, when you publish results.

## The scrapers behind this

-   [Twitter Scraper Unlimited Single tweets, reply threads, X Articles and small queries; no minimum $0.016 / query + $0.40 / 1K](/scrapers/twitter-scraper-lite/)
-   [Tweet Scraper V2 Tweets from searches, profiles, Lists and URLs; 49–64 tweets/sec; minimum 50 per query $0.40 / 1K tweets](/scrapers/twitter-scraper/)

## Questions people ask

### How many tweets do you need for sentiment analysis?

Enough per period to compare: a few hundred posts per week gives stable shares for most brands.

### Can you analyse sentiment in other languages?

Yes, collect with `lang:` and use a model trained on that language; see [how to search tweets by language](/blog/twitter-search/search-by-language/).

### Can you track sentiment continuously?

Yes, add scoring to a scheduled monitor; see [how to build a Twitter mention monitor](/blog/twitter-scraping/mention-monitor/).

## More X data use cases

-   [How to Monitor a Keyword on X With an API Build a keyword monitor for X: the query, hourly schedule, deduplication, Slack alerts and a cost table for 3 schedules.](/blog/twitter-scraping/monitor-keyword/)
-   [How to Build a Twitter Mention Monitor Build a mention monitor for X that catches tagged and untagged mentions, sorts them into complaints, questions and praise, and alerts your team.](/blog/twitter-scraping/mention-monitor/)
-   [How to Build a Competitor Tweet Tracker Track competitors' posts, engagement, formats and follower growth on X with a daily collector and a weekly pandas report.](/blog/twitter-scraping/competitor-tweet-tracker/)
-   [How to Build a Twitter Dataset for LLM and RAG Collect posts and threads, format them as JSON Lines with citation metadata, index them for retrieval and keep the dataset fresh.](/blog/twitter-scraping/llm-rag-dataset/)

![apidojo](/blog/authors/apidojo.png)

## [Apidojo](/blog/authors/apidojo/)

15+ years of experience in data retrieval.

API Dojo is always ready for you. With 15+ years of experience, data acquisition has never been that easy. Cheapest prices with high availability.

-   🏯 15+ years of experience in data retrieval.
-   💰 Cheapest prices, qualified data.
-   ⚙️ High availability, extreme maintenance.
-   🐉 Hyper communicative, excellent support.

-   [apify.com](https://apify.com/apidojo)
-   [x.com](https://x.com/apidojo_)
-   [github.com](https://github.com/apidojo-io)