What types of social media datasets does SocialIntel offer?

SocialIntel offers datasets from 100+ social media platforms including Reddit, GitHub, YouTube, Medium, Discord, Telegram, and more. Each dataset is enriched with 250+ fields covering sentiment analysis, topic detection, financial signals, engagement metrics, and content classification.

How much does SocialIntel cost?

Catalog datasets up to 10K rows are always free to download. Individual datasets start at $18. Pro plans range from $78 to $504 per month, with annual pricing from $780 to $5,040 per year.

Can I request a custom dataset?

Yes! You can submit a custom dataset request specifying the platform, fields, date range, and format. Custom datasets are priced based on row count, starting at $18.

What formats are datasets available in?

Datasets are available in CSV, JSON, JSONL, and Parquet formats. You can download them directly or access them via our REST API.

Is the data ethically sourced and legal?

Yes. We collect only publicly accessible data from social media platforms using official APIs and permissive robots.txt directives. We comply with each platform's terms of service and applicable laws. See our Legality & Compliance page for a full platform-by-platform breakdown.

Building a Real-Time Entity Extraction Pipeline for Social Media

Extracting people, organizations, products, and events from social media streams requires more than off-the-shelf NER. Here is our approach to high-recall entity extraction at scale.

Entity extraction transforms unstructured social media text into structured, queryable data. But standard named entity recognition (NER) models trained on news wire data perform poorly on tweets, Reddit comments, and forum posts where entities are referenced informally, abbreviations are common, and context is minimal.

The Challenge

Social media entities differ from formal text in several ways:

**Informal references** — "Elon" instead of "Elon Musk", "ETH" instead of "Ethereum"
**Context-dependent** — "Apple" could mean the company, the fruit, or a record label
**Evolving** — New products, people, and events appear daily
**Platform-specific** — Reddit uses flair and hashtags, YouTube uses channel names, Mastodon uses @mentions

Our Multi-Layer Approach

We combine multiple techniques for high-recall extraction:

Layer 1: Pattern Matching

Regex patterns handle high-confidence cases:

**Ticker symbols** — $AAPL, $TSLA, $BTC
**Crypto addresses** — Bitcoin, Ethereum wallet patterns
**Dates and times** — Relative and absolute references
**Prices and amounts** — $100, 50%, 10K followers

Layer 2: Dictionary Lookup

We maintain curated dictionaries for:

**People** — Public figures, politicians, celebrities, tech leaders (50K+ entries)
**Organizations** — Companies, nonprofits, government agencies (100K+ entries)
**Products** — Tech products, brands, services (200K+ entries)
**Cryptocurrencies** — Tokens, protocols, DeFi platforms (5K+ entries)

Dictionaries are updated weekly from authoritative sources.

Layer 3: Context-Aware Disambiguation

When multiple entity candidates exist, we use surrounding context:

**Co-occurrence signals** — "Apple" near "iPhone" or "MacBook" resolves to Apple Inc.
**Platform context** — In r/wallstreetbets, "GME" is GameStop; in r/gaming, it could be something else
**Temporal context** — Entity references shift with news cycles and events

Layer 4: ML Classification

For ambiguous cases that pattern matching and dictionaries cannot resolve, we use a lightweight classifier trained on social media text. This handles novel entities and context-dependent disambiguation.

Performance Metrics

On our validation set of 100K manually annotated posts:

**Precision** — 94% across all entity types
**Recall** — 89% (lower for rare or newly emerging entities)
**Latency** — <5ms per post for full entity extraction
**Coverage** — 15+ entity types including custom financial and crypto categories

Integration with Downstream Analysis

Extracted entities enable powerful downstream capabilities:

**Topic tracking** — Monitor mentions of specific companies, products, or people over time
**Competitive intelligence** — Compare brand mention volume and sentiment across platforms
**Financial signals** — Detect entity-anchored market discussions and event-driven chatter
**Knowledge graphs** — Build entity relationship maps from conversational data

— Heshan Sanjuka, Founder