SocialINTEL
We use essential cookies to keep you signed in and improve your experience. Cookie Policy
250+ deterministic fields available. Zero external APIs. Fully reproducible.
| Field | Type | Range | Method | Description |
|---|---|---|---|---|
| sentiment_label | string | positive / negative / neutral | VADER Lexicon | ≥0.1 → "positive", ≤-0.1 → "negative", else "neutral". |
| sentiment_score | float | -1.0 to +1.0 | VADER Lexicon | Normalized compound score: raw_sum / sqrt(raw_sum² + 15). ALL-CAPS words get +0.25 boost. Negation words invert polarity of next 3 scored tokens (-0.74 factor). |
| sentiment_positive | float | 0.0 to 1.0 | VADER Lexicon | Sum of positive token valences divided by total absolute valence. |
| sentiment_negative | float | 0.0 to 1.0 | VADER Lexicon | Sum of negative token valences divided by total absolute valence. |
| sentiment_neutral | float | 0.0 to 1.0 | VADER Lexicon | 1.0 - pos_ratio - neg_ratio. |
| sentiment_confidence | float | 0.0 to 1.0 | VADER Lexicon | min(1.0, match_density × 0.6 + magnitude × 0.4). match_density = ratio of tokens found in lexicon. |
| sentiment_method | string | vader_lexicon / vader_lexicon_emoji | VADER Lexicon | Set to emoji variant when no lexicon tokens matched and emoji fallback was used. |
| sentiment_vader_reliable | bool | true / false | Heuristic Formula | False when word_count < 5, sentiment_confidence < 0.1, or language_detected is not explicitly "en" or "eng". When False, all sentiment fields are zeroed/neutralized. |
| language_gated | bool | true / false | Heuristic Formula | True when language_detected is not explicitly "en" or "eng" — triggers all sentiment fields zeroed and vader_reliable = False. |
| tone | string | positive / negative / neutral / mixed | Heuristic Formula | Derived from sentiment_label. Corrected during quality checks when sentiment_label contradicts emotion_primary. |
| sentiment_vader_original | float | -1.0 to +1.0 | VADER Lexicon | The raw VADER compound before sarcasm-driven polarity inversion, retained for transparency. |
| sentiment_aspect_keywords | list | 0+ strings | Sentence-Bounded VADER | Aspect seed keywords whose containing sentence(s) were scored for aspect sentiment. |
| aspect_sentiment_summary | string | positive / negative / neutral / mixed | Sentence-Bounded VADER | Mean aspect sentiment aggregated across all seed mentions and their sentences. |
| sentiment_emoji_score | float | -1.0 to +1.0 | VADER Lexicon | Sentiment computed only from emoji valence mapping, used when no lexicon tokens matched. |
| Field | Type | Range | Method | Description |
|---|---|---|---|---|
| emotion_primary | string | joy / anger / fear / sadness / surprise / disgust / anticipation / trust / neutral | Emotion Lexicon | Argmax selection after dampening polarity-contradictory emotions (×0.3) and mutual-exclusivity pruning of opposite pairs. Minimum threshold 0.15. |
| emotion_secondary | string | Same 8 + "none" | Emotion Lexicon | Second-highest scoring emotion after all pruning rules applied. |
| emotion_scores | list | Each 0.0 to 1.0 | Emotion Lexicon | Dict of per-emotion word-match scores normalized to 0-1 per the NRC-style lexicon. |
Valence-scored lexicon of ~7,500 English tokens. Compound = sum / sqrt(sum² + raw_sum² × 0.0001 + 15). Handles negation (×-0.74 within 3 tokens), booster words, ALL-CAPS (+0.25), emoji, and slang (lmao, af).
Word-boundary regex matching (\bkeyword\b) against curated keyword lists — so "data" never matches "database" and "source" never matches "resource". Stance/signal detectors (bull/bear, FUD/FOMO/pump/shill, misinfo, political lean) are negation-aware: a match inside a negated clause (e.g. "not bullish", "not a scam") is excluded. Used for topic_tags, hate_signal, financial_bullish/bearish, and 143 other fields.
Weighted scoring formulas combining multiple inputs. Examples: text_quality_score = base 0.60 + length bonuses + density penalties. quality_score = tq×0.40 + pipeline×0.20 + sc×0.15 + content×0.10 + (1-flags×0.1)×0.15.
Pattern-based regex extraction: sarcasm markers (30 patterns: /s tag, quoted compliments, "oh great"), salary ranges ($XXk-$YYk), review ratings (X/10, ★★★★☆), tickers ($BTC).
Raw metadata from source platform — no computation applied. Includes engagement counts (likes, comments, shares), timestamps, platform fields (board, thread_id), GitHub metrics, YouTube tags.
Script-detection + function-word frequency scoring across 7 languages. No ML models used.
NRC-style emotion lexicon of ~200 tokens mapped to 8 emotions (joy, anger, fear, sadness, surprise, disgust, anticipation, trust). Token matches are negation-aware — only non-negated occurrences count (e.g. "no joy", "without fear"). Scores normalized 0-1 with mutual exclusivity pruning of opposite pairs.
VADER computed over the full containing sentence of each aspect mention (brand, product, pricing, service, UX, security), aggregating all mentions per aspect rather than a fixed char window. Avoids truncating mid-sentence and bleeding into adjacent-sentence polarity. Used for aspect_* fields.
SHA-256(user_content + ":" + content)[:32] for cross-platform deduplication. MD5 of normalized content for internal tracking.
Vertical Intelligence dashboards, Trend Radar signals, Coordinated Behavior maps, and NL Insights answers are all computed from these same deterministic fields — and the Dataset Cleaner applies the same rigor to your own files.