Social Intel Enrichment Methodology — 250+ Deterministic Fields

Complete technical documentation of the deterministic enrichment pipeline for social media data. All 250+ fields in the library are computed via heuristic, lexicon, and regex-based methods — no external ML APIs.

Core Identifiers
Base columns present on every row across all platforms and all export formats. No computation applied — passed through from the collector.

- id (string, unique per post): Platform-specific post ID or synthetic fallback ID when the source has none. [Method: Direct Passthrough]
- platform (string, source platform name): The platform the post was collected from (reddit, youtube, github, mastodon, ao3, etc.). [Method: Direct Passthrough]
- author (string, username or "unknown"): Display name or handle of the author. Sanitized to "unknown" when the value is an email, URL, hash, or a known platform placeholder. [Method: Direct Passthrough]
- url (string, valid URL): Canonical URL of the post for deduplication and sourcing. [Method: Direct Passthrough]
- published_at (string, ISO 8601): Post publication timestamp normalized to ISO 8601. [Method: Direct Passthrough]
- created_at (string, ISO 8601): Timestamp the record was first inserted into the database. [Method: Direct Passthrough]
- first_seen (string, ISO 8601): Timestamp when the post was first observed by the collector. [Method: Direct Passthrough]
- last_updated (string, ISO 8601): Timestamp of the most recent update to the record (re-sync or re-enrichment). [Method: Direct Passthrough]

Content Fields
The raw post content and its source-language metadata.

- title (string, 0-200 chars): Post/thread title, truncated to 200 characters. [Method: Direct Passthrough]
- content (string, 0+ chars): Full post body or comment text after HTML unescaping. [Method: Direct Passthrough]
- original_content (string, 0+ chars): Content in its original language before translation. [Method: Direct Passthrough]
- translated_from (string, language code or ""): Source language code when content was machine-translated to English. [Method: Direct Passthrough]
- media_type (string, text / image / video / audio / link / crosspost): Detected media form of the post from collector metadata or content inference. [Method: Direct Passthrough]
- language_detected (string, ISO 639-1 code or "unknown"): Detected language via script + function-word frequency scoring across 40 languages. Gates whether VADER sentiment and other English lexicons run. [Method: Heuristic]

Engagement Metrics
Raw interaction counts passed through from the source platform. These are the inputs engagement_rate and reach_estimate are derived from.

- likes (int, 0+): Like/upvote/reaction count from the platform. [Method: Direct Passthrough]
- comments (int, 0+): Number of top-level comments/replies reported by the platform. [Method: Direct Passthrough]
- shares (int, 0+): Share/reblog/retweet/forward count. [Method: Direct Passthrough]
- views (int, 0+): View or play count (YouTube, Twitch, TikTok, etc.). [Method: Direct Passthrough]
- saves (int, 0+): Bookmark/save count where supported. [Method: Direct Passthrough]
- reactions (int, 0+): Sum of all reaction types where the platform exposes them (Reddit awards, Facebook reacticons). [Method: Direct Passthrough]
- engagement_rate (float, 0.0 to 1.0): (likes + comments + shares + views + saves + reactions) / max(1, author_followers). Indicates how viral a post is relative to the author audience. [Method: Heuristic Formula]
- score (int, 0+): Platform score where defined (Reddit score = upvotes - downvotes). [Method: Direct Passthrough]
- upvote_ratio (float, 0.0 to 1.0): Ratio of upvotes to total votes (Reddit). [Method: Direct Passthrough]

Author Metadata
Metadata about the post author.

- author_followers (int, 0+): Follower count of the author at collection time. Used as the denominator for engagement_rate. [Method: Direct Passthrough]

Content Features
Structured features extracted from the post text.

- hashtags (list, 0+ strings): Hashtags extracted from title and content (also used by NSFW tag checks). [Method: Keyword/Regex]
- mentions (list, 0+ strings): @username mentions extracted from content and title. [Method: Keyword/Regex]
- tickers (list, 0+ strings): Stock tickers detected via $TICKER pattern. [Method: Regex Pattern]
- ticker_count (int, 0+): Number of distinct tickers detected in the post. [Method: Keyword/Regex]
- ticker_mentions (list, 0+ strings): Tickers found via keyword matching against a known ticker list (additionally to $TICKER detection). [Method: Keyword/Regex]

Platform-Specific Fields
Source-specific columns that only populate for their originating platform.

- subreddit (string, r/ name): Subreddit the Reddit post was collected from. [Method: Direct Passthrough]
- domain (string, domain or ""): Linked domain for link posts. [Method: Direct Passthrough]
- duration_seconds (int, 0+): Video/audio duration in seconds. [Method: Direct Passthrough]
- category (string, platform-specific): Platform category field (e.g., Steam app type). [Method: Direct Passthrough]
- is_self (bool, true / false): True when the post body is text rather than a link. [Method: Direct Passthrough]
- board (string, 0+ strings): Board name (4chan, etc.). [Method: Direct Passthrough]
- thread_id (string, —): Parent thread ID for replies/comment posts. [Method: Direct Passthrough]
- channel_username (string, —): Channel or group username (Telegram). [Method: Direct Passthrough]
- over_18 (bool, true / false): Platform-declared NSFW flag. [Method: Direct Passthrough]
- stickied (bool, true / false): True if the post is pinned on the source. [Method: Direct Passthrough]
- locked (bool, true / false): True if comments are locked on the source. [Method: Direct Passthrough]
- is_fork (bool, true / false): True for GitHub repositories that are forks. [Method: Direct Passthrough]
- archived (bool, true / false): True if the post/thread is archived on the source. [Method: Direct Passthrough]
- language_github (string, programming language or ""): Primary GitHub repository language. [Method: Direct Passthrough]
- license (string, SPDX or ""): Repository license identifier (GitHub). [Method: Direct Passthrough]

Sentiment Analysis
Two-tier sentiment system combining VADER lexicon and sarcasm detection with sentiment polarity inversion.

- sentiment_label (string, positive / negative / neutral): ≥0.1 → "positive", ≤-0.1 → "negative", else "neutral". [Method: VADER Lexicon]
- sentiment_score (float, -1.0 to +1.0): Normalized compound score: raw_sum / sqrt(raw_sum² + 15). ALL-CAPS words get +0.25 boost. Negation words invert polarity of next 3 scored tokens (-0.74 factor). [Method: VADER Lexicon]
- sentiment_positive (float, 0.0 to 1.0): Sum of positive token valences divided by total absolute valence. [Method: VADER Lexicon]
- sentiment_negative (float, 0.0 to 1.0): Sum of negative token valences divided by total absolute valence. [Method: VADER Lexicon]
- sentiment_neutral (float, 0.0 to 1.0): 1.0 - pos_ratio - neg_ratio. [Method: VADER Lexicon]
- sentiment_confidence (float, 0.0 to 1.0): min(1.0, match_density × 0.6 + magnitude × 0.4). match_density = ratio of tokens found in lexicon. [Method: VADER Lexicon]
- sentiment_method (string, vader_lexicon / vader_lexicon_emoji): Set to emoji variant when no lexicon tokens matched and emoji fallback was used. [Method: VADER Lexicon]
- sentiment_vader_reliable (bool, true / false): False when word_count < 5, sentiment_confidence < 0.1, or language_detected is not explicitly "en" or "eng". When False, all sentiment fields are zeroed/neutralized. [Method: Heuristic Formula]
- language_gated (bool, true / false): True when language_detected is not explicitly "en" or "eng" — triggers all sentiment fields zeroed and vader_reliable = False. [Method: Heuristic Formula]
- tone (string, positive / negative / neutral / mixed): Derived from sentiment_label. Corrected during quality checks when sentiment_label contradicts emotion_primary. [Method: Heuristic Formula]
- sentiment_vader_original (float, -1.0 to +1.0): The raw VADER compound before sarcasm-driven polarity inversion, retained for transparency. [Method: VADER Lexicon]
- sentiment_aspect_keywords (list, 0+ strings): Aspect seed keywords whose containing sentence(s) were scored for aspect sentiment. [Method: Sentence-Bounded VADER]
- aspect_sentiment_summary (string, positive / negative / neutral / mixed): Mean aspect sentiment aggregated across all seed mentions and their sentences. [Method: Sentence-Bounded VADER]
- sentiment_emoji_score (float, -1.0 to +1.0): Sentiment computed only from emoji valence mapping, used when no lexicon tokens matched. [Method: VADER Lexicon]

Enrichment Reliability Flags
Per-module boolean gates that state whether downstream scores for each enrichment module are trustworthy for this post.

- sarcasm_reliable (bool, true / false): False when sarcasm_score is high but language is non-English or content too short to disambiguate. [Method: Heuristic Formula]
- emotion_reliable (bool, true / false): False when the emotion lexicon had too few matches or language was gated. [Method: Heuristic Formula]
- aspect_reliable (bool, true / false): False when no aspect keyword anchored the sliding window to real text. [Method: Heuristic Formula]
- topic_tags_reliable (bool, true / false): False when topic tags were drawn from placeholder/empty content. [Method: Heuristic Formula]
- financial_reliable (bool, true / false): False when financial signals fired from weak keyword evidence. [Method: Heuristic Formula]
- signals_reliable (bool, true / false): False when the general signal detection produced low-confidence matches. [Method: Heuristic Formula]
- llm_reliable (bool, true / false): False for posts where LLM enrichment was skipped (deterministic-only pipeline default). [Method: Heuristic Formula]
- quality_reliable (bool, true / false): False when text quality scoring ran on too-short or placeholder content. [Method: Heuristic Formula]
- classification_reliable (bool, true / false): False when content classification had conflicting evidence. [Method: Heuristic Formula]
- domain_reliable (bool, true / false): False when domain/vertical extraction could not be confidently assigned. [Method: Heuristic Formula]

8-Emotion Model
NRC-style emotion detection using a hand-curated lexicon of ~200 emotion-bearing tokens.

- emotion_primary (string, joy / anger / fear / sadness / surprise / disgust / anticipation / trust / neutral): Argmax selection after dampening polarity-contradictory emotions (×0.3) and mutual-exclusivity pruning of opposite pairs. Minimum threshold 0.15. [Method: Emotion Lexicon]
- emotion_secondary (string, Same 8 + "none"): Second-highest scoring emotion after all pruning rules applied. [Method: Emotion Lexicon]
- emotion_scores (list, Each 0.0 to 1.0): Dict of per-emotion word-match scores normalized to 0-1 per the NRC-style lexicon. [Method: Emotion Lexicon]

Sarcasm Detection
Pattern-based sarcasm detection with polarity inversion on positive detection.

- sarcasm_flag (bool, true / false): True when sarcasm_score ≥ 0.35. When active, sentiment_score × -0.74 (polarity flipped), labels swapped. [Method: Regex Pattern]
- sarcasm_score (float, 0.0 to 0.95): Max score from 30 regex patterns: /s tag (0.95), quoted compliments (0.72), "oh great" (0.80), semantic incongruence (0.55), excessive punctuation (0.55). [Method: Regex Pattern]
- irony_type (string, explicit_marker / quoted_compliment / semantic_incongruence / sarcasm / excessive_punctuation / none): Classified from the matched sarcasm pattern origin. [Method: Regex Pattern]
- sarcasm_confidence (float, 0.0 to 1.0): Confidence of the sarcasm verdict: mirrors the match score when sarcastic, 0.0 otherwise. [Method: Regex Pattern]

Aspect Sentiment
Singular post-level aspect representation via sliding-window VADER around the first matched aspect keyword.

- aspect_sentiment (string, positive / neutral / negative / mixed): Sentiment of the full sentence containing each aspect mention (brand, product, pricing, service, UX, security), aggregated across all mentions. [Method: Sentence-Bounded VADER]

Financial Signals
Crypto and market sentiment signals extracted from word lists.

- financial_signal (string, bullish / bearish / neutral): Net polarity of bullish vs. bearish financial keyword matches. [Method: Keyword/Regex]
- financial_score (float, -1.0 to +1.0): Weighted bullish minus bearish signal strength. [Method: Keyword/Regex]
- financial_bullish (float, 0.0 to 1.0): Proportion of bullish financial keyword hits. [Method: Keyword/Regex]
- financial_bearish (float, 0.0 to 1.0): Proportion of bearish financial keyword hits. [Method: Keyword/Regex]

Topic Classification
Multi-label topic tags assigned from curated keyword groups.

- topic_tags (list, 1+ topic strings): Multi-label topics (e.g. crypto, ai_ml, gaming, politics) matched via word-boundary keyword groups. [Method: Keyword/Regex]
- topic_coherence (float, 0.0 to 1.0): Ratio of top-topic keyword hits to the primary topic group size; gauges how focused the post is on its topics. [Method: Heuristic Formula]

Text Quality & Content Scoring
Multi-dimensional quality assessment using heuristic scoring formulas.

- text_quality_score (float, 0.0 to 1.0): Base 0.60. Bonuses: length ≥80 chars (+0.05), ≥300 (+0.05), ≥800 (+0.05); sentence punctuation (+0.05); uppercase start (+0.03). Penalties: URL density (-0.20 max), ALL-CAPS >50% (-0.20), word repetition (-0.15), emoji soup >15% (-0.12), repeated chars (-0.08). [Method: Heuristic Formula]
- toxicity_score (float, 0.0 to 1.0): Word-boundary, negation-aware lexicon of toxic and severe terms (incl. identity-based slurs) plus hate-speech patterns; severe/crisis terms weighted higher. A term inside a negated clause ("not toxic", "no hate") is excluded. Sum capped at 1.0. [Method: Keyword/Regex]
- intensity_score (float, 0.0 to 1.0): Combines: lexicon magnitude (sum of |valences| / tokens×3), booster density, exclamation/question mark density, ALL-CAPS ratio. [Method: Heuristic Formula]
- subjectivity_score (float, 0.0 to 1.0): Base 0.2 (short) or 0.3 (long) + 0.12 per matched first-person opinion/evaluative phrase. Matched with word boundaries so "love" won't match "glove" and "prefer" won't match "preferably". [Method: Heuristic Formula]
- formality_score (float, 0.0 to 1.0): 0.50 - informal_count×0.08 + formal_count×0.10. Word-boundary matched so "you " isn't counted as the "u " chat abbreviation. Informal: slang, contractions. Formal: academic, technical terms. [Method: Heuristic Formula]
- argument_quality (float, 0.0 to 1.0): strong_hits×0.12 + light_hits×0.05, matched with word boundaries so "data" won't match "database" and "study" won't match "student". Strong: because, therefore, evidence, studies show. Light: imo, tbh, think, believe. [Method: Heuristic Formula]
- evidence_quality (float, 0.0 to 1.0): loose_hits×0.08 + strict_hits×0.15 + light_hits×0.04 + numeric-evidence boost (e.g. 80%, $1M, n=500). Strict: n=, peer-reviewed, clinical trial, randomized, doi., arxiv. Word-boundary matched. [Method: Heuristic Formula]
- source_credibility (float, 0.0 to 1.0): max(0.15, pos_hits×0.08 - neg_hits×0.12), word-boundary matched so "resource" won't match "source". Positive: source, official, confirmed. Negative: allegedly, rumor, unconfirmed. [Method: Heuristic Formula]
- reading_level (string, beginner / intermediate / advanced / difficult): Flesch Reading Ease (syllable/sentence based) for posts ≥30 words; shorter posts fall back to a sentence-length proxy. Maps to beginner / intermediate / advanced / difficult tiers. [Method: Heuristic Formula]
- writing_style (string, narrative / technical / educational / opinionated): First match from 4 style keyword groups. [Method: Keyword/Regex]
- content_format (string, article / tutorial / news / review / listicle / case_study / discussion / question / announcement / event / opinion / project_showcase / rant / ama / recommendation): First match from 15 format keyword groups with opinion→article and question→discussion cascading. [Method: Keyword/Regex]
- has_content (bool, true / false): True when content is non-empty after stripping HTML and trimming. Empty content returns a minimal dict with quality_score=0.0. [Method: Heuristic Formula]
- content_length (int, 0+): Character count of the post content. Quality check recalculates when inconsistent with actual length. [Method: Direct Passthrough]
- content_too_short (bool, true / false): True when content_len < 15 (text) or < 5 (gaming platform) — triggers a pre-enrichment gate with quality_score=0.0. [Method: Heuristic Formula]
- avg_sentence_length (float, 0.0+): Average words per sentence used in the Flesch-Kincaid reading level. [Method: Heuristic Formula]
- flesch_reading_ease (float, 0.0 to 100.0): 206.835 - 1.015×sentence_length - 84.6×syllables_per_word. Higher = easier to read. [Method: Heuristic Formula]
- link_density (float, 0.0 to 1.0): Ratio of URLs to total word tokens. Penalizes text_quality_score by -0.20 max when > 0.15. [Method: Heuristic Formula]
- capitalization_ratio (float, 0.0 to 1.0): Ratio of uppercase alpha chars to all alpha chars. Used for ALL-CAPS penalty. [Method: Heuristic Formula]
- quality_score (float, 0.0 to 1.0): tq×0.40 + pipeline×0.20 + sc×0.15 + content×0.10 + (1-flags×0.1)×0.15. Unified overall quality score. [Method: Heuristic Formula]
- virality_score (float, 0.0 to 1.0): Log-engagement normalized to subscriber/follower context — measures how the post performs relative to audience. [Method: Heuristic Formula]
- reach_estimate (float, 0.0+): Estimated impressions: author_followers × (1 + engagement_rate) with platform reach multipliers, when followers are known. [Method: Heuristic Formula]

Content Flags & Labels
Boolean flags and labels that mark special content states — safety, spam, freshness, price targets.

- is_nsfw (bool, true / false): True when nsfw_keyword_count ≥ 2 or weighted_nsfw_score ≥ 0.3 against 18 NSFW keywords (weights 0.08-0.25). Flows through the full pipeline — no hardcoded sentiment. [Method: Keyword/Regex]
- nsfw_flag (bool, true / false): Boolean alias of is_nsfw set earlier in the pipeline for consistent gating. [Method: Keyword/Regex]
- is_opinion (bool, true / false): True when subjectivity or explicit opinion markers are present. [Method: Keyword/Regex]
- is_fud (bool, true / false): Fear-Uncertainty-Doubt signal for financial markets (token-panic phrasing); negation-aware so "not a scam" or "no panic" is excluded. [Method: Keyword/Regex]
- is_fomo (bool, true / false): Fear-of-missing-out signal for crypto — urgency markers ("missed", "getting left behind"); negation-aware. [Method: Keyword/Regex]
- has_price_target (bool, true / false): True when a numeric price target ($XX, X$K, etc.) is present. [Method: Regex Pattern]
- has_dd (bool, true / false): Due-diligence / deep-analysis post marker (r/DD style). [Method: Keyword/Regex]
- volatility_signal (float, 0.0 to 1.0): Volatility intensity from price-shock and market-movement keywords. [Method: Heuristic Formula]
- is_stale_content (bool, true / false): True when the post is significantly older than the crawl reference and re-content age features detect staleness. [Method: Heuristic Formula]
- content_age_days (int, 0+): Age of the post in days at enricment time. [Method: Heuristic Formula]
- content_source_type (string, text / placeholder / email_only / hash_only / link_only / reddit_automod / platform_name / url_only): Classifies trick/non-substantive content so the pipeline short-circuits appropriately. [Method: Keyword/Regex]
- is_phishing (bool, true / false): True on phishing patterns (login lookalike, credential harvesting text). [Method: Keyword/Regex]
- is_spam (bool, true / false): True on spam patterns (repeated promo links, template wording, high URL density). [Method: Keyword/Regex]
- low_relevance (bool, true / false): True when the post is off-topic relative to catalog verticals (non-tech/non-relevant content). [Method: Keyword/Regex]

Syndication & Cross-Platform
Detection of content reposted across platforms (cross-post syndication).

- syndication_detected (bool, true / false): True when identical/highly-similar content is observed on multiple platforms. [Method: Heuristic Formula]
- original_platform (string, platform name or ""): The platform where the syndicated content first appeared (earliest timestamp). [Method: Heuristic Formula]
- cross_platform_reach (int, 0+ or null): Number of platforms the same content was syndicated to. [Method: Heuristic Formula]

Pipeline Metadata
Operational metadata about the enrichment run itself.

- enriched_at (string, ISO 8601): Timestamp when the enrichment pipeline processed this post. [Method: Direct Passthrough]
- quality_score (float, 0.0 to 1.0): Unified overall quality score (tq×0.40 + pipeline×0.20 + sc×0.15 + content×0.10 + flags components). [Method: Heuristic Formula]
- pipeline_complete (bool, true / false): True when all enrichment modules ran to completion without a crash-wrapped short-circuit. [Method: Heuristic Formula]

Content Length Metrics
Size and reading-time metrics derived from raw content length.

- content_length (int, 0+): Character count of the post content. [Method: Direct Passthrough]
- word_count (int, 0+): Word token count (CJK-aware — counts CJK characters individually). [Method: Direct Passthrough]
- reading_time_minutes (int, 0+): Estimated reading time = ceil(word_count / 200). [Method: Heuristic Formula]

Location & Geo Extraction
Geographic entities mentioned in the post.

- location_mentioned (bool, true / false): True when any location entity is detected. [Method: Keyword/Regex]
- locations_mentioned (list, 0+ strings): All matched location entities. [Method: Keyword/Regex]
- countries_mentioned (list, 0+ strings): Sub-list of matched countries. [Method: Keyword/Regex]
- cities_mentioned (list, 0+ strings): Sub-list of matched cities. [Method: Keyword/Regex]
- regions_mentioned (list, 0+ strings): Sub-list of matched regions/states. [Method: Keyword/Regex]
- primary_geo (string, location string or ""): Highest-confidence location entity. [Method: Keyword/Regex]
- geo_granularity (string, city / region / country / none): Granularity of the strongest detected location. [Method: Heuristic Formula]

Question & Request Intent
Detection of posts that ask questions or request information — useful for support/UGC research.

- is_question (bool, true / false): True when the post asks a question (wh-, how, or question-mark based). [Method: Keyword/Regex]
- question_type (string, informational / recommendation / how_to / yes_no / or:): Type of question expressed. [Method: Keyword/Regex]
- request_type (string, help / tip / recommendation / referral / feedback / none): Class of request/looking-for content. [Method: Keyword/Regex]
- question_markers (list, 0+ strings): The matched question markers (why, how, what, does anyone, etc.). [Method: Keyword/Regex]

Statistics & Data Density
Detection of posts that contain quantitative data, useful for research analytics.

- has_statistics (bool, true / false): True when numeric tokens exceed a density threshold. [Method: Heuristic Formula]
- statistics_type (string, numeric / percentage / ratio / currency / none): Dominant numeric form in the post. [Method: Regex Pattern]
- numeric_density (float, 0.0 to 1.0): Ratio of numeric tokens to total tokens. [Method: Heuristic Formula]
- statistics_keywords (list, 0+ strings): Data-language keywords (%, x%, %, yoy, market cap, "n=" and more). [Method: Keyword/Regex]

Tech Stack Extraction
Programming languages and tools mentioned in developer content.

- tech_stack (list, 0+ strings): Languages/frameworks/db names explicitly mentioned (Python, React, PostgreSQL,...). [Method: Keyword/Regex]
- language_github (string, programming language or ""): Primary Github repository language passed through. [Method: Direct Passthrough]

Deduplication & Hashing
Deterministic content hashes used to detect near-duplicates and cross-post syndication.

- content_hash (string, 32 hex chars): SHA-256(user_content + ":" + content)[:32] for cross-platform dedup. [Method: SHA-256 Hash]
- content_hash_md5 (string, 32 hex chars): MD5 of normalized (whitespace-collapsed, lowercased) content for internal tracking. [Method: SHA-256 Hash]
- is_near_duplicate (bool, true / false): True when content is a near-duplicate of an already seen hash family. [Method: Heuristic Formula]

Political & Civic
Detection of politically-leaning or civic-discussion posts for researcher segmentation.

- political_lean (string, left / right / center / libertarian / socialist / non_political): Assigns a lean direction when the post uses politically-identifying language; left/right/strong keyword counts are negation-aware (e.g. "not a liberal" is excluded). [Method: Keyword/Regex]
- policy_topic (string, civic policy topic): First matched policy domain (e.g. climate, healthcare, tax, education). [Method: Keyword/Regex]
- stance (string, supportive / opposed / neutral): Sentiment-ish stance toward the policy discussed. [Method: Keyword/Regex]
- party_mentioned (list, 0+ strings): Political parties explicitly mentioned. [Method: Keyword/Regex]
- candidate_mentioned (list, 0+ strings): Political candidates explicitly mentioned. [Method: Keyword/Regex]
- argument_quality (float, 0.0 to 1.0): Reuses the reasoning-evidence scoring to gauge argument depth in political posts. [Method: Heuristic Formula]
- civic_topic (string, civic theme or ""): Civic-engagement topic the post belongs to (voting, protest, governance). [Method: Keyword/Regex]

Gaming
Game title / esports / hype metadata extracted from gaming posts.

- game_title (list, 0+ strings): Game titles matched against a known-game dictionary. [Method: Keyword/Regex]
- game_genre (string, shooter / rpg / strategy / moBA / puzzle / horror / simulation / sports / racing / none): Dominant game genre detected in the post. [Method: Keyword/Regex]
- platform_type (string, pc / console / handheld / mobile / cloud / none): Platform (PC/console/mobile) the post is about. [Method: Keyword/Regex]
- hype_score (float, 0.0 to 1.0): Hype-signal density from explicit hype vocabulary ("hype", "OMG release"). [Method: Heuristic Formula]
- content_type (string, gameplay / review / guide / news / discussion / none): Type of gaming content in the post. [Method: Keyword/Regex]
- esports_event (string, event string or ""): Esports tournament/event matched. [Method: Keyword/Regex]
- patch_mentioned (string, patch string or ""): Game patch/version explicitly mentioned. [Method: Keyword/Regex]
- release_signal (string, released / announced / delayed / none): Status signal about a game release. [Method: Keyword/Regex]

AI & ML
Model / framework mentions and technical-depth gauging for AI content (paper discussion, benchmarks).

- models_mentioned (list, 0+ strings): Named ML models matched (GPT-4, LLaMA, SDXL, ...). [Method: Keyword/Regex]
- techniques_mentioned (list, 0+ strings): ML techniques matched (fine-tuning, RLHF, diffusion...). [Method: Keyword/Regex]
- frameworks_mentioned (list, 0+ strings): ML frameworks matched (PyTorch, TensorFlow, ...). [Method: Keyword/Regex]
- is_paper_discussion (bool, true / false): True when the post discusses an academic paper. [Method: Keyword/Regex]
- is_benchmark (bool, true / false): True when the post a benchmark (bleu, mmlu, ...) comparison. [Method: Keyword/Regex]
- technical_depth (float, 0.0 to 1.0): Scales by technical-vocabulary density (architectures, loss functions...). [Method: Keyword/Regex]
- ml_topic (string, llm / generative / reinforcement / classic-ast / computer-vision / nlp / none): Dominant ML sub-domain. [Method: Keyword/Regex]
- open_source_flag (bool, true / false): True when the post references open-source models/repos. [Method: Keyword/Regex]
- alignment_topic (bool, true / false): True when the post about AI alignment / safety. [Method: Keyword/Regex]

Health & Wellness
Health-condition and wellness-topic metadata, and whether claims have scientific backing.

- condition_type (string, mental / physical / chronic / disease / none): Primary health-condition type discussed. [Method: Keyword/Regex]
- wellness_topic (string, nutrition / fitness / sleep / meditation / none): Wellness vertical the post belongs to. [Method: Keyword/Regex]
- support_type (string, peer / professional / community / none): Type of support the post is seeking or offering. [Method: Keyword/Regex]
- is_professional_content (bool, true / false): True when authored/presented by professional (doctor, therapist...) context. [Method: Keyword/Regex]
- health_has_scientific_ref (bool, true / false): True when the health claim cites scientific evidence (study, DOI, trial). [Method: Keyword/Regex]
- evidence_quality (float, 0.0 to 1.0): Reuses loose/strict evidence-hit scoring for health claims. [Method: Heuristic Formula]
- mental_health_flag (bool, true / false): True when mental-health sensitive topic terms present. [Method: Keyword/Regex]
- medication_mentioned (bool, true / false): True when medication/drug names are mentioned. [Method: Keyword/Regex]
- is_progress_post (bool, true / false): True when the fitness/health post tracks progress updates. [Method: Keyword/Regex]

Climate & Environment
Climate topic classification with ESG- and renewable-energy-specific sub-fields.

- climate_topic (string, climate-weather / renewable / policy / impact / none): Dominant climate topic in the post. [Method: Keyword/Regex]
- solution_type (string, renewable / carbon-capture / efficiency / adaptation / none): Climate solution the post discusses. [Method: Keyword/Regex]
- climate_has_scientific_ref (bool, true / false): True when a scientific reference grounds climate claims. [Method: Keyword/Regex]
- policy_mentioned (bool, true / false): True when climate policy instruments are named (tax, cap-and-trade...). [Method: Keyword/Regex]
- carbon_signal (string, positive / negative / neutral / none): Sentiment-toned carbon footprint signal in the post. [Method: Keyword/Regex]
- renewable_energy_type (string, solar / wind / hydro / geothermal / nuclear / none): Renewable energy source discussed. [Method: Keyword/Regex]
- esg_signal (string, esg / greenwashing / none): ESG claims vs greenwashing detection. [Method: Keyword/Regex]
- corporate_target_mentioned (bool, true / false): True when corporate carbon targets are mentioned. [Method: Keyword/Regex]

Crypto & DeFi
Token / coin mentions and market-movement signals with pump/exchange/protocol annotation.

- coins_mentioned (list, 0+ strings): Coin names/symbols matched against a curated dictionary. [Method: Keyword/Regex]
- coin_count (int, 0+): Number of distinct coins/namesets matched. [Method: Keyword/Regex]
- pump_signal (bool, true / false): True when pump-group phrasing detected ("to the moon", "#pump"). [Method: Keyword/Regex]
- exchange_mentioned (list, 0+ strings): Exchanges named (Binance, Coinbase...). [Method: Keyword/Regex]
- defi_protocol_mentioned (list, 0+ strings): DeFi protocols named (Uniswap, Aave...). [Method: Keyword/Regex]
- nft_mentioned (bool, true / false): True when NFT/collections terminology present. [Method: Keyword/Regex]
- whale_activity_signal (bool, true / false): True when large-holder ("whale") activity mentioned. [Method: Keyword/Regex]

Jobs & Career
Job posting and career Q&A extraction: title, salary, seniority, remote policy.

- job_title (string, title or ""): Job title matched from the post. [Method: Keyword/Regex]
- company_mentioned (string, company or ""): Company hiring or referenced. [Method: Keyword/Regex]
- visa_sponsorship_mentioned (bool, true / false): True when visa sponsorship is mentioned. [Method: Keyword/Regex]
- salary_mentioned (bool, true / false): True when any salary figure is present. [Method: Regex Pattern]
- salary_min (int, 0+ or null): Lower salary bound parsed from "$XXk-$YYk" patterns. [Method: Regex Pattern]
- salary_max (int, 0+ or null): Upper salary bound parsed from "$XXk-$YYk" patterns. [Method: Regex Pattern]
- yoe (int, 0+ or null): Years-of-experience requirement parsed, when stated. [Method: Keyword/Regex]
- seniority (string, entry / mid / senior / lead / staff / executive / none): Seniority level detected. [Method: Keyword/Regex]
- remote_policy (string, remote / hybrid / on_site / none): Location-structure of the role. [Method: Keyword/Regex]
- role_type (string, engineer / data / design / product / marketing / sales / other / none): Role function level for job posts. [Method: Keyword/Regex]
- company_size (int, 0+ or null): Company headcount when stated. [Method: Keyword/Regex]

Creator Economy
Influencer / creator partnership metadata: monetization, revenue model, sponsorships.

- creator_platform (string, tiktok / youtube / instagram / x / twitch / onlyfans / none): Creator platform the post is about. [Method: Keyword/Regex]
- monetization_type (string, ads / subscriptions / sponsorships / tips / merch / none): How the creator monetizes. [Method: Keyword/Regex]
- follower_tier (string, micro (<10k) / mid (10k-100k) / macro (100k-1M) / mega (1M+) / none): Follower-size tier when a creator is discussed. [Method: Keyword/Regex]
- revenue_model (string, ads / subs / affiliate / digital_goods / none): Revenue-model pattern detected. [Method: Keyword/Regex]
- sponsorship_mentioned (bool, true / false): True when a brand sponsorship or deal is mentioned. [Method: Keyword/Regex]
- niche_topic (string, niche or ""): Creator content niche (fitness, gaming, ASMR...). [Method: Keyword/Regex]
- collab_mentioned (bool, true / false): True when a creator collaboration is mentioned. [Method: Keyword/Regex]

Meme & Viral Culture
Viral-culture metadata: meme formats, origin, spread velocity.

- meme_format (string, macro / remix / template / cap / boomer / none): Meme format detected. [Method: Keyword/Regex]
- origin_platform (string, platform or ""): Platform where the meme originated (tiktok, reddit...). [Method: Keyword/Regex]
- spread_velocity (float, 0.0 to 1.0): How fast the meme spread (based on hit timing data). [Method: Heuristic Formula]
- cultural_reference (string, reference or ""): Pop-culture reference in the meme. [Method: Keyword/Regex]
- reaction_type (string, clapping_cap / 🙌 / lol / gif / none): Reaction-content type (recipe-format humor). [Method: Keyword/Regex]

Misinformation & Narratives
Claim-type classification, verification status, bot/echo-chamber signals.

- claim_type (string, scientific / health / political / financial / neutral): Category of claim being made. [Method: Keyword/Regex]
- verified_flag (bool, true / false): True when the claim can be considered already verified by context. [Method: Heuristic Formula]
- debunked_flag (bool, true / false): True when debunk markers (debunk, myth, false) are present. [Method: Keyword/Regex]
- source_credibility (float, 0.0 to 1.0): max(0.15, pos hits×0.08 - neg hits×0.12), word-boundary matched, for misinformation-adjacent posts. [Method: Heuristic Formula]
- echo_chamber_signal (float, 0.0 to 1.0): How strongly the post reinforces existing beliefs (repetition of core memes/phrases). [Method: Heuristic Formula]
- bot_signal (bool, true / false): True when bot-like behavior detected (template text, high URL/hashtag density, low lexical diversity). [Method: Heuristic Formula]
- bot_likelihood_score (float, 0.0 to 1.0): Continuous bot probability from the signal stack. [Method: Heuristic Formula]
- coordinated_inauthentic_flag (bool, true / false): True when coordinated inauthentic behavior is suspected across posts. [Method: Heuristic Formula]
- narrative_type (string, narrative or ""): Story-sharing narrative frame in the post. [Method: Keyword/Regex]
- fact_check_url (string, URL or ""): Link to a fact-check reference when included. [Method: Keyword/Regex]

Extremism Signals
Hate / extremist / recruitment language detection used to exclude that content from consumer datasets (when configured).

- ideology_type (string, classified ideology or ""): Ideology category, if one is matched against the curated lexicon. [Method: Keyword/Regex]
- target_group (string, target group or ""): group targeted by hate content, when matchable. [Method: Keyword/Regex]
- hate_signal (bool, true / false): True when hate-language or coded/identity-based slur patterns are detected (expanded lexicon incl. antisemitic and other slur dogwhistles); matched with word boundaries and negation-awareness. [Method: Keyword/Regex]
- recruitment_signal (bool, true / false): True when recruitment wording (join, DM me, telegram...2 recruitment phrasing) present. [Method: Keyword/Regex]
- propaganda_format (string, format or ""): Propaganda stylistic format matched. [Method: Keyword/Regex]
- platform_ban_signal (bool, true / false): True when ban-evasion / platform-policy signals present. [Method: Keyword/Regex]

Live Streaming
Live / VOD metadata for streaming platforms: viewers, chat, subs, VOD availability.

- stream_platform (string, twitch / youtube / kick / rumble / other / none): Streaming platform the post is about. [Method: Keyword/Regex]
- viewer_peak (int, 0+ or null): Peak simultaneous viewers reported. [Method: Direct Passthrough]
- chat_velocity (float, 0.0 to 1.0): Chat-message velocity relative to viewers (rate, not count). [Method: Heuristic Formula]
- subscriber_count (int, 0+ or null): Current subscriber/follower count for the channel. [Method: Direct Passthrough]
- stream_category (string, stream category or ""): Category/game being streamed. [Method: Direct Passthrough]
- vod_available (bool, true / false): True when a VOD of the stream is available. [Method: Direct Passthrough]

E-commerce & Reviews
Product reviews and retailer commentary: ratings, verification, review organization.

- product_mentioned (string, product or ""): Product name/model referred to in the review. [Method: Keyword/Regex]
- price_tier (string, budget / mid / premium / luxury / none): Price tier inferred for the product discussed. [Method: Keyword/Regex]
- business_model (string, d2c / b2b / b2c / marketplace / subscription / none): Business model of the product/company. [Method: Keyword/Regex]
- business_stage (string, seed / series_a / series_b / late / public / none): Stage of the business being discussed. [Method: Keyword/Regex]
- niche_category (string, category or ""): E-commerce niche (electronics, fashion, home...). [Method: Keyword/Regex]
- refund_signal (bool, true / false): True when refund or return issues are mentioned. [Method: Keyword/Regex]
- review_rating (float, 0.0 to 5.0): Star/X rating parsed from the review (X/10, X/5, ★). [Method: Regex Pattern]
- verified_purchase (bool, true / false): True when the review is marked as a verified purchase. [Method: Keyword/Regex]
- review_helpful_count (int, 0+): Number of people who found the review helpful. [Method: Direct Passthrough]
- review_response (string, string or ""): Text of the merchant response to the review. [Method: Direct Passthrough]
- review_recommend (string, yes / no / mixed): Aggregated verdict stance in the review (recommend or avoid). [Method: Keyword/Regex]
- review_title (string, string or ""): Title of the review post. [Method: Direct Passthrough]
- pros_mentioned (list, 0+ strings): Positives explicitly listed by the reviewer. [Method: Keyword/Regex]
- cons_mentioned (list, 0+ strings): Negatives explicitly listed by the reviewer. [Method: Keyword/Regex]
- management_response (string, string or ""): Management/merchant reply text when present. [Method: Direct Passthrough]
- rating_boost (float, 0.0 to 1.0): Adjustment factor applied to review_rating by verified_purchase/recency weighting. [Method: Heuristic Formula]

Fediverse
Federation metadata for Mastodon/lemmy/pleroma/Misskey posts.

- instance_domain (string, domain or ""): The instance domain the post lives on. [Method: Direct Passthrough]
- federation_protocol (string, activitypub / nostr / matrix / atproto / none): Federation protocol used to transport the post. [Method: Direct Passthrough]
- software_type (string, mastodon / lemmy / pleroma / misskey / other / none): Server software serving the instance. [Method: Direct Passthrough]
- boost_count (int, 0+): Number of boosts/reblog of the post (federated share-language ). [Method: Direct Passthrough]
- reply_count (int, 0+): Replies to the post (federated). [Method: Direct Passthrough]
- local_only_flag (bool, true / false): True if the post is local-only (visibility to local server). [Method: Direct Passthrough]
- content_warning (string, C with warning or ""): Content-warning subject line set by the author. [Method: Direct Passthrough]
- visibility_scope (string, public / unlisted / private / followers / direct): Post visibility scope on the network. [Method: Direct Passthrough]

Enrichment Methods:
- VADER Lexicon: Valence-scored lexicon of ~7,500 English tokens. Compound = sum / sqrt(sum² + raw_sum² × 0.0001 + 15). Handles negation (×-0.74 within 3 tokens), booster words, ALL-CAPS (+0.25), emoji, and slang (lmao, af).
- Keyword/Regex: Word-boundary regex matching (\bkeyword\b) against curated keyword lists — so "data" never matches "database" and "source" never matches "resource". Stance/signal detectors (bull/bear, FUD/FOMO/pump/shill, misinfo, political lean) are negation-aware: a match inside a negated clause (e.g. "not bullish", "not a scam") is excluded. Used for topic_tags, hate_signal, financial_bullish/bearish, and 143 other fields.
- Heuristic Formula: Weighted scoring formulas combining multiple inputs. Examples: text_quality_score = base 0.60 + length bonuses + density penalties. quality_score = tq×0.40 + pipeline×0.20 + sc×0.15 + content×0.10 + (1-flags×0.1)×0.15.
- Regex Pattern: Pattern-based regex extraction: sarcasm markers (30 patterns: /s tag, quoted compliments, "oh great"), salary ranges ($XXk-$YYk), review ratings (X/10, ★★★★☆), tickers ($BTC).
- Direct Passthrough: Raw metadata from source platform — no computation applied. Includes engagement counts (likes, comments, shares), timestamps, platform fields (board, thread_id), GitHub metrics, YouTube tags.
- Heuristic: Script-detection + function-word frequency scoring across 7 languages. No ML models used.
- Emotion Lexicon: NRC-style emotion lexicon of ~200 tokens mapped to 8 emotions (joy, anger, fear, sadness, surprise, disgust, anticipation, trust). Token matches are negation-aware — only non-negated occurrences count (e.g. "no joy", "without fear"). Scores normalized 0-1 with mutual exclusivity pruning of opposite pairs.
- Sentence-Bounded VADER: VADER computed over the full containing sentence of each aspect mention (brand, product, pricing, service, UX, security), aggregating all mentions per aspect rather than a fixed char window. Avoids truncating mid-sentence and bleeding into adjacent-sentence polarity. Used for aspect_* fields.
- SHA-256 Hash: SHA-256(user_content + ":" + content)[:32] for cross-platform deduplication. MD5 of normalized content for internal tracking.