The foundation of everything we build

Data quality, baked in.

Every dataset passes through a multi-layer quality pipeline — from collection to delivery — ensuring production-ready data for AI training, research, and analytics.

Quality Stages

Collection to delivery

Enrichment Checks

Deterministic auto-corrections

Structural Fixes

Phase 1 + Phase 2 repairs

Dedup Strategies

Hash, URL, author, content

▸ Quality Pipeline

6-stage quality pipeline.

Every stage has specific quality gates — posts failing any gate are flagged, corrected, or rejected.

1. Collection

Raw data is scraped from 100+ platforms and quality-checked before it ever touches the database.

Content hash generated for deduplication
Post-level quality scoring (0-100)
NSFW content filtered out
Duplicate detection against existing DB hashes
Intra-batch duplicate removal

2. Post Validation

Every post is validated across 7 dimensions before insertion. Posts scoring below 50/100 are rejected.

Content: min length, truncation, duplicate title
Author: missing, platform name, email/URL/hash
URL: missing, signup/login patterns, domain mismatch
Metrics: negatives, outliers, implausible ratios
Dates: future timestamps, excessive age (3yr+)
Content source: placeholder, URL-only, RSS teaser
Engagement sanity: likes > views, comments > views

3. AI Enrichment

150+ enrichment fields computed per post: sentiment, emotions, topics, financial signals, and more.

VADER sentiment + confidence scoring
8-emotion NRC model (joy, trust, fear, anger…)
Topic classification (54+ domains)
Financial signal detection (FUD, FOMO, pump)
Sarcasm & subjectivity scoring
Language detection (15 language codes)
Content quality & intensity scoring

4. Quality Correction

20 deterministic validation checks auto-correct enrichment data. Errors trigger pipeline flags.

Hate speech → force negative sentiment
Emotion score capping (max threshold)
Sentiment↔emotion contradiction auto-fix
Tone alignment with sentiment label
Language-VADER mismatch detection
Phishing/scam content override
Timestamp validation & future-date fix
Platform mismatch correction
Missing critical fields filled with defaults

5. Export & Dedup

Multi-strategy deduplication runs at export time. Schema is standardized across all formats.

Content-hash dedup (most reliable)
URL dedup (exact match)
Title + author combination dedup
Content-similarity dedup (first 200 chars)
Standardized column ordering
CSV · JSON · JSONL · Parquet formats

6. S3 Delivery

Cleaned data streams directly from S3. Catalog refresh updates record counts in real time.

Files uploaded to S3 with correct headers
CloudFront CDN for fast global delivery
Catalog refresh updates DB record counts
API cache invalidation after refresh
Correct Content-Disposition for downloads

▸ Enrichment Quality

20 deterministic quality checks.

Each check runs after AI enrichment. Auto-corrections fix issues silently; errors set pipeline flags.

Hate Speech Overrideerror

Language-Sentiment Consistencywarning

Emotion Score Cappinginfo

Syndication Evidencewarning

Pipeline Completenesswarning

Engagement Rate Capinfo

Tone-Sentiment Alignmentinfo

Sentiment-Emotion Contradictionwarning

Language-VADER Mismatchwarning

Sentinel Value Removalinfo

Tech Stack Validationinfo

Content Length Recalcinfo

Emotion Mutual Exclusivityinfo

Phishing/Scam Detectionerror

Spam Detectionwarning

Platform Mismatchinfo

Engagement Sanityinfo

Language Code Validationwarning

Timestamp Validationwarning

Missing Null Fallbacksinfo

▸ Database Repairs

19 structural & enrichment fixes.

The fix_quality.py script runs idempotently across all datasets, repairing data integrity at scale.

Phase 1 — Structural

1. Duplicate Records

Remove exact + content-hash duplicates from the database

2. Missing Authors

Fill from platform-appropriate fallback values

3. author_followers 0→NULL

Distinguish "unknown" from "0 followers"

4. HTML Stripping

Strip HTML tags from content fields

5. Engagement Rate

Recalculate from raw metric values

6. Bad URLs

Remove signup/login/search pattern URLs

7. Placeholder Content

Remove "click here", "loading…", "see more"

8. Ingestion Timestamps

Null out ingestion dates, not creation dates

9. Off-topic Noise

Remove blocked platform data from wrong datasets

10-12. Metric Mappings

YouTube views/shares, Reddit views, etc.

Phase 2 — Enrichment

A. Tone Correction

Tone derived from sentiment label, not emotion

B. Contradictory Emotions

Mutual exclusivity + polarity alignment

C. Sentinel Values

Replace -1→None, "n/a"→null across fields

D. Language Detection

Re-run detection on unknown-language records

E. Inapplicable Fields

Clear platform-specific fields from wrong records

F. UI Boilerplate

Strip "Skip to Navigation" etc. from content

G. URL/Platform Mismatch

Fix records where URL doesn't match platform

▸ Schema & Standards

Consistent schemas, every time.

Every dataset follows the same 130+ column standard, regardless of platform or format.

Standardized Schema

130+ columns with consistent ordering across all datasets. Core identifiers first, then content, metrics, enrichment fields.

Same columns for every platform
Predictable ordering
Backward-compatible additions

Dedup at Every Level

Four complementary strategies ensure no duplicate records reach your pipeline.

Content hash (SHA-256)
URL exact match
Title + author composite
Content-similarity check

Idempotent Repairs

The fix script is safe to run any number of times. All operations are idempotent.

Dry-run mode for preview
Per-step granular control
Enrichment-only mode
Safe for production use

▸ Get started

Ready for quality data?

Browse the catalog, download a free sample, and see the quality for yourself. No account needed.

Browse Datasets All Features

SocialINTEL

Fetching datasets…