Scaling Data Quality Assurance Across 100+ Social Media Platforms

Data quality is the silent killer of AI projects. Models trained on noisy, incomplete, or biased data produce unreliable outputs — and the problem compounds silently through downstream systems. When your data spans 100+ platforms with wildly different formats, quality assurance becomes an engineering discipline in itself.

The Quality Challenge

Consider what "a social media post" means across platforms:

• Reddit — Title, body, comments, votes, flair, nested threading
• YouTube — Title, description, comments, view counts, channel metadata
• Mastodon — Content, boosts, favorites, instance metadata, content warnings
• 4chan — Anonymous posts, image attachments, board-specific norms
• GitHub — Issues, pull requests, code comments, structured metadata

Each platform produces different fields, at different quality levels, with different noise characteristics.

Our Quality Scoring Framework

Every record in our system receives a composite quality score from 0-100, computed from several dimensions:

Completeness Score (0-25)

Measures how many expected fields are present and populated. A Reddit post with title, body, score, and comments scores higher than one with only a title. Missing fields reduce the score proportionally.

Reliability Score (0-25)

Evaluates source trustworthiness based on:

• Account age and activity history — Established accounts are more reliable
• Bot probability — Higher bot scores reduce reliability
• Content quality signals — Grammar, coherence, effort level
• Engagement authenticity — Flagged engagement patterns reduce score

Freshness Score (0-25)

Temporal relevance matters. Recent data scores higher, with decay curves tuned per use case. Real-time applications need fresh data; historical analysis values temporal depth.

Signal Strength Score (0-25)

Measures how much useful information the record contains. High-engagement posts, detailed opinions, and content-rich posts score higher than low-effort reactions or single-word replies.

Platform-Specific Calibrations

We maintain platform-specific quality thresholds because quality distributions vary:

• GitHub — High baseline quality (structured, technical content)
• Reddit — Wide variance; top-subreddit posts are high quality, some subreddits produce mostly noise
• 4chan — Low average quality; useful signals require aggressive filtering
• YouTube comments — Bimodal distribution; insightful reviews alongside emoji-only reactions

Automated Quality Monitoring

We run continuous quality checks:

1. Daily sampling — Random samples are scored and compared against historical baselines
2. Anomaly detection — Sudden drops in quality metrics trigger alerts
3. Collector health — Quality degradation often signals a broken or rate-limited collector
4. Drift monitoring — Track how quality distributions shift over time

Impact on Downstream Systems

Quality scores directly impact:

• Dataset exports — Customers can filter by minimum quality score
• Enrichment accuracy — Higher-quality input produces higher-quality enrichment
• Training data curation — Quality-weighted sampling improves model performance
• Research validity — Quality-aware analysis produces more reliable insights

— Heshan Sanjuka, Founder