Designing a Content Moderation System

Content moderation ensures user-generated content complies with platform policies at scale.

Requirements

• Review 10M posts/day
• <100ms auto-moderation
• Human review for edge cases
• Appeals process
• Multi-modal: text, images, video

Moderation Pipeline

Content Submitted
      ↓
Pre-moderation (fast, auto)
  ├── Hash matching (known bad content)
  ├── Rule-based filters (keywords)
  └── ML classifier (confidence score)
      ↓
High confidence violation → Auto-remove
Low confidence → Human review queue
Clearly clean → Publish

Hash Matching

PhotoDNA / hash matching for known CSAM and copyrighted content:

hash(image) → lookup in known violation database
Match found → Auto-remove (0 false positives)

Text Classification

Input: "Buy cheap pills here clickme.xyz"

Classifiers:
  Spam:       0.97
  Hate speech: 0.12
  Violence:   0.03
  Adult:      0.05

Action: Auto-remove (spam > 0.95)

Human Review Queue

Content with confidence 0.5-0.95:
    ↓
Review Queue (priority by virality/reports)
    ↓
Moderator reviews → Remove / Approve / Escalate
    ↓
Decision fed back to ML training pipeline

Appeals Process

User appeals removal
    ↓
Appeal Queue (separate, senior moderators)
    ↓
Review original + appeals context
    ↓
Uphold / Overturn decision
    ↓
If overturned → Retrain ML (false positive)

Metrics

Precision: % of removed content that was truly violating
Recall:    % of violating content that was removed
Over-removal rate: Good content removed (user trust)
Under-removal rate: Bad content published (safety)

Conclusion

Effective moderation is a three-layer system: instant hash/rule-based for known bad content, ML for probabilistic cases, and humans for edge cases.