Designing a Chaos Engineering Framework

Chaos engineering deliberately injects failures to discover weaknesses before users do.

Principles

"If something hurts, do it more often" — Martin Fowler

1. Define steady state (what does healthy look like?)
2. Hypothesize: This failure won't degrade steady state
3. Introduce chaos in production (or staging)
4. Compare to control group
5. Fix weaknesses discovered
6. Increase blast radius gradually

Types of Failure Injection

Infrastructure:
  Kill random instances (Chaos Monkey)
  Kill entire availability zone
  Network partition between services

Network:
  Add latency (100ms, 500ms)
  Packet loss (1%, 10%)
  Bandwidth throttling
  DNS failure

Application:
  Return 500 errors from dependencies
  Slow down specific API calls
  Exhaust thread pool
  Fill disk to 95%
  Memory leak simulation

Netflix Simian Army

Chaos Monkey:      Randomly kills instances
Chaos Gorilla:     Kills entire AWS AZ
Chaos Kong:        Kills entire AWS region
Latency Monkey:    Injects network delays
Conformity Monkey: Finds non-compliant instances

Safety Controls

Blast radius limiting:
  Never kill more than 1 instance per service
  Only run during business hours
  Automatically stop if error rate > 10%
  Only in regions with redundancy

Rollback:
  One command to stop all chaos
  Runbook for each experiment

Game Days

Scheduled: Quarterly regional failover drills
Unannounced: Random chaos during business hours
Process:
  1. Announce experiment to stakeholders
  2. Define success criteria
  3. Execute experiment
  4. Observe system behavior
  5. Debrief: What worked? What didn't?
  6. Fix discovered weaknesses

Measuring Resilience

MTTR (Mean Time to Recover): How fast do we detect + fix?
MTTF (Mean Time to Failure): How long between failures?
Error budget: How much unreliability is acceptable?

Target: MTTR < 5 minutes for P1 incidents

Tools

Litmus Chaos (Kubernetes)
AWS Fault Injection Simulator
Gremlin (enterprise)
Chaos Toolkit (open-source)

Conclusion

Chaos engineering shifts you from reactive to proactive reliability. Start small (kill one instance), measure impact, fix weaknesses, and gradually increase blast radius.