Designing a Service Level Objective (SLO) Framework

SLOs quantify reliability as a contract between engineering and customers, driving data-driven prioritization decisions.

Hierarchy

SLI (Service Level Indicator):
  Metric measuring service behavior
  Example: % of requests returning 200 OK

SLO (Service Level Objective):
  Target for an SLI
  Example: 99.9% of requests should succeed

SLA (Service Level Agreement):
  Legal commitment to customers
  Example: 99.9% uptime or 10% credit refund

SLA ≤ SLO ≤ Actual reliability

Good SLI Types

Availability:  % of requests succeeding
  Good: HTTP 200-499 (non-5xx)
  Total: All requests

Latency:       % of requests faster than threshold
  Good: requests < 300ms
  Total: All requests

Freshness:     % of data updated within target time
  Good: records updated < 10 min ago
  Total: All records

Correctness:   % of results matching expected output

Error Budget

SLO: 99.9% availability
Error budget: 100% - 99.9% = 0.1%

Over 30 days (43,200 minutes):
  Allowed downtime: 43.2 minutes/month

Budget tracking:
  Budget remaining = 43.2 - actual_downtime_minutes
  Rate: burned / allowed budget per day

Burn Rate Alerting

Burn rate = actual error rate / SLO error rate

Burn rate 1x: Consuming exactly at budget
Burn rate 14x: Will exhaust monthly budget in 2 hours

Alert tiers:
  Burn rate > 14x for 1 min  → PAGE (critical)
  Burn rate > 6x  for 5 min  → Page (major)
  Burn rate > 3x  for 30 min → Ticket
  Burn rate > 1x  for 6 hours → Weekly review

Error Budget Policy

Budget > 50% remaining:
  Ship features, take risks

Budget 0-50% remaining:
  Increase scrutiny, slow deployments

Budget exhausted:
  Freeze feature work
  All hands on reliability improvements
  No deploys until budget partially restored

SLO Dashboard

For each service:
  30-day SLO compliance: 99.94% (target 99.9%) ✓
  Error budget remaining: 82%
  Budget burn rate: 0.6x (healthy)
  Recent incidents: 2 (total 12.5 min)

Conclusion

SLOs make reliability decisions objective rather than political. Error budget policies align engineering incentives: good reliability = freedom to ship; poor reliability = mandatory reliability work.