Distributed Tracing and Observability

In microservices, a single user request spans multiple services. Distributed tracing tracks this journey end-to-end.

The Three Pillars of Observability

• Logs: What happened (events)
• Metrics: How much/often (counters, gauges)
• Traces: Where time was spent (spans)

Distributed Tracing Concepts

Trace

End-to-end journey of a single request.

Span

A single operation within a trace (e.g., DB query, API call).

Context Propagation

Trace ID passed via headers through all services.

Client → Service A → Service B → Database
  TraceID: abc123 propagated through all

OpenTelemetry

Vendor-neutral standard for instrumentation.

const tracer = opentelemetry.trace.getTracer('order-service');
const span = tracer.startSpan('processOrder');
try {
  // do work
} finally {
  span.end();
}

Stack

Layer	Tool
Instrumentation	OpenTelemetry
Collection	Jaeger / Zipkin
Storage	Elasticsearch / Cassandra
Visualization	Jaeger UI / Grafana

Key Metrics to Track

• RED: Rate, Errors, Duration
• USE: Utilization, Saturation, Errors
• P99 latency across service boundaries

Alerting

Alert: P99 latency > 500ms for 5 min
Alert: Error rate > 1%
Alert: Service down (no heartbeat)

Conclusion

Observability is non-negotiable in microservices. OpenTelemetry + Jaeger + Grafana is the modern open-source stack.