Observability: Logs, Metrics & Traces

Observability is the ability to understand what a system is doing from its external outputs. The three pillars are logs (what happened), metrics (how much and how fast), and distributed traces (where time was spent across services). A system with all three lets you move from alert to root cause without guessing.

Logs should be structured JSON — machine-readable and filterable. Metrics expose counters, gauges, and histograms — request rate, error rate, and latency percentiles (p50, p95, p99) are the minimum set. Distributed tracing assigns a trace ID to every request that propagates across service boundaries — each service records a span, and the trace view shows the full call tree with timing. OpenTelemetry is the vendor-neutral standard for all three; emit once, ship to any backend.

Before

Unstructured logs — impossible to query

1// Cannot filter, aggregate, or alert on this2console.log("User 123 placed order 456 for $99.00");3console.log("Error processing payment for order 456");4 5// Try answering: "orders over $50 that failed last hour"6// → impossible without parsing free-form strings

After

Structured logs + metrics sketch (pseudo-code counters)

1// Structured log — filterable by any field2logger.info({3  event: 'order.placed',4  userId: '123',5  orderId: '456',6  amount: 99.00,7  traceId: span.traceId,   // links log to trace8});9 10// Metric — alertable, graphable11orderCounter.inc({ status: 'success' });12orderLatency.observe(Date.now() - startTime);

Exercise

Convert one console.log string into a structured JSON log with event name, ids, and a traceId field. List three metrics you would alert on (e.g. error rate, p99 latency, queue depth).