Observability: Logs, Metrics, and Traces
At 3am you need three answers: what happened, how bad is it, and where time went. Logs record discrete events. Metrics aggregate rate, errors, and latency. Traces follow one request across services. Together they are observability — monitoring tells you something is wrong; these pillars help you explain why. A microservices architecture blueprint shows why correlation becomes essential when one request crosses many owners.
This article connects the three with one checkout request, shows OpenTelemetry-shaped instrumentation, and covers the failure path when a dependency times out. Use those signals to verify graceful degradation patterns instead of assuming a fallback worked.
Logs
Logs are timestamped events: errors, warnings, business facts. A useful line includes time, severity, service, trace id, and structured fields (order_id, path, error code). Unstructured strings are hard to query at scale; JSON fields are filterable.
Centralize with your platform’s store (Loki, Elasticsearch, CloudWatch, etc.). Never log secrets. When not to lean on logs alone: you need trends or SLOs — that is metrics work — or cross-service latency — that is tracing.
Quick reference
- Best for: incident reconstruction, audit trails, unexpected edge cases.
- Strengths: rich context; searchable when structured.
- Weaknesses: volume cost; poor for aggregates without heavy pipelines.
- Put trace_id / span_id on every line that participates in a request.
- Retention: hot window for ops, colder archive for compliance.
- ERROR = needs action; WARN = recoverable; INFO = business events; DEBUG = off in prod by default.
Remember this
Logs answer “what happened?” — structure them and always include a trace id.
Metrics
Metrics are numeric series: counters (requests), gauges (in-flight), histograms (latency). They are cheap to store and query compared to raw logs. Define SLIs (availability, latency, error rate) and SLOs as targets you can alert on — example: “99.9% of checkout requests succeed under a latency budget you measured,” not a copied number from a blog.
RED (Rate, Errors, Duration) fits most HTTP services; USE (Utilization, Saturation, Errors) fits hosts and queues. Alert on user-visible symptoms (error rate, SLO burn), not every CPU blip.
Quick reference
- Best for: dashboards, alerting, capacity planning, SLO tracking.
- Strengths: cheap aggregates; trend detection.
- Weaknesses: no per-request story; needs intentional instrumentation.
- RED for services; USE for infrastructure.
- Cardinality: careful with high-cardinality labels (raw user ids) on metrics.
- When not to: debugging one user’s odd payload — use logs + traces.
Remember this
Metrics answer “how much / how fast?” — instrument RED and alert on SLO burn.
Distributed Traces
A trace is one request’s timeline across services as nested spans. A shared trace_id links them; parent/child ids show nesting. When checkout feels slow, the trace shows whether time sat in Order, Payment, or an external API.
Request trace: Browser → API gateway (span) → Order service (span) → Postgres (child span) → publish order.placed → Payment service (span) → card network (span). On timeout, Payment records an error span and Order logs with the same trace_id so you jump from the alert to the exact slow child.
Sample intelligently at high volume: always keep error traces; sample a fraction of successes. Propagate context via traceparent (HTTP) and broker headers for async work.
Quick reference
- Best for: latency debugging, dependency maps, bottleneck finding.
- Strengths: end-to-end timing; exact slow span.
- Weaknesses: storage cost; needs sampling and consistent propagation.
- OpenTelemetry is the common portable instrumentation API.
- Always retain error traces; sample successful traffic.
- When not to: a single-process app with clear logs — add traces when hops multiply.
Remember this
Traces answer “where did the time go?” — propagate context on every hop, including async.
OpenTelemetry in Production
OpenTelemetry provides SDKs, auto-instrumentation for common libraries, and a Collector that receives, processes, and exports to backends (Prometheus, Jaeger/Tempo, Loki, vendor APM). Typical path: app → OTel SDK → Collector → backends → Grafana (or equivalent).
Start with auto-instrumentation for HTTP, DB, and queues. Add custom spans around business-critical steps (authorize payment, reserve inventory). Correlate pillars: logs carry trace_id; metric spikes link to exemplar traces.
Quick reference
- Pipeline: SDK → Collector → metrics/traces/logs backends.
- Auto-instrument HTTP, gRPC, DB, Redis, Kafka where supported.
- Custom spans for domain steps metrics alone cannot explain.
- Run the Collector as sidecar/daemon — not a heavy embed in every process if you can avoid it.
- Dashboard per service: RED + link to traces + recent error logs.
Remember this
Standardize on OpenTelemetry — auto-instrument first, custom-span the business path, unify in one UI.
Key takeaway
Logs explain events, metrics explain trends, traces explain latency across hops. Instrument RED early, add structured logs with trace ids, and turn on distributed tracing when more than one service owns a user request.
Practice (25 min): Pick one checkout (or login) path. Emit one structured log line with trace_id, one counter/histogram for that route, and one custom span around the slowest dependency. Force a timeout in a test; confirm you can jump from the error metric to the failing span to the log line with the same id.
Related Articles
Explore this topic