Observability with OpenTelemetry on GCP

OpenTelemetry defines three signal types with a shared instrumentation model: traces are a tree of spans representing one request's path through your services, each span carrying a start/end time, attributes, and a parent-child relationship that lets a trace viewer reconstruct the whole call graph. Metrics are aggregated numeric measurements over time (request count, latency histogram, error rate) — cheaper to store and query at scale than traces, but they answer "how much/how often," not "what exactly happened in this one request." Logs are discrete timestamped events, and OTel's log API (newer and less universally adopted than tracing) lets a log line carry the same trace/span id as the request it happened during, which is what makes "click a slow trace, jump straight to its logs" possible instead of grepping by timestamp and hoping.

The practical discipline: instrument all three from the start using the same OTel SDK, with correlated ids (trace id, span id) attached to metrics and log lines wherever the emitting code has an active span — that correlation is what turns three separate observability surfaces into one investigation instead of three tools you cross-reference by hand. For the conceptual foundation this signal model rests on, see observability: logs, metrics, and traces.

Application code exports OTel signals over OTLP (the OpenTelemetry wire protocol) to a Collector — a separate process (not the application) that receives, optionally batches/filters/samples, and forwards to one or more backends. On GCP, the Collector's googlecloud exporter translates OTel traces into Cloud Trace's format and OTel metrics into Cloud Monitoring's format, so application code never needs a GCP-specific SDK — it speaks standard OTLP to the Collector, and only the Collector's config knows it's talking to Google Cloud.

This indirection is the actual value of the Collector layer: swapping backends (adding a second export target, or migrating off GCP later) is a Collector config change, not an application redeploy across every service. It also centralizes sampling and PII-scrubbing logic in one place instead of duplicating that logic in every service's instrumentation code.

Two deployment shapes for the Collector on GKE, and they trade off differently. A sidecar Collector runs in every pod alongside the application container — lowest latency to the exporting app (localhost), and a Collector crash only affects that one pod's telemetry, but it multiplies Collector resource usage by pod count and duplicates sampling/config across every replica. A gateway Collector runs as its own Deployment (often behind a Service, or as a DaemonSet — one per node rather than per pod) that every application pod exports to over the cluster network — centralized config and far fewer Collector instances to run, at the cost of one more network hop and a shared failure domain if the gateway Collector itself falls behind.

For most teams, a DaemonSet Collector (one per node, shared by every pod on that node) is the practical middle: fewer instances than sidecars, but still local to the node rather than a fully centralized bottleneck. Start there; move to a fully centralized gateway only if you need one place to enforce cluster-wide sampling or scrubbing policy that a per-node DaemonSet can't coordinate.

Cloud Trace bills per span ingested past a free monthly quota, and Cloud Monitoring bills per time series and per sample for custom metrics — which means an observability decision ("sample every trace at 100%") is silently also a cost decision, and teams frequently discover this only when a traffic spike produces a billing spike with no corresponding incident. The realistic failure: a service under normal load exports acceptable trace volume, then a retry storm or a traffic spike multiplies request volume 20x, and 100% trace sampling multiplies Cloud Trace ingestion 20x in the same hour, without anyone having made an explicit decision to spend that much.

The fix is tail-based sampling with an explicit policy, not a fixed percentage applied blindly: always keep error traces and traces above a latency threshold (the ones you'll actually want to look at during an incident), and sample ordinary fast successful traces down to a small percentage — 100% of interesting traces, 5-10% of routine ones. Set a Cloud Billing budget alert on the Trace and Monitoring SKUs specifically, so a sampling misconfiguration surfaces as a billing alert within a day, not as a surprise on the next invoice.