OpenTelemetry: a practical guide to modern observability 🔭

Observability is the answer to the question: “can I understand what’s happening in production without attaching a debugger to it live and without guessing?” If the answer is “sort of,” then you need this article.

Observability vs monitoring 🧭

Often used as synonyms, they’re not. A brutally honest summary:

Monitoring tells you that you’re losing blood. Observability helps you discover where and why.

How deep observability needs to be depends on operational SLOs, often derived from contractual SLAs: without clear SLOs you risk redundant telemetry and excessive costs; with well-defined SLOs you measure only what’s needed to meet your commitments.

What observability is for 🎯

Observability isn’t an aesthetic whim for having dashboards full of colors: it’s the operational lever that makes the difference between reacting in panic or governing a complex system with clarity. A truly observable system lets you ask new questions without having anticipated every single counter beforehand.

In practice it allows you to:

• Reduce MTTR (Mean Time To Recovery) because you spend less time formulating wrong hypotheses and more time acting on the precise point of degradation.
• Prevent regressions: when you enable a feature flag or do a canary release, you can observe the real effect on latency, errors, and conversions instead of relying on gut feelings.
• Correlate user experience and infrastructure: understand that the increase in abandoned carts coincides with the pricing service’s latency in eu-west-1 after a slow autoscaling event.
• Identify bottlenecks and leaks before they become nighttime incidents: a queue that grows slowly, a pool that doesn’t release resources, a query that worsens weekly.
• Support capacity planning with trend and historical saturation data instead of improvised extrapolations.
• Facilitate causal analysis: move from “let’s try restarting” to “this span slows down after the DB driver upgrade.”
• Align signals to SLOs and error budget (driven by SLAs): depth, retention, and cardinality are sized starting from external and internal reliability objectives.

Result: less noise, fewer opinions, faster diagnoses, and informed decisions.

What is OpenTelemetry 🧪

OpenTelemetry (OTel) is an open standard (CNCF) that provides specifications, SDKs, and components for generating, collecting, and transporting observability signals (today: traces, metrics, logs; tomorrow: profiles, and more). It’s not a database, it’s not a dashboard, it’s not “yet another magic agent.” It’s the lingua franca that avoids lock-in on backend tools.

Its strength is the layered architecture: the APIs define neutral interfaces so application code doesn’t depend on a specific vendor. The SDKs implement buffering, sampling, export, and performance for individual languages (Go, Java, Python, JS, .NET, Rust…). Automatic instrumentation gives you immediate visibility (HTTP, gRPC, DB, frameworks), while manual instrumentation adds business context where it really matters (for example an “apply-discount-rule” span). The Collector acts as a hub: it receives (receivers), transforms (processors), and forwards (exporters) data to multiple backends simultaneously, allowing you to migrate or compare solutions without refactoring services. Semantic conventions provide consistency: using http.method everywhere avoids the chaos of creative variants.

OTLP 📡

OTLP (OpenTelemetry Protocol) is the native protocol used by OpenTelemetry to transport observability signals (traces, metrics, logs). It’s optimized for efficiency and interoperability, supporting transport via gRPC (binary, streaming) and HTTP/JSON (simpler to integrate). In practice, your apps and agents/SDKs send data in OTLP to the Collector, which processes it and forwards it to the desired backends.

Key operational points:

• Transport: otlp/grpc for performance; otlp/http for environments with restrictive proxies/firewalls.
• Security: optional TLS mTLS; watch out for certificates and SNI when passing through an LB.
• Compat: OTLP receivers/exporters are available in almost every component of the ecosystem.
• Migrations: using OTLP reduces lock-in because it separates instrumentation from the backend.

The three pillars: metrics, logs, and tracing 🏛️

OpenTelemetry calls them “signals.” Each answers different types of questions.

Metrics 📊

Metrics are the macro view of the system: compact numbers that are fast to query to understand if something is diverging from normal behavior. They don’t tell fine-grained details (that’s the domain of tracing) but they excel at showing you trends, saturations, and seasonality.

The main types cover distinct patterns: Counters only grow (total requests, bytes sent) and facilitate rates & derivatives; UpDownCounters measure quantities that go in and out (active connections, current goroutines); Histograms capture distributions (latencies, payload sizes) so you can reason in percentiles and not in lying averages; Gauges describe point-in-time state (free disk space, queue length).

To use them well: use histograms instead of averages when measuring latency or duration (the average hides the long tails); watch the cardinality of labels – every combination generates a series, and adding user_id is the fastest way to burn memory and performance; version metrics when you change semantics or naming, keeping an overlap period for legacy dashboards/alerts.

Logs 🪵

Logs are the granular narrative: discrete events with context that explain the relevant steps. They don’t replace metrics or traces: they complement them. If you use them to compute trends with grep you’re creating an inefficient metrics system; if you use them to explain why a decision branch was taken, then you’re winning.

For more, see the detailed article: Logging best practices.

From an OTel perspective: Prefer structured logs (JSON with consistent fields) so they become queryable; include the trace_id and span_id when you’re in the context of a request to jump between logs and traces; send them to the Collector, which redistributes them to specialized systems (Loki for low costs, Elasticsearch/OpenSearch for advanced full-text, Splunk for enterprise analysis). Less copy & paste of useless stack traces, more stable and searchable fields.

Tracing 🔍

Tracing answers: “what exactly happened during this request?” and “where are we spending time?” Essential in distributed systems.

Basic concepts:

• Trace: represents the entire journey of a single end-to-end operation (e.g. user request → gateway → service A → DB → service B → cache…). It’s a DAG of spans correlated by identifiers.
• Span: atomic unit of work (e.g. HTTP call, SQL query, batch processing). It has: name, start_time, end_time, key/value attributes, events (lightweight logs), links, and status.
• SpanContext: contains trace_id, span_id, flags, baggage.
• Baggage: key/value pairs propagated to enrich context (watch out for privacy and size).

Purpose of tracing:

Practical types:

• Distributed tracing: full cross-service path.
• Local in-process spans: internal granularity (function boundaries, pipeline steps).
• Synthetic / test traces: generated by scheduled jobs for critical paths.
• Cold start / init tracing: bootstrap measurement of serverless functions.

Simplified example (tree):

1
2
3
4
5
6
7

trace 4bf92f3577b34da6a3ce929d0e0e4736
└─ span GET /checkout (350ms)
   ├─ span call inventory (80ms)
   ├─ span call pricing (120ms)
   │   └─ span query SELECT price... (40ms)
   └─ span call payment (140ms)
       └─ span POST /psp/authorize (110ms)

Correlation of signals 🧬

The magic isn’t having lots of data, but having correlatable data. An SLO burn rate alert takes you to the latency metric; from there you filter the P99 and jump to the slow traces; in the individual trace you identify the problematic span and open the contextual log thanks to the trace_id. This path reduces minutes (or hours) of searching to a natural sequence of clicks.

Typical examples

• Log with trace_id → narrative reconstruction of the execution.
• Red SLO → drill-down on anomalous traces (extreme percentiles) → root cause.
• Cluster of slow traces with a common attribute (db.system = postgres) → targeted optimization.
• Deploy annotation on the timeline → immediate correlation with error spikes.

Pipeline with OpenTelemetry Collector 🔄

The Collector is the neutral crossroads: it centralizes ingest, transformation, and forwarding of signals. The receivers talk to the world (OTLP, Prometheus scrape, Jaeger, Zipkin, StatsD); the processors apply policies (batching for efficiency, tail sampling to keep only the interesting traces, attribute enrichment, PII redaction); the exporters send to multiple destinations (Jaeger, Tempo, Loki, Prometheus remote write, SaaS platforms). The extensions add health checks, pprof, zPages.

The practical benefits: total decoupling between services and backend (changing destination is config, not code), reduced vendor lock-in (parallel multi-export), centralized dynamic sampling instead of replicated, single points of consistent redaction/enrichment. In short: fine-grained control without touching applications.

Common tools by signal 📦

Metrics 📊

For time-series metrics, Prometheus is the de facto standard: simple pull model + labels. When scale grows or retention lengthens, VictoriaMetrics or Mimir come into play for efficiency and compression. Graphite survives in legacy environments while InfluxDB remains strong in IoT contexts. On the managed side, Datadog and New Relic focus on out-of-the-box integrations; cloud providers (CloudWatch, Google Cloud Monitoring, Azure Monitor) offer native integration; Grafana Cloud provides a managed Prometheus/Mimir stack.

Logs 🪵

For structured logs, Loki leverages label-based indexing (cheap on high volumes). Elasticsearch/OpenSearch remain the choice for powerful full-text search, at the cost of management and tuning. Fluent Bit / Fluentd and Vector act as high-performance agents/forwarders. In SaaS, Splunk dominates the enterprise, Datadog Logs integrates with the rest of the platform, Elastic Cloud simplifies deploying the Elastic stack.

Tracing 🔍

Jaeger and Tempo are the most widespread open solutions: the first mature and battle-tested, the second very efficient on object storage. Zipkin is still relevant for simplicity. SigNoz and self-hosted Elastic APM offer an integrated approach. In SaaS: Honeycomb excels in ad-hoc exploratory queries, Lightstep (ServiceNow) in complex causal analysis, Datadog/New Relic/Dynatrace in platform coverage, AWS X-Ray for quick integration on the Amazon stack.

All-in / converging 🧩

Converging platforms aggregate signals (metrics+logs+traces) and often add profiling: Datadog, New Relic, Dynatrace. The Grafana stack (Prometheus + Loki + Tempo + Mimir) is the modular open option. Elastic Observability unifies ingest and correlation. Coralogix and Sumo Logic represent flexible SaaS alternatives.

Strategic purposes of observability 🧠

Behind the graphs there’s a strategic goal: compressing the idea→learning cycle. Mature observability accelerates feedback loops because the response to the effect of a change is immediate. It enables SLOs based on real user perception (end-to-end latency, success rates) and turns error budgets into an engineering governance tool.

From a governance perspective, the SLA → SLO → indicators chain guides the depth of observability: traces, logs, and metrics should be collected at the minimum level that allows meeting SLOs and managing the error budget. Every extra level (denser sampling, longer retention, high-cardinality labels) has a measurable cost: always evaluate the benefit/cost ratio before enabling it.

It directly reduces the cost of incidents (fewer man-hours, fewer interruptions, less lost context). It enables distributed, asynchronous, event-driven architectures without turning debugging into archaeology. It unlocks continuous verification: every canary or feature flag is monitored with thresholds and controlled regression. Finally, it strengthens ownership and accountability: teams see the impact of their decisions in real time.

Build vs buy: managing observability “in-house” 🏗️

Before choosing a stack or platform, start from the SLOs derived from SLAs: they determine which signals are really needed, at what granularity, and for how long. The choice between self-host, SaaS, or hybrid should be weighed against the total cost (ingest, storage, query, on-call) and the benefit relative to SLO objectives.

Self-host advantages ✅

Offers full control: you can define multi-level retention (hot 7 days, warm 30, cold 180), adaptive sampling rules, data confinement in a region for compliance. Costs become optimizable: you move old logs to cheap storage, apply custom compression, downsample historical metrics.

It reduces vendor lock-in (changing backend is a configuration change in the Collector) and allows deep customization of the pipeline (redaction, business-specific enrichment, conditional tail-sampling on attributes). It also facilitates integration with internal ecosystems (feature flags, domain events, billing systems).

Self-host disadvantages ❌

The flip side is operational complexity: scaling Prometheus (sharding, federation), Elastic/OpenSearch clusters hungry for tuning, retention and backup policies to maintain. It introduces a new domain “to manage” with its own on-call (if the system observing goes down, you fly blind).

It requires constant tuning (metric cardinality, slow queries, filling storage). Major upgrades require controlled plans. And above all, dedicated skills are needed: it’s not enough to know how to install a Helm chart, you need to understand data models, the impact of labels, hot path management.

SaaS advantages ☁️

The main benefit is speed of adoption: minimal instrumentation and you’re operational with preconfigured dashboards and alerts. Modern platforms bring advanced features (anomaly detection, automatic correlations between signals, eBPF for codeless tracing) that would be costly to replicate.

Elastic scalability is transparent and the operational burden is zero. In enterprise contexts, already guaranteed SLAs, certifications, and audit trails also matter. At the start, costs are predictable and justified by time-to-value.

SaaS disadvantages 🧾

Volume growth (verbose logs, high-cardinality traces, extended retention) can generate non-linear cost escalation. Adoption of proprietary features (query DSL, AI functions) increases psychological and technical lock-in.

Platforms often limit deep customizations: advanced conditional sampling, complex redaction in the pipeline, custom ingest-side enrichment. There’s also the issue of data sovereignty: exporting logs and traces outside the perimeter can be critical in regulated sectors if encryption/pseudonymization isn’t applied upstream.

Hybrid model 🧪

A pragmatic approach: keep high analytical-value signals in SaaS (SLO metrics, intelligently sampled tracing, error logs) and manage raw volumes in self-host (debug logs, non-critical traces, very high-cardinality metrics). This way you get quick insights where needed and cost control on bulk data. The Collector easily enables differentiated routing (multi-export) and inbound filtering policies.

Incremental adoption strategies 🚀

Prerequisite: clarify SLA/SLOs and translate them into operational indicators (latency, availability, error rate) from which the necessary metrics/traces/logs derive.

Suggested path:

1. Quick correlation: add trace_id to logs without changing the backend yet.
2. Canary instrumentation: instrument a low-risk service to validate overhead and flows.
3. Dual-export Collector: send to two backends for comparison and gradual migration.
4. Smart sampling: combine percentage head sampling with tail sampling on errors and slowness.
5. Standardization: apply semantic conventions before every team invents local conventions.
6. Operational SLOs: link the burn rate to decisions (feature freeze, hardening priority).

Common mistakes to avoid 🧨

Recurring mistakes:

• Observability without SLOs: instrumenting without objectives tied to SLAs/SLOs leads to high costs and unhelpful insights.
• “Shotgun” tracing: instrumenting everything at 100% without filtering generates noise and costs.
• High-cardinality labels: inserting unique identifiers turns Prometheus into a RAM furnace.
• Duplication in logs: replicating attributes already present in the span burdens ingest and storage.
• Infinite retention: rarely-consulted cold data drains budget; define expiration policies.
• Blind Collector: if you don’t monitor backlog, drop rate, and export errors, you’re flying instrumented… but blind.

Quick checklist ✅

• All services expose at least one main latency histogram.
• Trace_id propagated from edge to DB.
• Structured logs with trace correlation.
• Dynamic sampling for errors and slow p99s.
• SLOs defined and monitored (error + latency).
• SLAs mapped to measurable SLOs and technical indicators.
• Incident runbooks updated with dashboard/trace links.
• Cost-aware pipeline (filters, redaction, storage tiering).

Conclusion 🔮

OpenTelemetry doesn’t “solve” observability: it gives you a vocabulary and a toolbox. The value comes from smart questions, correlated data, and discipline in keeping useful signals (not just many). Start simple, measure, prune what’s not needed, and let the data tell the story of your system.

If you want to improve your logs, next step: Logging best practices. Then move on to advanced tracing with targeted sampling.

Happy observability (the real kind, not the buzzword). 😉