Logging: essential best practices 🪵

Writing logs isn’t an act of faith: it’s an investment in reliability. Well-designed logs reduce diagnosis times, avoid sleepless nights, and, surprise, also save money. Here’s a set of practices you can apply right away, without religion or magic.

Use a standard logging library 📦

Avoid homemade solutions: mature libraries handle concurrency, formats, levels, sinks, and performance better.

• In production, prefer structured output (e.g. JSON) for ingestion into log systems.
• Reliable examples: log/slog (Go), pino/winston (TypeScript/Node.js).
• Configure formatter, levels, and destinations through configuration, not code.

Why it matters: a standard library solves boring problems (buffering, backpressure, formats, performance) so you can focus on content. It also guarantees consistency across services and environments.

Write meaningful messages 🧠

The log must explain the “why,” not just the “what.”

• Describe the action and the outcome: “payment authorized,” “login attempt failed.”
• Indicate the entity involved: resource, operation, quantity, duration.
• Avoid noise: no logs that repeat obvious things or spam in a loop.

Better to avoid cryptic phrases or digital onomatopoeia. A good message answers: what was I doing, on what object, with what result, in how much time. If needed, add the “why” (parameters and conditions).

Examples:

• Bad: Processing completed (…of what? with what outcome?)
• Good: order processed with fields {order_id, amount, duration_ms, outcome: 'authorized'}

Add context to logs 🧭

Context turns a line into a diagnosis. Add consistent and stable fields:

• request_id / trace_id, user_id/session_id, service/component, version/build, region/zone.
• Keep field names consistent across services to facilitate queries.
• Propagate tracing identifiers along the call chain.

Tip: use child/scoped loggers so you don’t have to repeat the same fields on every line. In TypeScript: const log = logger.child({ trace_id, user_id }).

Avoid sensitive information 🔒

Privacy isn’t optional. Don’t log PII, secrets, tokens, or unnecessary payloads.

• Selectively mask: emails, IBANs, card numbers, addresses, tokens.
• Implement an application-side redactor and, if possible, a downstream filter in the ingestion pipeline.
• Verify retention based on compliance (GDPR, PCI-DSS, etc.).

Apply the principle of data minimization: log only what’s useful for diagnosis. Encrypt logs in transit and at rest in the central system. Periodically perform a review of fields to spot inappropriate data.

Choose the right log levels 🎚️

Consistent levels = lighter on-call pages.

• DEBUG: development details. Disabled in prod (or only at runtime for limited periods).
• INFO: relevant business/operational events.
• WARN: recoverable anomalies or non-blocking degradations.
• ERROR: failures of the current operation, action required.
• (Opt.) FATAL: terminates the process; use sparingly.

Practical rule for HTTP:

• 2xx/3xx → INFO
• Expected 4xx (e.g. 401/403) → WARN or contextualized INFO
• 5xx → ERROR (investigation necessary)

Allow overrides per module/component and dynamic level variation via env/feature-flag.

Configure log rotation and retention 🔁

Files grow. Always. Set clear limits.

• Rotation by size and/or time; keep N compressed files.
• Send to STDOUT in containers, delegate retention to the log stack.
• Avoid persistent local logs in ephemeral environments (k8s, serverless).

In non-containerized environments, use system tools (logrotate) with compression and a defined retention. In clusters, prefer a shipper (e.g. Fluent Bit) with a limited-capacity disk buffer.

Synchronize server clocks ⏱️

Inconsistent timestamps = impossible timelines.

• Enable NTP/Chrony and monitor skew and drift.
• Log in UTC and show timezone only in UI/analysis.

For reliable measurements, also record durations (ms) next to timestamps. For internal calculations, use monotonic timers to avoid clock jumps.

Log errors with stack traces 🧩

Without a stack, debugging is archaeology.

• Include stack traces for ERROR/FATAL; for WARN only if useful.
• Enrich with root cause (error wrapping) and metadata (key inputs, retries, latency).

Avoid truncating stacks: fragments out of context are of little use. If the stack is very verbose, send it whole but keep the field separate (e.g. error.stack) for faster searches.

Avoid over-logging 🧹

Too many logs hide the real problems and cost money.

• Avoid logging inside tight loops; deduplicate and rate limit.
• Don’t duplicate the same event at multiple levels of the call stack.
• Make the level configurable at runtime and document the policies.

When traffic increases, apply sampling (e.g. 1% of DEBUG, 10% of INFO) and burst control. For repetitive errors, use deduplication keys (e.g. error_fingerprint).

Centralize logs 🗂️

Collect everything in a log lake for correlations and searches.

• Typical stacks: ELK/OpenSearch, Loki, the provider’s Cloud Logging.
• Standardize schema and fields for cross-service queries.
• Link logs, metrics, and tracing for complete observability.

Define indexes/streams per service and lifecycle policies (ILM) to control space and retention times. Reduce the cardinality of indexed fields: it’s the leading cause of runaway costs.

Analyze and monitor logs 📈

Logs are useless if nobody looks at them.

• Extract derived metrics (error rate, p95 latency, timeouts).
• Create actionable alerts on patterns and thresholds (not on isolated single events).
• Use correlation rules to reduce noise during incidents.

Keep saved queries and dashboards for recurring cases (DB timeouts, external rate limits, full queues). Link alerts to clear, short runbooks.

Test the log configuration 🧪

Trust is good, testing is better.

• Integration tests: verify format, levels, mandatory fields, and redaction.
• Chaos logging: simulate high volumes, error bursts, backend loss.
• Validate the end-to-end pipeline (app → shipper → index → dashboard → alert).

Add snapshot tests for the JSON format and a schema drift check. In staging, validate that the logs of a simulated incident generate the same expected alerts.

Define a log schema and versioning 🧬

Establish a contract: keys, types, semantics. Introduce log_schema_version and manage its evolution in a backward-compatible way. Document the mandatory and optional fields.

Benefits: simpler queries, fewer surprises, and the ability to validate automatically on ingestion.

Example of a minimal but useful JSON event:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

{
  "ts": "2025-09-09T08:00:00Z",
  "level": "info",
  "msg": "order processed",
  "service": "checkout",
  "version": "1.8.3",
  "trace_id": "3f0b6b0e2a1c9b7d",
  "span_id": "e7b2d1a3",
  "request_id": "req_01HZW...",
  "user_id": "u_12345",
  "order_id": "ord_98765",
  "amount": 129.90,
  "currency": "EUR",
  "duration_ms": 84,
  "outcome": "authorized",
  "log_schema_version": 2
}

Link logs, metrics, and distributed tracing 🔗

Use OpenTelemetry and propagate trace_id/span_id (W3C traceparent). Logs should allow you to jump to metrics and traces with a click. Investigations go from “what” to “where/when” in an instant.

• Adopt OTel Semantic Conventions for common fields (http.method, url.path, db.system, net.host.name, peer.service).
• Link logs → span: attach trace_id/span_id and, when useful, span_link to significant events (retries, timeouts, errors).
• Link metrics → trace with exemplars (e.g. p95 latency with pointers to the slowest traces).
• Fallback: if propagation headers are missing, generate and log a local trace_id to maintain at least intra-service correlation.

Manage costs and cardinality 💸

Set a cardinality budget for free-form keys (e.g. user_id). Avoid indexing very high-variability fields. Apply adaptive sampling and differentiated retention: short for raw logs, long for derived metrics.

• Indicate which fields are indexed and at what precision; limit indexes to a few fields actually used in operational queries.
• Reduce cardinality: normalize values (e.g. group similar messages via template + error_fingerprint).
• Storage tiering: hot (7–14 days), warm (30–90 days), cold/archive (S3/GCS with compression); keep only derived metrics long term.
• Set query quotas and timeboxes (e.g. 7 days) to avoid accidentally expensive searches.
• Use rollup/downsampling for historical analysis (e.g. hourly/daily aggregates) instead of raw logs.

Design resilient log shipping 🚚

Decide the strategy in case the backend is unavailable:

• Fail-open: the app continues and discards excess logs
• Fail-closed: the app degrades/stops (only for regulated cases)

Use local buffers with limits, exponential backoff, and telemetry on the shipper’s status.

• Drop policy by severity: don’t lose ERROR/FATAL if possible, apply aggressive sampling to DEBUG.
• Backoff with jitter and circuit breaker when the backend is degraded; gradually re-enable on recovery.
• Clear limits: max_batch_size, max_queue_bytes, max_retry_interval; highlight overruns with metrics/alerts.
• Disk spooling with a cap and expiration (TTL) to avoid running out of disk; send counts of dropped_events_total.
• Dead letter for events that don’t conform to the schema (inspectable separately) to avoid blocking the flow.
• Minimal telemetry: queue_depth, flush_latency, retries_total, success_rate, backend_status.

Logging in containers and Kubernetes 🧩

• Write to STDOUT/STDERR; leave collection to the container runtime + log shipper (e.g. Fluent Bit/Vector).
• In k8s, avoid unnecessary sidecars: prefer a shipper DaemonSet with structured parsing.
• Add useful labels/annotations: app, version, pod, node, namespace, cluster.
• Manage runtime file rotation (containerd/docker): set limits for size and count.
• Don’t persist logs in the image; no logs in /tmp without limits.

Further practical notes:

• Prefer one log per line in JSON to avoid multiline issues; if you have multiline stacks, configure the shipper’s parser.
• With containerd/docker evaluate json-file vs journald drivers; standardize at the cluster level.
• Avoid colors/ANSI in logs: they clutter JSON output and complicate parsing.
• Allocate resources to the shipper (CPU/memory) and limit collection of noisy namespaces with filters/regex.
• Use ephemeral containers for debugging, don’t modify pods’ logging runtime in production.

Quick implementation examples 🧪

Go (slog, JSON):

1
2
3

logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo}))
log := logger.With("service", "checkout", "version", "1.8.3", "trace_id", traceID)
log.Info("order processed", "order_id", ordID, "amount", amount, "duration_ms", dur.Milliseconds(), "outcome", "authorized")

Node.js (pino):

1
2
3
4

const pino = require('pino')
const logger = pino({ level: process.env.LOG_LEVEL || 'info' })
const log = logger.child({ service: 'checkout', version: '1.8.3', trace_id })
log.info({ order_id: ordId, amount, duration_ms: 84, outcome: 'authorized' }, 'order processed')

Python (structlog):

1
2
3
4

import structlog, sys
structlog.configure(processors=[structlog.processors.JSONRenderer()])
log = structlog.get_logger(service="checkout", version="1.8.3", trace_id=trace_id)
log.info("order processed", order_id=ord_id, amount=amount, duration_ms=84, outcome="authorized")

Performance and overhead ⚙️

• Avoid expensive interpolations when the level is disabled; use lazily evaluated logging.
• Batch and buffering: enable asynchronous flushes and safe backpressure.
• Reduce the payload: use short but clear keys (duration_ms, not theTimeItTookInMilliseconds).
• Avoid synchronous I/O on hot-path routes; move non-critical logs to asynchronous queues.

Other tips:

• Avoid JSON serialization of large structures; log hashes or sizes and keep details elsewhere.
• Reuse logger and buffer instances to reduce allocations (pooling).
• Consider the CPU/space compression trade-off in the shipping pipeline.
• Standardize timestamps (RFC3339 UTC) and consider epoch to reduce parsing costs on the backend.

Governance, linters, and conventions 📏

Standardize naming (e.g. snake_case or lowerCamelCase), avoid synonyms (user_id vs uid). Add a log linter in the pipelines that rejects non-conforming formats or forbidden keys.

• Maintain a small schema registry with versions and changelog; require approval for breaking changes.
• Run CI validations (samples of logs generated by tests) against the schema.
• Define forbidden keys (PII, secrets) and a field deprecation process.
• Pre-commit: hooks that prevent introducing unstructured console.log or sporadic prints.

Security events and immutable audit 🔐

Separate security/audit logs (append-only, timestamped, signed/immutable). Define access roles and retention compliant with policies.

• Consider WORM storage (e.g. S3 Object Lock) and RFC3161 timestamps for non-repudiation.
• Use hash chains/signing of log blocks to highlight tampering.
• Segregate KMS keys and access audit trails; alert on deletions or time gaps.

Language and localization of logs 🌐

Prefer messages in English or neutral terminology to facilitate search and sharing. Localization should be reserved for UIs; logs serve machines and global on-call teams.

• Separate the human text from the error code (error_code, event_id); queries use the codes, not the phrases.
• Keep message templates stable; avoid language variations that complicate searches.

Runbooks, queries, and ready-to-use dashboards 🧰

Every alert must have:

• A saved diagnostic query
• A linked dashboard
• A runbook with reproducible steps and expected times

Practical tips:

• Typical queries: error rate ERROR/5m, timeouts per dependency, p95 latency spike per endpoint.
• Minimal dashboards: error overview per service, latency per route, correlation with releases (deploy overlay).
• Link alerts to the runbook and to ready-made queries with filters (service, version, trace_id).

Quick checklist 📝

• Standard library with structured output
• Clear and useful messages, with consistent context
• Active redaction for sensitive data
• Levels configured and documented
• Rotation and retention set up
• Time synchronization enabled (UTC)
• Stack traces on significant errors
• Noise under control (dedup/rate limit)
• Centralization and correlation with metrics/tracing
• Alerts and end-to-end pipeline tests
• Log schema/versioning documented
• Cardinality budget and cost governance
• Fail-open/fail-closed strategy for shipping
• Linter and naming conventions applied
• Runbooks, queries, and dashboards linked