How to Enable Distributed Tracing

This guide shows you how to enable distributed tracing on the Axual components and export the spans to an OpenTelemetry backend.

Type

How-to guide

Goal

Get traces from the Axual components into a tracing backend.

Audience

Platform Operator who can edit the component values and apply them.

When to use

Use this guide when diagnosing latency across components, or as part of setting up observability.

Unlike logs and metrics, traces record fine-grained detail for each individual request. That is what makes them valuable in a microservice architecture for debugging, identifying bottlenecks and understanding how a request moved through the platform. For where tracing sits alongside logs and metrics, see Monitoring.

Contents

The sections below cover each task in this guide:

Prerequisites

Confirm the following before you begin.

Access and permissions required

You need the following access and permissions:

  • Permission to edit the values of the component you want traces from, and to apply them.

  • Permission to create or read the Kubernetes Secret that holds the backend credentials.

Tools and versions required

You need the following tools:

  • helm >= 3.12, or access to the pipeline that applies the values.

  • kubectl >= 1.28, with access to the namespace the component runs in.

Resources that must exist before starting

The following must already exist:

  • A running Axual installation containing at least one of the components listed in Enable tracing on a component.

  • A tracing backend that accepts OpenTelemetry Protocol (OTLP) spans, together with its endpoint and any credentials it requires. Axual does not deliver one.

Enable tracing on a component

Axual supports distributed tracing but does not deliver it as part of the platform. On a production installation, the infrastructure team provides the tracing backend.

The components instrument themselves with OpenTelemetry, which is what produces the spans this guide exports.

The following components carry the OpenTelemetry configuration this guide uses, and each one’s readme documents its own tracing keys:

The Runtime Provisioner also exports traces with OpenTelemetry, but it takes its settings from environment variables rather than the keys below. See Enable distributed tracing.

Tracing is off per component until you set it, so add the following to that component’s values:

config:
  management:
    tracing:
      enabled: true

config is the component’s own configuration key, so the fragment nests under wherever the chart places that component. In the Axual Governance chart, Platform Manager sits at axual-governance.platform-manager, which makes the full path look like this:

axual-governance:
  platform-manager:
    config:
      management:
        tracing:
          enabled: true

Apply the values the same way you applied the rest of that component’s configuration, then move on to exporting the spans.

Exporting traces

Enabling tracing only produces the spans; a backend has to receive them. For example, to set up Honeycomb to export the traces and visualise them, add the endpoint and the API key header under the same config key:

config:
  management:
    otlp:
      tracing:
        endpoint: https://api.honeycomb.io
        headers:
          x-honeycomb-team: <HONEYCOMB_API_KEY>
Replace every <VALUE> placeholder with your own value before applying the configuration.
Provide the endpoint and the headers from a Kubernetes Secret rather than writing them into the values file, because the header carries a credential.

Each backend requires its own headers. Refer to the documentation of the one you use:

Adjusting sampling

The config.management.tracing.sampling.probability value controls the fraction of spans the component collects. It defaults to 1.0, which exports every span. Lower it to cut the volume the backend receives:

config:
  management:
    tracing:
      sampling:
        probability: 0.1  # 10% sampling

Verify traces arrive

A component that restarts cleanly is not proof the spans are leaving it, because an unreachable collector and a rejected credential both fail silently. Confirm the backend receives spans before relying on the configuration.

  1. Apply the values and wait for the component’s pods to restart.

  2. Send a request that passes through the component, for example a Self-Service call routed through the API Gateway.

  3. Search your tracing backend for spans from that request.

Expected result: the backend shows one trace for the request, carrying a span for each traced component the request passed through. An empty result while the pod is healthy points at the export settings in Exporting traces, or at a sampling probability low enough that the request was not sampled.