Distributed Tracing

Tracing

Difference from logs and metrics, traces can provide a very fine-grained level of detail for each individual request. Distributed Tracing has become very valuable in a microservice architecture to debug, identify bottlenecks and understand applications.

Setup

Distributed tracing is supported by Axual, but not delivered as part of the platform. For production installations, we expect a distributed tracing backend to be made available by the infrastructure team.

We have used OpenTelemetry that is the mechanism by which application code is instrumented, to help make a system observable.

For all our components that support OpenTelemetry, tracing can be enabled as below:

config:
  management:
    tracing:
      enabled: true

Exporting traces

For example, to set up Honeycomb to export the traces and visualize them, you can follow the below configuration:

config:
  management:
    otlp:
      tracing:
        endpoint: https://api.honeycomb.io
        headers:
          x-honeycomb-team: teamUID

Endpoint and headers information should be provided using secrets

Depending on the instrumentation backend, you may want to add custom headers. Please refer to thr desired link to get more information about required headers:

Adjusting Sampling

The trace-sampling-probability property controls the fraction of spans that are collected. Setting it to 1.0 means all spans will be exported.

To adjust the sampling rate (default is 100%):

config:
  management:
    tracing:
      sampling:
        probability: 0.1  # 10% sampling