Distributed Tracing

Tracing

Difference from logs and metrics, traces can provide a very fine-grained level of detail for each individual request. Distributed Tracing has become very valuable in a microservice architecture to debug, identify bottlenecks and understand applications.

Setup

Distributed tracing is supported by Axual, but not delivered as part of the platform. For production installations, we expect a distributed tracing backend to be made available by the infrastructure team.

We have used OpenTelemetry that is the mechanism by which application code is instrumented, to help make a system observable.

For all our components that support OpenTelemetry, tracing can be enabled as below:

global:
  tracing:
    enabled: true
    otelExporter:
      endpoint: "https://endpoint_of_tracing_backend"
      headers:
        authorization: "Basic ..."

Example: Honeycomb

To set up Honeycomb to export the traces and visualize them, you can follow the below configuration:

global:
  tracing:
    enabled: true
    otelExporter:
      endpoint: https://api.honeycomb.io
      headers:
        x-honeycomb-team: teamUID
        x-honeycomb-dataset: dataset

Depending on the instrumentation backend, you may want to add custom headers. Please refer to thr desired link to get more information about required headers: