How to Ship Logs to a Central Stack

This guide shows you how to make an Axual component emit JSON instead of human-readable log lines, which components support it, and what a central logging stack needs from the cluster to collect those lines.

Type

How-to guide

Goal

Get component logs into a central logging stack in a format that stack can index.

Audience

Platform Operator who can edit a component’s values.yaml, working with whoever runs the cluster’s logging stack.

When to use

Use this guide when setting up central log collection, or when logs are being collected but arrive as unparsed text.

Axual Platform does not ship a central logging stack. Collection is the cluster’s job, and the platform’s part is emitting logs a collector can parse. On a cluster running many services, that central stack is what makes a pod’s logs findable at all, because reading them one kubectl logs at a time does not scale.

The sections below cover each task in this guide:

Prerequisites
Emit logs as JSON
Choose a central stack
Verify the lines arrive parsed

Prerequisites

Confirm the following before you begin.

Access and permissions required

You need the following access and permissions:

Permission to edit the component’s values.yaml and apply it.

Tools and versions required

You need the following tools:

helm >= 3.12, or access to the pipeline that applies the values.

Resources that must exist before starting

The following must already exist:

A log collector running in the cluster, such as Fluentd, gathering pod output. Without one, changing the log format changes nothing downstream.
A central store and query front end for the collector to write to, chosen from the options in Choose a central stack.

Emit logs as JSON

A central stack has to parse human-readable log lines with pattern matching before it can index their fields. JSON arrives already structured, so the fields are queryable without a parsing rule per component.

Supply a full Logback configuration through logbackConfig, which replaces the logging block rather than adding to it.

component-name: (1)
  logbackConfig: |-
    <?xml version="1.0" encoding="UTF-8"?> (2)
    <configuration scan="true" scanPeriod="15 seconds">
        <appender name="CONSOLE_SYSTEM" class="ch.qos.logback.core.ConsoleAppender">
            <encoder class="net.logstash.logback.encoder.LogstashEncoder"/> (3)
        </appender>

        <logger name="io.axual" level="INFO"/> (4)
        <logger name="org.apache.kafka.clients.admin.AdminClientConfig" level="INFO"/>
        <logger name="org.springframework.boot.web" level="INFO"/>
        <logger name="org.springframework.security" level="INFO"/>

        <root level="INFO"> (5)
            <appender-ref ref="CONSOLE_SYSTEM"/>
        </root>
    </configuration>
  logging: (6)
    rootLoglevel: DEBUG
    loggers:
      io.axual: DEBUG

1	One of `platform-manager`, `api-gateway`, `topic-browse` or `rest-proxy`. These are the components that support JSON logging.
2	The Logback documentation covers how the rest of this XML is composed.
3	`LogstashEncoder` is what turns each line into JSON.
4	Per-package levels, as in the `logging` block. The package names depend on the component.
5	Applies to every logger no `<logger>` element above overrides.
6	Ignored entirely, because `logbackConfig` is non-empty. See the note below.

logbackConfig and logging are mutually exclusive. While logbackConfig holds anything at all, the whole logging block is ignored, so setting a level there has no effect. To go back to logging, remove logbackConfig or set it to an empty string, and accept that JSON logging goes with it.

For setting levels through the logging block instead, see How to Configure Logging.

Choose a central stack

Any stack that collects pod output works, and the platform is indifferent to which one. Three are in common use:

Elasticsearch, Fluentd and Kibana (EFK). Elasticsearch indexes and searches the log volume, and Kibana is the web front end for querying it and building dashboards. This is what the Axual cloud runs.
Loki, which pairs with Grafana and so shares a front end with the platform’s metrics.
Splunk.

Point the collector at whichever of these the organisation already runs, rather than standing up a second one for the platform.

Verify the lines arrive parsed

A component that restarts cleanly is no proof the fields are queryable, because a collector that cannot parse a line stores it anyway, as one block of text. Check both ends: what the pod emits, and what the stack made of it.

Replace every <VALUE> placeholder with your own value before running a command.

Read one line straight from the pod and confirm it is JSON:
```
kubectl logs <POD_NAME> --namespace <NAMESPACE> --tail=1
```
Expected output is a single JSON object on one line, with one key per field. A line that still reads as formatted text means logbackConfig never applied, most often because it sits under the wrong component key.
Find those lines in the central stack and confirm they arrived as fields. Filter on log_level, as How to Search Logs in Kibana describes. The filter returns the component’s new lines, and each one carries that field populated.

A line the collector could not parse still arrives, with its whole content in the message field and no log_level to filter on. That is the failure this check exists to catch, and the pod side alone never shows it.