How to Enable JSON Logging in the Axual Distributor

This guide shows you how to create the logging ConfigMap the Axual Distributor needs, in the format matching your Strimzi version, and how to point the Axual Distributor at it.

Type

How-to guide

Goal

Make the Axual Distributor write its log entries as JSON so a log collector can index the fields.

Audience

Platform Operator who can create ConfigMaps in the Axual Distributor’s namespace and edit its values.yaml.

When to use

Use this guide when Axual Distributor logs are being collected into a central stack that parses JSON.
Some Axual Distributor subcomponents never write JSON, usually during startup. Those lines reach the collector as plain text and may produce parse warnings in the logging stack. This is expected and is not a misconfiguration.

Contents

The sections below cover each task in this guide:

Prerequisites

Confirm the following before you begin.

Access and permissions required

You need the following access and permissions:

  • Permission to create ConfigMaps in the namespace the Axual Distributor runs in.

  • Permission to edit the Axual Distributor’s values.yaml and apply it.

Tools and versions required

You need the following tools:

  • kubectl >= 1.28.

  • helm >= 3.12, or access to the pipeline that applies the values.

Resources that must exist before starting

The following must already exist:

  • Your Strimzi Operator version. It decides which of the two formats below to use, and the two are not interchangeable.

Strimzi versions before 0.46.1 take Log4J. Strimzi 0.46.1 and newer take Log4J2. Supplying the wrong format leaves the Axual Distributor with no working appender.

Create the logging ConfigMap

Create the ConfigMap in the same namespace as the Axual Distributor, in the format matching your Strimzi version.

For Strimzi versions before 0.46.1, use Log4J:

apiVersion: v1
kind: ConfigMap
metadata:
 name: distributor-json-logging-generic
data:
 log4j.properties: |-
   ### Configure logging to write to the console
   log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
   ### Configure logging to in Json format to the console
   log4j.appender.CONSOLE.layout=net.logstash.log4j.JSONEventLayoutV1
   ### Default log level is set to INFO, and uses the CONSOLE appender
   log4j.rootLogger=INFO, CONSOLE
   ### Specify some default log levels for the Distributor
   log4j.logger.io.axual.distributor.common=INFO
   log4j.logger.io.axual.distributor.message=INFO
   log4j.logger.io.axual.distributor.offset=INFO
   log4j.logger.io.axual.distributor.schema=INFO
   log4j.logger.org.apache.kafka.clients.consumer=WARN
   log4j.logger.org.apache.kafka.clients.producer=WARN
   log4j.logger.org.apache.kafka.clients.admin=WARN

For Strimzi 0.46.1 and newer, use Log4J2:

apiVersion: v1
kind: ConfigMap
metadata:
 name: distributor-json-logging-generic
data:
 log4j2.properties: |-
   name = DistributorJsonLogging
   appenders = console
   appender.console.type = Console
   appender.console.name = STDOUT
   appender.console.layout.type = JsonTemplateLayout
   appender.console.layout.eventTemplateUri = classpath:EcsLayout.json
   rootLogger.level = INFO
   rootLogger.appenderRefs = console
   rootLogger.appenderRef.console.ref = STDOUT
   logger.distributorCommon.name = io.axual.distributor.common
   logger.distributorCommon.level = INFO
   logger.distributorMessage.name = io.axual.distributor.message
   logger.distributorMessage.level = INFO
   logger.distributorOffset.name = io.axual.distributor.offset
   logger.distributorOffset.level = INFO
   logger.distributorSchema.name = io.axual.distributor.schema
   logger.distributorSchema.level = INFO
   logger.kafkaConsumer.name = org.apache.kafka.clients.consumer
   logger.kafkaConsumer.level = WARN
   logger.kafkaProducer.name = org.apache.kafka.clients.producer
   logger.kafkaProducer.level = WARN
   logger.kafkaAdmin.name = org.apache.kafka.clients.admin
   logger.kafkaAdmin.level = WARN

Point the Axual Distributor at the ConfigMap

Set connect.logging to external and reference the key inside the ConfigMap. The key name has to match the one used above, so it differs with the format.

For Strimzi versions before 0.46.1:

connect:
  logging:
    type: external
    valueFrom:
      configMapKeyRef:
        name: distributor-json-logging-generic
        key: log4j.properties

For Strimzi 0.46.1 and newer:

connect:
  logging:
    type: external
    valueFrom:
      configMapKeyRef:
        name: distributor-json-logging-generic
        key: log4j2.properties

Apply the values and read a few lines of the Axual Distributor’s log output to confirm they arrive as JSON. Read a pod’s logs covers the command.