Kafka Chart Values Reference

This reference describes the values the Axual Kafka chart exposes: the Strimzi resources it creates, and the controller and broker configuration behind them.

Type

Reference

Goal

Look up a Kafka chart value while writing or reviewing the Axual Kafka values.

Audience

Platform Operator configuring the Apache Kafka cluster the platform runs on.

When to use

While configuring Kafka, alongside the procedure that installs the cluster.

For the procedure that applies these values, see How to Deploy Axual Kafka.

Contents

The sections below cover each area in this reference:

About Strimzi Operator

Strimzi deploys and monitors a Kafka cluster in KRaft mode, in which Kafka controllers manage cluster metadata and state directly. A cluster is made up of controllers and brokers, and both are configured through the values below.

Strimzi Configuration

The values below cover the settings an Axual Streaming installation normally sets, with examples you can build your own values.yaml from. The Strimzi Documentation covers the remaining options.

Deployment name

The deployment name decides the names Strimzi gives to the resources it deploys, such as pods and Secrets. By default it is the name of the Helm chart, for example axual-streaming.

You have two options to change the deployment name:

  1. With fullnameOverride, the deployment name is replaced entirely. In the example below it becomes local.

    values.yaml
    kafka:
  fullnameOverride: "local"
  nameOverride: ""

  2. With nameOverride, the deployment name takes the form <chart-name>-<nameOverride>. For the previous example that gives axual-streaming-local.

Enable Kafka and Kafka version

Enable Kafka and pin its version as follows. Check that the Strimzi version you run supports the Kafka version you ask for, on the Strimzi supported versions page.

values.yaml
kafka:
  kafka:
    enabled: true
    version: 3.8.0

Rack topology key

Rack awareness spreads the replicas of a partition across racks or zones, which keeps the partition available when one rack fails. Set rack.enabled to true to turn it on, then name the node label that identifies the rack in rack.topologyKey. The Strimzi documentation covers the topology keys a cluster can use.

values.yaml
kafka:
  kafka:
    rack:
      enabled: true
      topologyKey: topology.kubernetes.io/zone

Listeners

A Kafka listener is a network endpoint a broker uses to accept incoming client connections. Each listener carries its own network protocol, port and settings.

An internal listener serves clients inside the same Kubernetes cluster. An external listener serves clients outside it.

Internal listener

The Axual Streaming chart defines an internal listener on port 9093. Set kafka.internalListenerTlsEnabled to true to enable TLS on it; it defaults to false.

kafka.internalListenerAuthenticationType sets the authentication on the internal listener. Leave it empty for no authentication. The supported values are tls, scram-sha-512, oauth and custom.

External listeners

To reach the Kafka cluster from outside the Kubernetes cluster, configure the externalListener:

  • externalListenerType defines the type of external listener. You can use NodePort or LoadBalancer.

  • externalListenerTlsEnabled is the TLS feature toggle

  • externalListenerAuthenticationType defines the authentication on the listener. Leave it empty for no authentication. The supported values are tls, scram-sha-512, oauth and custom.

  • In externalListenerConfiguration you need to define:

    • bootstrap: the external bootstrap service that clients use to initially connect to the Kafka cluster

    • list of brokers: for each of them, the advertisedHost, advertisedPort, nodePort and annotations

The external listener types are NodePort, LoadBalancer, Ingress and Route. The Strimzi documentation describes how the listener types differ.

The example below runs one Kafka node, with TLS on the internal listener and on an external listener of type NodePort.

values.yaml
kafka:
  kafka:
    replicas: 1
    internalListenerTlsEnabled: "true"
    internalListenerAuthenticationType: "tls"

    externalListenerTlsEnabled: "true"
    externalListenerAuthenticationType: "tls"
    externalListenerType: nodeport
    externalListenerConfiguration:
      bootstrap:
        annotations: {}
        alternativeNames:
          - <alternative-name-1>
      brokers:
        - broker: 0
          advertisedHost: <advertised-host-broker-0>
          advertisedPort: <advertised-port-broker-0>
          nodePort: <node-port-broker-0>
          annotations: <annotation-broker-0>
This is a configuration example. The replica count and the rest of the settings depend on what the Kafka cluster has to carry.

Defining storage

The storage values set how the Kafka cluster stores its data: the storage type, the size, and whether the storage claim is deleted when the cluster is removed. The example below shows a Kafka storage configuration. The Strimzi documentation covers Kafka storage in full.

values.yaml
kafka:
  kafka:
    storageType: jbod # acceptable values are jbod, ephemeral, persistent
    volumes:
      - id: 0
        type: persistent-claim
        size: 1Gi
        deleteClaim: false
This is a configuration example. The volume size and the rest of the settings depend on what the Kafka cluster has to carry.

Deploying broker replicas

You can change the number of replicas of your Kafka cluster by changing the value of replicas. When you change the number of replicas, add the new brokers to externalListenerConfiguration. Storage needs no new volumes. The snippet below shows a cluster with 3 replicas.

values.yaml
kafka:
  kafka:
    replicas: 3
    externalListenerType: nodeport
    externalListenerConfiguration:
      bootstrap:
        annotations: {}
        alternativeNames:
          - <alternative-name-1>
      brokers:
        - broker: 0
          advertisedHost: <advertised-host-broker-0>
          advertisedPort: <advertised-port-broker-0>
          nodePort: <node-port-broker-0>
          annotations: <annotation-broker-0>
        - broker: 1
          advertisedHost: <advertised-host-broker-1>
          advertisedPort: <advertised-port-broker-1>
          nodePort: <node-port-broker-1>
          annotations: <annotation-broker-1>
        - broker: 2
          advertisedHost: <advertised-host-broker-2>
          advertisedPort: <advertised-port-broker-2>
          nodePort: <node-port-broker-2>
          annotations: <annotation-broker-2>
    storageType: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 1Gi
        deleteClaim: false
This is a configuration example. The replica count and the rest of the settings depend on what the Kafka cluster has to carry.

Controllers configuration (KRaft mode)

When running in KRaft mode, Strimzi deploys controller pods to manage metadata and cluster state. Controllers manage metadata for topics, brokers, and configurations, and they form a quorum to ensure high availability.

Configure the number of controller replicas and their storage in your values.yaml file, the same way you configure brokers.

values.yaml
kafka:
  kafka:
    kraft:
      enabled: true
      controllers:
        replicas: 3
        storage:
          type: persistent-claim
          size: 1Gi
          deleteClaim: false

Three values in that block decide the controller layout:

  • replicas, the number of controller nodes, which must be odd to form a quorum

  • storage, the storage configuration for controller metadata

  • deleteClaim, whether the PersistentVolumeClaim is deleted when the Kafka cluster is removed

When kraft.enabled is set to true, Strimzi configures the internal communication between controllers and brokers itself.
This is a configuration example. The replica count and the rest of the settings depend on what the Kafka cluster has to carry.

Other listeners

You can define further listeners, such as interClusterListener, scramsha512listener and oauthListener. The Strimzi documentation describes what each one is for.

values.yaml
kafka:
  kafka:
    scramsha512listener:
      enabled: true
      listenerType: nodeport
      listenerConfiguration:
        bootstrap:
          annotations: {}
          alternativeNames:
            - <alternative-name>
        brokers:
          - broker: 0
            advertisedHost: <advertised host>
            advertisedPort: <advertised port>
            nodePort: <node port>
            annotations: <annotations>

Kafka configs

Broker settings go under config. Two entries in the example below are worth calling out:

  • default.replication.factor is 3 and min.insync.replicas is 2, which match the 3-replica deployment above

  • principal.builder.class is io.axual.security.auth.SslPrincipalBuilder, which only works on Axual Kafka images. Principal Chain Builder explains how that builder assembles the principal chain.

values.yaml
kafka:
  kafka:
    config:
      # Change based on the installed Kafka version
      inter.broker.protocol.version: "3.8"
      # Enable custom principal builder class
      principal.builder.class: io.axual.security.auth.SslPrincipalBuilder
      unclean.leader.election.enable: false
      background.threads: 16
      num.replica.fetchers: 4
      replica.lag.time.max.ms: 20000
      message.max.bytes: 1000012
      replica.fetch.max.bytes: 1048576
      replica.socket.receive.buffer.bytes: 65536
      offsets.retention.minutes: 20160
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      transaction.state.log.num.partitions: 3
      default.replication.factor: 3
      min.insync.replicas: 2
This is a configuration example. The exact settings depend on what the Kafka cluster has to carry.

Superusers definition

A superuser is a principal the brokers allow every action, whatever the permission grants say. That covers creating and deleting topics, and viewing and modifying consumer group offsets.

The example below identifies each principal by its SSL chain. To use the standard Kafka identity instead, give only the CN.

Every certificate listed must be issued by a Certificate Authority (CA) the Kafka brokers trust.

values.yaml
kafka:
  kafka:
    authorization:
      superUsers:
        - "[0] CN=Root CA, [1] CN=Intermediate CA, [2] CN=Demo Superuser,O=Axual B.V.,L=Utrecht,ST=Utrecht,C=NL"
        - "[0] CN=Root CA, [1] CN=Intermediate CA, [2] CN=local-kafka,O=io.strimzi"
This is a configuration example. The exact settings depend on the certificates in use.

KafkaExporter

kafkaExporter is optional. Once enabled, it exports Kafka metrics such as consumer group lag and topic and partition sizes to a monitoring system.

values.yaml
kafka:
  kafka:
    kafkaExporter:
      enabled: true

generateCertificateAuthority option for Strimzi

generateCertificateAuthority decides whether Strimzi generates its own CA certificates. Set it to true and Strimzi creates a new CA when you deploy a Kafka cluster. That CA then signs the certificates used between brokers, and between clients and the cluster.

values.yaml
kafka:
  kafka:
    generateCertificateAuthority: true

Security

The security section holds the CA details that secure traffic inside the Kafka cluster: for each CA, a private key and a certificate in Privacy Enhanced Mail (PEM) format.

  • clientsCaCert: the public certificate of the CA that issues client certificates. Kafka brokers validate the certificates clients present against it.

  • clientsCa: the private key of that CA, which signs the client certificates.

  • clusterCaCert: the public certificate of the cluster CA, which issues the broker certificates. Clients and other brokers verify a broker’s identity against it.

  • clusterCa: the private key of the cluster CA, which signs the broker certificates.

Configure them as follows:

values.yaml
kafka:
  kafka:
    security:
      clientsCaCert: <clients-ca-cert>
      clientsCa: <clients-ca>
      clusterCaCert: <cluster-ca-certificate>
      clusterCa: <cluster-certificate>
      extraCaCerts: {}
      clientsCaCertGeneration: "0"
      clientsCaGeneration: "0"
      clusterCaCertGeneration: "0"
      clusterCaGeneration: "0"

Kafka PodMonitoring

If you run Prometheus, enable Kafka PodMonitoring by adding the following to your values.yaml file.

values.yaml
kafka:
  kafka:
    metrics: true

To change scrapeTimeout and interval, add the following.

values.yaml
kafka:
  kafka:
    podMonitor:
      scrapeTimeout: "20s"
      interval: "30s"
      labels: {}

Alerting

The deployment includes a PrometheusRule, which provides the alerts for Kafka and for the Strimzi operator. Acting on Alerts lists every alert, with its severity, its default threshold, what it means and the response for it.

To enable alerting, set kafka.strimziAlerts.enabled to true.

values.yaml
kafka:
  strimziAlerts:
    enabled: true
    labels: {}

kafka.strimziAlerts.labels adds custom labels to every alert the chart ships. A label such as severity: LEVEL lets your alerting stack filter the alerts and route them to different channels.

kafka.prometheusRule.labels adds labels to the PrometheusRule resource itself.

values.yaml
kafka:
  prometheusRule:
    labels: {}