Axual Kafka

Axual Kafka is a Helm chart designed to deploy and manage a Strimzi-based Axual Kafka cluster with Kraft (Kafka Raft) mode for easy scalability and high performance. This chart simplifies the provisioning of Kafka clusters and related resources, making it highly customizable to suit diverse use cases.

Features

  • Kafka Kraft Support: Leverage Kafka’s built-in Raft protocol, eliminating the need for Zookeeper in your deployments.

  • Highly Configurable:

    • Support for custom Kafka and Zookeeper metrics.

    • Advanced logging configurations (inline or external).

    • Optional Cruise Control for Kafka optimization.

    • Simple authorization with support for super users.

  • Observability:

    • Built-in Prometheus metrics for Kafka and Zookeeper.

    • Pre-configured Prometheus rules and alerts.

    • Integration with JMX Prometheus Exporter for fine-grained monitoring.

  • Template Expansion:

    • Ability to extend Kafka or Zookeeper configurations using native Helm templating.

  • SSL & Security:

    • Cluster CA and client CA management.

    • SSL/TLS configuration for secure Kafka communication.

Prerequisites

Before using this chart, ensure the following are in place:

  1. Helm: Version 3 or above.

  2. Kubernetes Cluster: v1.22+ for compatibility with custom resource definitions (CRDs).

  3. Strimzi Operator: The Kafka Custom Resource and related operations are managed by Strimzi. Install the Strimzi operator in your Kubernetes cluster.

Installation

Log into the Axual Harbor Registry

helm registry login -u [your-user] registry.axual.io/axual-charts

Download the Dependencies

helm dependency update

Install Strimzi Operator

Check the versions you need from the Strimzi downloads page: https://strimzi.io/downloads/.

helm upgrade --install strimzi strimzi/strimzi-kafka-operator \
--version=0.43.0 \
--namespace kafka \
--set watchAnyNamespace=true \
--set kafka.image.registry=registry.axual.io \
--set kafka.image.repository=axual/streaming/strimzi \
--set image.imagePullSecrets='axualdockercred'

Configuration

The chart includes a wide range of configuration options available in the values.yaml. Below are all the configurable parameters:

General Settings

Parameter Description Default

fullnameOverride

Fully override the full resource names

""

nameOverride

Override the base name of the chart

""

Axual Settings

Parameter Description Default

axual.sslPrincipalBuilder.enabled

Enable Axual SSL Principal Builder

false

axual.sslPrincipalBuilder.addChainedSuperUsers

Add chained super users to the SSL Principal Builder

false

axual.sslPrincipalBuilder.chain

List of additional chained super users

[]

Kafka Settings

Parameter Description Default

kafka.enabled

Enable Kafka deployment

true

kafka.version

Kafka version

""

kafka.kraft

Use Kafka Raft (KRaft) mode (enabled, disabled, or migration)

enabled

kafka.nodepools.enabled

Enable Kafka NodePools for the deployment

true

kafka.rack.enabled

Enable rack awareness

false

kafka.rack.topologyKey

Kubernetes topology key for rack awareness

topology.kubernetes.io/zone

kafka.metrics

Enable Kafka metrics for Prometheus

false

kafka.logging.type

Kafka logging type: inline or external

inline

kafka.logging.externalConfig

External logging configuration (only used if logging.type is external)

See values.yaml

kafka.logging.loggers

Inline logger configuration

{}

kafka.kafkaExporter.enabled

Enable the Kafka Exporter for metrics

false

kafka.kafkaExporter.spec

Specification for the Kafka Exporter

Defaults from Strimzi

kafka.authorization.superUsers

List of superusers for Kafka authorization

[]

kafka.listeners

List of Kafka listeners defined per your requirements

{}

kafka.config

Kafka configuration properties

See below:

kafka.config.auto.create.topics.enable

Automatically create topics

false

kafka.config.offsets.topic.replication.factor

Replication factor for offsets topic

1

kafka.config.transaction.state.log.replication.factor

Replication factor for transaction state

1

kafka.config.transaction.state.log.min.isr

Minimum in-sync replicas for transaction state

1

kafka.config.default.replication.factor

Default replication factor for topics

1

kafka.config.min.insync.replicas

Minimum in-sync replicas for producing

1

kafka.config.allow.everyone.if.no.acl.found

Allow everyone when ACL is not found

false

kafka.clientsCa.generateCertificateAuthority

Generate a Clients Certificate Authority

true

kafka.clusterCa.generateCertificateAuthority

Generate a Cluster Certificate Authority

true

kafka.entityOperator

Config for the Kafka Entity Operator

{}

kafka.cruiseControl.enabled

Enable Cruise Control for dynamic optimization

true

kafka.cruiseControl.jmxPrometheusMetricsEnabled

Enable JMX Prometheus metrics for Cruise Control

true

kafka.cruiseControl.spec

Additional specifications for Cruise Control

{}

kafka.templ

Additional templ section for custom Kafka specs

{}

kafka.extendSpec

Additional Kafka spec values

{}

Zookeeper Settings (Not Needed if KRaft is Enabled)

Parameter Description Default

zookeeper.enabled

Enable Zookeeper for Kafka

false

zookeeper.replicas

Number of Zookeeper replicas

3

zookeeper.storage.size

Persistent storage size for Zookeeper

10Gi

zookeeper.metrics

Enable Prometheus metrics for Zookeeper

false

zookeeper.logging.type

Logging type for Zookeeper: inline or external

inline

Monitoring & Observability

Parameter Description Default

podMonitor.kafka.enabled

Enable PodMonitor for Kafka

false

podMonitor.kafka.labels

Labels for the Kafka PodMonitor

{}

podMonitor.kafka.scrapeTimeout

Scrape timeout for Kafka Prometheus metrics

20s

podMonitor.kafka.interval

Scrape interval for Kafka Prometheus metrics

30s

podMonitor.entityOperator.enabled

Enable PodMonitor for Kafka Entity Operator

false

podMonitor.entityOperator.labels

Labels for the Entity Operator PodMonitor

{}

podMonitor.entityOperator.scrapeTimeout

Scrape timeout for Entity Operator Prometheus metrics

20s

podMonitor.entityOperator.interval

Scrape interval for Entity Operator Prometheus metrics

30s

prometheusRule.enabled.kafka

Enable Prometheus Rules for Kafka alerts

false

prometheusRule.enabled.zookeeper

Enable Prometheus Rules for Zookeeper alerts

false

prometheusRule.labels

Labels for Prometheus Rules

{}

Node Pool Settings (Optional)

Parameter Description Default

kafkaNodePools.controller

Optional settings for the Kafka Controller NodePools

{}

kafkaNodePools.broker

Optional settings for the Kafka Broker NodePools

Values Examples

Basic Kafka Cluster with Kraft

kafka:
  enabled: true
  kraft: true
  version: 3.5.0
  replicas: 3
  metrics: true
  logging:
    type: inline
    loggers:
      root: "INFO"
zookeeper:
  enabled: false
prometheusRule:
  enabled:
    kafka: true

Kafka with External Logging Config

kafka:
  enabled: true
  kraft: true
  logging:
    type: external
    externalConfig:
      log4j.properties: |
        log4j.rootLogger=INFO, stdout
        log4j.logger.org.apache.kafka=DEBUG

Observability & Prometheus Alerts

The chart configures Prometheus alerts for Kafka and Zookeeper. Example of configured alerts:

  • KafkaRunningOutOfSpace: Indicates low disk space on Kafka brokers.

  • UnderReplicatedPartitions: Triggers when Kafka partitions are under-replicated.

  • OfflinePartitions: Alerts when certain Kafka partitions do not have an active leader.

For additional information, refer to the official Strimzi documentation: https://strimzi.io/documentation/.