Migrating to axual-kafka charts

Objective

As part of the preparation for the migration from Zookeeper to Kraft, your Kafka cluster should be deployed using axual-kafka charts, which support Kraft deployment. See Axual Kafka.

Replace axual-streaming with axual-kafka charts

For this step, we are going to replace axual-streaming charts with axual-kafka charts, which support Kraft. Your current Chart.yaml file is using axual-streaming charts version 1.2.2 or higher:

Chart.yaml

dependencies:
  - name: "axual-streaming"
    version: "1.2.2"
    repository: "oci://registry.axual.io/axual-charts"
    condition: "axual-streaming.enabled"

Update the dependencies as follows, using axual-kafka charts version 0.1.5:

Chart.yaml

dependencies:
  - name: "axual-kafka"
    version: "0.1.5"
    repository: "oci://registry.axual.io/axual-charts"
    condition: "axual-kafka.enabled"

If you are deploying other components (not Kafka) with axual-streaming charts, keep the dependency on axual-streaming in your Chart.yaml and disable Kafka by setting the following in your values.yaml, axual-streaming.global.kafka.enabled to false

Migrate cluster configs to axual-kafka charts

Set axual-kafka config

Add the following to axual-kafka values. This will enable Kafka deployment, add the kafkaNodePool config, and provide your Kafka cluster name from axual-streaming.kafka.fullnameOverride value:

values.yaml

axual-kafka:
  enabled: true
  kafkaNodePools:
    kafka:
      replicas: <axual-streaming.kafka.kafka.replicas>
      resources:
        limits:
          memory: <axual-streaming.kafka.kafka.resources.limits.memory>
        requests:
          cpu: <axual-streaming.kafka.kafka.resources.requests.cpu>
          memory: <axual-streaming.kafka.kafka.resources.requests.memory>
      templ: <axual-kafka.kafka.templ>
      roles:
        - broker
      storage:
        type: <axual-streaming.kafka.kafka.storageType>
        volumes:
          <axual-streaming.kafka.kafka.volumes>
  fullnameOverride: <axual-streaming.kafka.fullnameOverride>

If you are using affinity rules for broker scheduling, use the following labels for broker:

strimzi.io/broker-role: "true"
strimzi.io/cluster: <axual-kafka.fullnameOverride>

Set Zookeeper config

Copy axual-streaming.kafka.zookeeper to axual-kafka.zookeeper.
Add enabled: true to axual-kafka.zookeeper.
Add metrics: true if you have Kafka metrics enabled (i.e., if axual-streaming.kafka.kafka.metrics is true), as it is disabled by default.
If you have axual-streaming.kafka.zookeeper.externalLoggingEnabled, then replace it with the following:
values.yaml
```
    logging:
      type: external
      externalConfig: |-
        <PLACE_YOUR_LOGGING_CONFIG_HERE>
```
For the pod template, rename the key axual-kafka.zookeeper.template to axual-kafka.zookeeper.templ.

Set Kafka brokers config

Copy axual-streaming.kafka.kafka to axual-kafka.kafka.
Remove axual-kafka.kafka.replicas, axual-kafka.kafka.resources, and axual-kafka.kafka.volumes, since the number of replicas is coming from KafkaNodePool.
Add kraft: "disabled" to axual-kafka.kafka.

If you have axual-streaming.kafka.kafka.externalLogging enabled, replace it with the following:

values.yaml

    logging:
      type: external
      externalConfig: |-
        <PLACE_YOUR_LOGGING_CONFIG_HERE>

If no external logging is defined, add the following default logging config to axual-kafka.kafka:

values.yaml

    logging:
      type: inline
      loggers:
        kafka.root.logger.level: "INFO"
        log4j.rootLogger: "INFO"
        log4j.logger.kafka.controller: "TRACE"
        log4j.logger.kafka.network.Processor: "FATAL"
        log4j.logger.kafka.network.RequestChannel$: "WARN"
        log4j.logger.kafka.request.logger: "WARN"
        log4j.logger.kafka.server.KafkaApis: "FATAL"
        log4j.logger.state.change.logger: "TRACE"

If you have Kafka internalListenerTlsEnabled config, remove it, as the new charts require you to provide internal listener config in the listeners part in the coming steps.
If you have an internal listener (internalListenerTlsEnabled is true), provide internal listener config under axual-kafka.kafka.listeners:
values.yaml
```
      internal:
        type: internal
        authentication:
          type: tls
        tls: true
        port: 9093
```

For the external listener with TLS, provide the config under axual-kafka.kafka.listeners.external:

values.yaml

      external:
        type: loadbalancer
        authentication:
          type: tls
        tls: true
        port: 9094
        configuration:
          <copy-of_axual-kafka.kafka.externalListenerConfiguration>

If you have any other listeners (interClusterListener, scramsha512listener, and additionalListeners), the listeners config will be as follows:

values.yaml

      intcluster:
        type: loadbalancer
        authentication:
          type: tls
        tls: true
        port: 9096
        configuration:
          <copy-of_axual-kafka.kafka.interClusterListener.listenerConfiguration>
      scramsha512:
        type: loadbalancer
        authentication:
          type: scram-sha-512
        tls: true
        port: 9095
        configuration:
          <copy-of_axual-kafka.kafka.scramsha512listener.listenerConfiguration>
      ## additional listener config
      <copy-of_axual-kafka.kafka.additionalListeners>

Now, you can remove the old definition of the listeners from the values file: axual-kafka.kafka.externalListenerType, axual-kafka.kafka.externalListenerConfiguration, interClusterListener, scramsha512listener, and additionalListeners.
For the pod template, rename the key axual-kafka.kafka.template to axual-kafka.kafka.templ.
If you have Cruise Control enabled, move it from axual-streaming.kafka.cruiseControl to axual-kafka.kafka.cruiseControl.spec and add enabled: true.

For the security part of cluster and client CAs:

If you’re not setting up the clientsCa and clusterCa certificates, you may skip this step, since their creation is disabled by default.

Copy axual-streaming.kafka.kafka.security.clientsCaCert to axual-kafka.clientsCaCert.cert.caCrt.
Copy axual-streaming.kafka.kafka.security.clientsCa to axual-kafka.clientsCaCert.key.caKey.
Copy axual-streaming.kafka.kafka.security.clientsCaCertGeneration to axual-kafka.clientsCaCert.cert.generation.

Copy axual-streaming.kafka.kafka.security.clientsCaGeneration to axual-kafka.clientsCaCert.key.generation.

Do the same steps for cluster CA as well. The CA section will look like the following:

values.yaml

axual-kafka:
  clientsCa:
    create: true
    cert:
      generation: <axual-streaming.kafka.kafka.security.clientsCaCertGeneration>
      caCrt: |
        <axual-streaming.kafka.kafka.security.clientsCaCert>
      extraCerts:
        <axual-streaming.kafka.kafka.security.extraCaCerts>
    key:
      generation: <axual-streaming.kafka.kafka.security.clientsCaGeneration>
      caKey: |
        <axual-streaming.kafka.kafka.security.clientsCa>

  clusterCa:
    create: true
    cert:
      generation: <axual-streaming.kafka.kafka.security.clusterCaCertGeneration>
      caCrt: |
        <axual-streaming.kafka.kafka.security.clusterCaCert>
    key:
      generation: <axual-streaming.kafka.kafka.security.clusterCaGeneration>
      caKey: |
        <axual-streaming.kafka.kafka.security.clusterCa>

Now, you can delete axual-kafka.kafka.security.
Copy axual-streaming.kafka.kafka.generateCertificateAuthority to axual-kafka.kafka.clientsCa.generateCertificateAuthority and axual-kafka.kafka.clusterCa.generateCertificateAuthority.
Move axual-kafka.kafka.podMonitor to axual-kafka.podMonitor.kafka and add enabled: true under axual-kafka.podMonitor.kafka.
Add the following configuration to axual-kafka.kafka.kafkaExporter:
values.yaml
```
      spec:
        logging: debug
```
Before deploying your changes, you can expect the following diff:
- The deployment labels will be updated to use new chart labels.
- cc-metrics ConfigMap added.
- kafka-metrics ConfigMap updated with Kraft config.
- zk-metrics ConfigMap added.
- If there is any Kafka config diff, update the config until there is no diff.
- There will be some listener differences; make sure all the listener configurations are present.

You can use helm diff to check the diff between what you have changed in your config and what is actually deployed.

If there is any additional diff, keep updating the config until it is resolved.

Verification

While the Kafka brokers and Zookeeper are restarting:

Watch if the brokers are getting to the ready state.
Check if your client applications (producers/consumers) have any issues.
Verify that all the listeners are there using the openssl command for all listeners.

openssl s_client -connect <listener_host>:<port>

Rollback

If the verification steps are not passed, you can roll back to the axual-streaming charts by updating the dependencies in the Chart.yaml file again, as follows:

Chart.yaml

dependencies:
  - name: "axual-streaming"
    version: "1.2.2"
    repository: "oci://registry.axual.io/axual-charts"
    condition: "axual-streaming.enabled"