Migrating to axual-kafka charts

Objective

As part of the preparation for the migration from Zookeeper to Kraft, your Kafka cluster should be deployed using axual-kafka charts, which support Kraft deployment.

Replace axual-streaming with axual-kafka charts

For this step, we are going to replace axual-streaming charts with axual-kafka charts, which support Kraft. Your current Chart.yaml file is using 1.2.2 axual-streaming charts:

Chart.yaml
dependencies:
  - name: "axual-streaming"
    version: "1.2.2"
    repository: "oci://registry.axual.io/axual-charts"
    condition: "axual-streaming.enabled"

Update the dependencies as follows, using axual-kafka charts version 0.1.4:

Chart.yaml
dependencies:
  - name: "axual-kafka"
    version: "0.1.4"
    repository: "oci://registry.axual.io/axual-charts"
    condition: "axual-kafka.enabled"

Migrate cluster configs to axual-kafka charts

Set axual-kafka config

Add the following to axual-kafka values. This will enable Kafka deployment, add the kafkaNodePool config, and provide your Kafka cluster name from axual-streaming.kafka.fullnameOverride value:

values.yaml
axual-kafka:
  enabled: true
  kafkaNodePools:
    kafka:
      replicas: <axual-streaming.kafka.kafka.replicas>
      resources:
        limits:
          memory: <axual-streaming.kafka.kafka.resources.limits.memory>
        requests:
          cpu: <axual-streaming.kafka.kafka.resources.requests.cpu>
          memory: <axual-streaming.kafka.kafka.resources.requests.memory>
      templ: <axual-kafka.kafka.templ>
      roles:
        - broker
      storage:
        type: <axual-streaming.kafka.kafka.storageType>
        volumes:
          <axual-streaming.kafka.kafka.volumes>
  fullnameOverride: <axual-streaming.kafka.fullnameOverride>

If you are using affinity rules for broker scheduling, use the following labels for broker:

  • strimzi.io/broker-role: "true"

  • strimzi.io/cluster: <axual-kafka.fullnameOverride>

Set Zookeeper config

  1. Copy axual-streaming.kafka.zookeeper to axual-kafka.zookeeper.

  2. Add enabled: true to axual-kafka.zookeeper.

  3. Add metrics: true if you have Kafka metrics enabled (i.e., if axual-streaming.kafka.kafka.metrics is true), as it is disabled by default.

  4. If you have axual-streaming.kafka.zookeeper.externalLoggingEnabled, then replace it with the following:

    values.yaml
        logging:
          type: external
          externalConfig: |-
            <PLACE_YOUR_LOGGING_CONFIG_HERE>
  5. For the pod template, rename the key axual-kafka.zookeeper.template to axual-kafka.zookeeper.templ.

Set Kafka brokers config

  1. Copy axual-streaming.kafka.kafka to axual-kafka.kafka.

  2. Remove axual-kafka.kafka.replicas, axual-kafka.kafka.resources, and axual-kafka.kafka.volumes, since the number of replicas is coming from KafkaNodePool.

  3. Add kraft: "disabled" to axual-kafka.kafka.

  4. If you have axual-streaming.kafka.kafka.externalLogging enabled, replace it with the following:

    values.yaml
        logging:
          type: external
          externalConfig: |-
            <PLACE_YOUR_LOGGING_CONFIG_HERE>

    If no external logging is defined, add the following default logging config to axual-kafka.kafka:

    values.yaml
        logging:
          type: inline
          loggers:
            kafka.root.logger.level: "INFO"
            log4j.rootLogger: "INFO"
            log4j.logger.kafka.controller: "TRACE"
            log4j.logger.kafka.network.Processor: "FATAL"
            log4j.logger.kafka.network.RequestChannel$: "WARN"
            log4j.logger.kafka.request.logger: "WARN"
            log4j.logger.kafka.server.KafkaApis: "FATAL"
            log4j.logger.state.change.logger: "TRACE"
  5. If you have Kafka internalListenerTlsEnabled config, remove it, as the new charts require you to provide internal listener config in the listeners part in the coming steps.

  6. If you have an internal listener (internalListenerTlsEnabled is true), provide internal listener config under axual-kafka.kafka.listeners:

    values.yaml
          internal:
            type: internal
            authentication:
              type: tls
            tls: true
            port: 9093
  7. For the external listener with TLS, provide the config under axual-kafka.kafka.listeners.external:

    values.yaml
          external:
            type: loadbalancer
            authentication:
              type: tls
            tls: true
            port: 9094
            configuration:
              <copy-of_axual-kafka.kafka.externalListenerConfiguration>
  8. If you have any other listeners (interClusterListener, scramsha512listener, and additionalListeners), the listeners config will be as follows:

    values.yaml
          intcluster:
            type: loadbalancer
            authentication:
              type: tls
            tls: true
            port: 9096
            configuration:
              <copy-of_axual-kafka.kafka.interClusterListener.listenerConfiguration>
          scramsha512:
            type: loadbalancer
            authentication:
              type: scram-sha-512
            tls: true
            port: 9095
            configuration:
              <copy-of_axual-kafka.kafka.scramsha512listener.listenerConfiguration>
          ## additional listener config
          <copy-of_axual-kafka.kafka.additionalListeners>
  9. Now, you can remove the old definition of the listeners from the values file: axual-kafka.kafka.externalListenerType, axual-kafka.kafka.externalListenerConfiguration, interClusterListener, scramsha512listener, and additionalListeners.

  10. For the pod template, rename the key axual-kafka.kafka.template to axual-kafka.kafka.templ.

  11. If you have Cruise Control enabled, move it from axual-streaming.kafka.cruiseControl to axual-kafka.kafka.cruiseControl.spec and add enabled: true.

  12. For the security part of cluster and client CAs:

    • Copy axual-streaming.kafka.kafka.security.clientsCaCert to axual-kafka.clientsCaCert.cert.caCrt.

    • Copy axual-streaming.kafka.kafka.security.clientsCa to axual-kafka.clientsCaCert.key.caKey.

    • Copy axual-streaming.kafka.kafka.security.clientsCaCertGeneration to axual-kafka.clientsCaCert.cert.generation.

    • Copy axual-streaming.kafka.kafka.security.clientsCaGeneration to axual-kafka.clientsCaCert.key.generation.

      Do the same steps for cluster CA as well. The CA section will look like the following:

      values.yaml
      axual-kafka:
        clientsCa:
          create: true
          cert:
            generation: <axual-streaming.kafka.kafka.security.clientsCaCertGeneration>
            caCrt: |
              <axual-streaming.kafka.kafka.security.clientsCaCert>
            extraCerts:
              <axual-streaming.kafka.kafka.security.extraCaCerts>
          key:
            generation: <axual-streaming.kafka.kafka.security.clientsCaGeneration>
            caKey: |
              <axual-streaming.kafka.kafka.security.clientsCa>
      
        clusterCa:
          create: true
          cert:
            generation: <axual-streaming.kafka.kafka.security.clusterCaCertGeneration>
            caCrt: |
              <axual-streaming.kafka.kafka.security.clusterCaCert>
          key:
            generation: <axual-streaming.kafka.kafka.security.clusterCaGeneration>
            caKey: |
              <axual-streaming.kafka.kafka.security.clusterCa>
  13. Now, you can delete axual-kafka.kafka.security.

  14. Copy axual-streaming.kafka.kafka.generateCertificateAuthority to axual-kafka.kafka.clientsCa.generateCertificateAuthority and axual-kafka.kafka.clusterCa.generateCertificateAuthority.

  15. Move axual-kafka.kafka.podMonitor to axual-kafka.podMonitor.kafka and add enabled: true under axual-kafka.podMonitor.kafka.

  16. Add the following configuration to axual-kafka.kafka.kafkaExporter:

    values.yaml
          spec:
            logging: debug
  17. Before deploying your changes, you can expect the following diff:

    • The deployment labels will be updated to use new chart labels.

    • cc-metrics ConfigMap added.

    • kafka-metrics ConfigMap updated with Kraft config.

    • zk-metrics ConfigMap added.

    • If there is any Kafka config diff, update the config until there is no diff.

    • There will be some listener differences; make sure all the listener configurations are present.

You can use helm diff to check the diff between what you have changed in your config and what is actually deployed.

If there is any additional diff, keep updating the config until it is resolved.

Verification

While the Kafka brokers and Zookeeper are restarting:

  1. Watch if the brokers are getting to the ready state.

  2. Check if your client applications (producers/consumers) have any issues.

  3. Verify that all the listeners are there using the openssl command for all listeners.

openssl s_client -connect <listener_host>:<port>

Rollback

If the verification steps are not passed, you can roll back to the axual-streaming charts by updating the dependencies in the Chart.yaml file again, as follows:

Chart.yaml
dependencies:
  - name: "axual-streaming"
    version: "1.2.2"
    repository: "oci://registry.axual.io/axual-charts"
    condition: "axual-streaming.enabled"