Performing the upgrade using HELM charts

Typical upgrade steps

A lot of the upgrade steps can be performed without impact on your users. Basically, the deployment or upgrade of components is split in two actions:

Configuration changes, such as added or changed configuration parameters, including the new component’s version
Deployment of the upgraded component, using the helm upgrade command.

Performing the upgrade

Step 1 - Update Keycloak Configuration

In order to enable PKCE flow, you need to change some configurations from the Keycloak Admin console.

Step 1a - Configure the Client

Access the Keycloak Admin Console.

Select Clients > Self-service.

The Self-Service application needs to be Public and the Standard Flow needs to be enabled.

Step 1b - Set Proof Key for Code Exchange Code Challenge Method

Scroll further down the settings page and expand the Advanced Settings section. For the Proof Key for Code Exchange Code Challenge Method option, select S256

If you need to tweak Keycloak configuration further, go to Keycloak Configuration

In Axual Platform v0.6.2, default Management UI version has been updated to v5.13.0

If you have overridden any of image.tag, csp.extraCSP, oidc.responseTypes in your UI values.yaml, remove those overrides in order to get the latest version of the Management UI running.

Step 2 - Reconfiguring Schema Registry

In the new release v5.1.0, multi cluster support has been added that requires configuration changes in Schema Registry to initialize.

To accommodate the feature following changes are required:

defining distributionPrincipal and distributionPrincipalChain
defining all Cluster instances in an array format as part of the deploySchemasTopic init container.
defining client certificates that will use to make SSL connection to cluster-api to deploy schema topics.

Update the Schema Registry configurations in your values.yaml file.

Example:

platform:
  instance:
    # SR override
    schemaregistry:
        deploySchemasTopic:
          replicationFactor: 3
          minIsr: 2
          registerPatternACLforTopic:
            principal: "CN=schemas.cloud.axual.io,O=Axual B.V.,L=Utrecht,ST=Utrecht,C=NL"
            principalChain: "[0] CN=Axual Root 2021,O=Axual B.V.,L=Utrecht,ST=Utrecht,C=NL, [1] CN=schemas.cloud.axual.io,O=Axual B.V.,L=Utrecht,ST=Utrecht,C=NL"
            ## When distribution is not enabled, same principal as above can be configured below
            distributionPrincipal: "CN=distribution.cloud.axual.io,O=Axual B.V.,L=Utrecht,ST=Utrecht,C=NL"
            distributionPrincipalChain: "[0] CN=Axual Root 2021,O=Axual B.V.,L=Utrecht,ST=Utrecht,C=NL, [1] CN=distribution.cloud.axual.io,O=Axual B.V.,L=Utrecht,ST=Utrecht,C=NL"

          clusterConfig:
            - name: cluster-a
              primary: true
              # {cluster-internal-ip}
              clusterHealthUrl: cluster-a-cluster-api-mgmt:8081
              # {cluster-intercluster-advertised-host}
              clusterBaseUrl: cluster-a-cluster-api.kafka:9080
            - name: cluster-b
              primary: false
              # {cluster-internal-ip}
              clusterHealthUrl: cluster-b-cluster-api-mgmt:8081
              # {cluster-intercluster-advertised-host}
              clusterBaseUrl: cluster-b-cluster-api.kafka:9080

           # client certs to make SSL connection to cluster-api
          security:
            deploy:
              crt: |
                -----BEGIN CERTIFICATE-----
                MIIEwzCCAqugAwIBAgIQaZA4yVYjXRxy8RojSwEfxjANBgkqhkiG9w0BAQsFADAr
                ...
                -----END CERTIFICATE-----
              key: |
                -----BEGIN RSA PRIVATE KEY-----
                MIIEowIBAAKCAQEAv6f7sBue5E1HFKY1FNFWLOVmKmZ7jziez5W8oXXCpK3lJBet
                ...
                -----END RSA PRIVATE KEY-----
              ca: |
                -----BEGIN CERTIFICATE-----
                MIIFLTCCAxWgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwIDEeMBwGA1UEAwwVQXh1
                ...
                -----BEGIN CERTIFICATE-----

If running in single cluster:

The value of distributionPrincipal and distributionPrincipalChain will be the same as principal and principalChain.
clusterConfig will have single cluster.

Step 3 - Upgrade Axual Operator to 0.3.0

Step 3a - Upgrade Axual Operator

Update the Axual helm repository to download the latest charts available.
```
helm repo update
```
Upgrade Axual Operator
```
helm upgrade --install strimzi --set watchAnyNamespace=true axual-stable/axual-operator --version=0.3.0 -n kafka
```
This command will restart the strimzi-cluster-operator, Zookeeper & kafka pods.

Verify:
- strimzi-cluster-operator pod should use image: docker.axual.io/axual/strimzi/operator:0.22.1
- zookeeper pod should use image: docker.axual.io/axual/strimzi/kafka:0.22.1-kafka-2.5.0
- kafka pod should use image: docker.axual.io/axual/strimzi/kafka:0.22.1-kafka-2.5.0
Upgrade Axual Operator CRDs

You can skip this step if deploying via ArgoCD as it automatically includes the CRDs.
- Download CRDs
  helm pull axual-stable/axual-operator --version 0.3.0
- Extract the downloaded tar file
  tar xvzf axual-operator-0.3.0.tgz
- Upgrade CRDs using kubectl as there is no support at this time for upgrading CRDs using Helm
  kubectl replace -f ./axual-operator/crds/

Kafka brokers will do a rolling restart with a new docker image (0.22.1-kafka-2.5.0). Once restarted, verify everything is running fine before moving to next step.

Step 3b - Modify Kafka Configuration

What changed in Axual Operator 0.3.0?

Axual Operator 0.3.0 needs a change in configuration parameters, as the old ones will not be supported anymore in newer versions. The following parameters need changes:

externalListenerIngressClass changed to class and moved under externalListenerConfiguration section parallel to bootstrap and brokers.
Under bootstrap, dnsAnnotations is renamed to annotations.
Under brokers, dnsAnnotations is renamed to annotations.

Updating the configurations

Update the Kafka configurations in your values.yaml file.

Example for externalListenerType=ingress:

Kafka Configuration prior to version 0.3.0

platform:
  core:
    kafka:
      kafka:
        version: 2.5.0
        ...
        ...
        externalListenerType: ingress
        externalListenerIngressClass: nginx-public
        externalListenerConfiguration:
          bootstrap:
            dnsAnnotations:
              external-dns.alpha.kubernetes.io/hostname: kafka-bootstrap.dev.axual.cloud
              external-dns.alpha.kubernetes.io/ttl: 60
            host: "kafka-bootstrap.dev.axual.cloud"
          brokers:
            - broker: 0
              dnsAnnotations:
                external-dns.alpha.kubernetes.io/hostname: kafka-0.dev.axual.cloud
                external-dns.alpha.kubernetes.io/ttl: 60
              host: "kafka-0.dev.axual.cloud"
            - broker: 1
              dnsAnnotations:
                external-dns.alpha.kubernetes.io/hostname: kafka-1.dev.axual.cloud
                external-dns.alpha.kubernetes.io/ttl: 60
              host: "kafka-1.dev.axual.cloud"
            - broker: 2
              dnsAnnotations:
                external-dns.alpha.kubernetes.io/hostname: kafka-2.dev.axual.cloud
                external-dns.alpha.kubernetes.io/ttl: 60
                axual.com/service.class: dev
              host: "kafka-2.dev.axual.cloud"
          brokerCertChainAndKey:
            certificate: tls.crt
            key: tls.key
            secretName: dev-axual-cloud-wildcard
        config:
          inter.broker.protocol.version: "2.5"
          log.message.format.version: "2.5"
          ...

New Kafka Listeners Configuration:

platform:
  core:
    kafka:
      kafka:
        version: 2.5.0
        ...
        ...
        externalListenerType: ingress
        externalListenerConfiguration:
          class: nginx-public  (1)
          bootstrap:
            annotations: (2)
              external-dns.alpha.kubernetes.io/hostname: kafka-bootstrap.dev.axual.cloud
              external-dns.alpha.kubernetes.io/ttl: 60
            host: "kafka-bootstrap.dev.axual.cloud"
          brokers:
            - broker: 0
              annotations: (3)
                external-dns.alpha.kubernetes.io/hostname: kafka-0.dev.axual.cloud
                external-dns.alpha.kubernetes.io/ttl: 60
              host: "kafka-0.dev.axual.cloud"
            - broker: 1
              annotations:
                external-dns.alpha.kubernetes.io/hostname: kafka-1.dev.axual.cloud
                external-dns.alpha.kubernetes.io/ttl: 60
              host: "kafka-1.dev.axual.cloud"
            - broker: 2
              annotations:
                external-dns.alpha.kubernetes.io/hostname: kafka-2.dev.axual.cloud
                external-dns.alpha.kubernetes.io/ttl: 60
              host: "kafka-2.dev.axual.cloud"
          brokerCertChainAndKey:
            certificate: tls.crt
            key: tls.key
            secretName: dev-axual-cloud-wildcard
        config:
          inter.broker.protocol.version: "2.5"
          log.message.format.version: "2.5"
          ...

Example for externalListenerType=loadbalancer :

Kafka Configuration prior to version 0.3.0

platform:
  core:
    kafka:
      kafka:
        version: 2.5.0
        ...
        ...
        externalListenerType: loadbalancer
        externalListenerConfiguration:
        externalBootstrapOverrides:
          address: "kafka-bootstrap.dev.axual.cloud"
          dnsAnnotations:
            external-dns.alpha.kubernetes.io/hostname: kafka-bootstrap.dev.axual.cloud
            external-dns.alpha.kubernetes.io/ttl: 60
            axual.com/service.class: [your-namespace]
        externalBrokerOverrides:
          - id: 0
            advertisedHost: "kafka-0.dev.axual.cloud"
            dnsAnnotations:
              external-dns.alpha.kubernetes.io/hostname: kafka-0.dev.axual.cloud
              external-dns.alpha.kubernetes.io/ttl: 60
              axual.com/service.class: [your-namespace]
          - id: 1
            advertisedHost: "kafka-1.dev.axual.cloud"
            dnsAnnotations:
              external-dns.alpha.kubernetes.io/hostname: kafka-1.dev.axual.cloud
              external-dns.alpha.kubernetes.io/ttl: 60
              axual.com/service.class: [your-namespace]
          - id: 2
            advertisedHost: "kafka-2.dev.axual.cloud"
            dnsAnnotations:
              external-dns.alpha.kubernetes.io/hostname: kafka-2.dev.axual.cloud
              external-dns.alpha.kubernetes.io/ttl: 60
              axual.com/service.class: [your-namespace]
          brokerCertChainAndKey:
            certificate: tls.crt
            key: tls.key
            secretName: dev-axual-cloud-wildcard
        config:
          inter.broker.protocol.version: "2.5"
          log.message.format.version: "2.5"
          ...

New Kafka Listeners Configuration:

platform:
  core:
    kafka:
      kafka:
        version: 2.5.0
        ...
        ...
        externalListenerType: loadbalancer
        externalListenerConfiguration:
          bootstrap: (1)
            annotations: (2)
              external-dns.alpha.kubernetes.io/hostname: kafka-bootstrap.dev.axual.cloud
              external-dns.alpha.kubernetes.io/ttl: 60
              axual.com/service.class: [your-namespace]
          brokers: (3)
            - broker: 0
              advertisedHost: "kafka-0.dev.axual.cloud" (4)
              annotations: (5)
                external-dns.alpha.kubernetes.io/hostname: kafka-0.dev.axual.cloud
                external-dns.alpha.kubernetes.io/ttl: 60
                axual.com/service.class: [your-namespace]
            - broker: 1
              advertisedHost: "kafka-1.dev.axual.cloud"
              annotations:
                external-dns.alpha.kubernetes.io/hostname: kafka-1.dev.axual.cloud
                external-dns.alpha.kubernetes.io/ttl: 60
                axual.com/service.class: [your-namespace]
            - broker: 2
              advertisedHost: "kafka-2.dev.axual.cloud"
              annotations:
                external-dns.alpha.kubernetes.io/hostname: kafka-2.dev.axual.cloud
                external-dns.alpha.kubernetes.io/ttl: 60
                axual.com/service.class: saastrial
          brokerCertChainAndKey:
            certificate: tls.crt
            key: tls.key
            secretName: dev-axual-cloud-wildcard
        config:
          inter.broker.protocol.version: "2.5"
          log.message.format.version: "2.5"
          ...

Step 3c - Apply the new configurations

To apply new Kafka configuration, upgrade Axual platform using the modified values.yaml Step 3b

In Axual Platform v0.6.2, default Kafka version updated to v2.7.0

If you have not overridden the kafka.version, inter.broker.protocol.version and log.message.format.version in your values.yaml file, after Axual platform upgrade, Kafka will also upgrade to v2.7.0.
If you have overridden the kafka.version, inter.broker.protocol.version and log.message.format.version, continue to follow Step 4 - Upgrade Kafka to 2.7.0

helm upgrade --install platform axual-stable/platform -f values.yaml --version=0.6.2 -n kafka

Verify the following:

Zookeeper will do a rolling restart with new image (0.22.1-kafka-2.7.0).
Kafka brokers will do a rolling restart with new image (0.22.1-kafka-2.7.0).
Verify everything is running fine.

Check strimzi-cluster-operator pod logs and there should not be any ERROR or WARN logs.

Step 4 - Upgrade Kafka to 2.7.0

After Axual Operator has been upgraded, Kafka can be upgraded to version 2.7.0. The upgrade of Kafka is executed in two steps: upgrading the Kafka binaries and upgrading the inter.broker.protocol.version and log.format.version. For both steps, a rolling restart of all the brokers is executed.

Modify the Kafka version in your values.yaml file

Example:

platform:
  core:
    kafka:
      kafka:
        version: 2.7.0
        ...
        config:
          inter.broker.protocol.version: "2.5"
          log.message.format.version: "2.5"
          ...

Upgrade Axual platform using the above modified values.yaml.
```
helm upgrade --install platform axual-stable/platform -f values.yaml --version=0.6.2 -n kafka
```
Verify:
- Zookeeper will perform a rolling restart with a new docker image (0.22.1-kafka-2.7.0).
- Kafka brokers will perform a rolling restart with a new docker image (0.22.1-kafka-2.7.0).
- Verify everything is running fine.

Edit your values.yaml again and update broker config inter.broker.protocol.version & log.message.format.version to 2.7.

Example:

platform:
  core:
    kafka:
      kafka:
        version: 2.7.0
        ...
        config:
          inter.broker.protocol.version: "2.7"
          log.message.format.version: "2.7"
          ...

Apply above changes:

helm upgrade --install platform axual-stable/platform -f values.yaml --version=0.6.2 -n kafka

Verify

Kafka brokers will perform a rolling restart with a new docker image (0.22.1-kafka-2.7.0). Once restarted verify everything is running fine.

After logging in to the updated Self Service UI, disable the Implicit Flow in Keycloak.