Performing the upgrade

Typical upgrade steps

A lot of the upgrade steps can be performed without impact on your users. Basically, the deployment or upgrade of components is split in two actions:

Configuration changes, such as added or changed configuration parameters, including the new component’s version
Deployment of the upgraded component, by (re)starting

The configuration changes can be done in advance most of the time, limiting downtime for your end users.

In the following upgrade steps, platform-config refers to the location where your platform configuration is stored for that particular environment.

Verifying every step of the way

When performing the upgrade, we strongly advise to verify whether things are working every step of the way. It is pointless to continue the upgrade if halfway 1 of the services fail to start. In general, we can give you the following tips that apply to every service when performing (re)starts after an upgrade:

Check whether the new docker image version has been pulled successfully
Check whether the container actually starts and is at least up for > 30 seconds, and not in "Restarting" mode

There are also verification steps that depend on the service which is being upgraded. Those steps can be found in the upgrade docs itself.

Performing the upgrade

Step 1 - Upgrade Broker to 6.3.0

Step 1a - Override configurations, restart brokers

This upgrade must be done broker by broker in a rolling fashion

Override the inter.broker.protocol.version and log.message.format.version properties and literally give them the values CURRENT_KAFKA_VERSION and CURRENT_MESSAGE_FORMAT_VERSION respectively:

File platform-config/clusters/{cluster-name}/broker.sh
```
INTER_BROKER_PROTOCOL_VERSION=CURRENT_KAFKA_VERSION
LOG_MESSAGE_FORMAT_VERSION=CURRENT_MESSAGE_FORMAT_VERSION
```

Restart the broker after modifying the configuration. Enter the following command:

axual.sh -v restart cluster broker

Wait for the broker to restart, and the under replicated partition count on the Broker stats per node dashboard to go back to zero, before you continue with the next broker.

The following output is expected (for a 3 cluster broker):

Axual Platform 2021.3
Loading cluster definition: [Cluster A]
PREFIX for loading cluster definition is: CLUSTER_[Tenant]_[Cluster A]_
Loading cluster definition: [Cluster B]
PREFIX for loading cluster definition is: CLUSTER_[Tenant]_[Cluster B]_
Loading cluster definition: [Cluster C]
PREFIX for loading cluster definition is: CLUSTER_[Tenant]_[Cluster C]_
Analyzing cluster [Cluster B]-inter-broker-listener
Analyzing cluster [Cluster B]-mgmt-api-db
Analyzing cluster [Cluster B]
Analyzing cluster [Cluster A]-inter-broker-listener
Analyzing cluster [Cluster A]
Analyzing cluster [Cluster C]-inter-broker-listener
Analyzing cluster [Cluster C]
Hostname: '[Worker Hostname]'. Using configuration for cluster '[Cluster C]', node ID: '1'
Loading tenant definition: axual
Loading tenant definition: [Tenant]
Loading instance definition: [Tenant]-[Instance A]
Loading instance definition: [Tenant]-[Instance B]
Stopping cluster services for node [Worker Hostname] in cluster [Cluster C]
...
Stopped
Loading cluster definition: [Cluster A]
PREFIX for loading cluster definition is: CLUSTER_[Tenant]_[Cluster A]_
Loading cluster definition: [Cluster B]
PREFIX for loading cluster definition is: CLUSTER_[Tenant]_[Cluster B]_
Loading cluster definition: [Cluster C]
PREFIX for loading cluster definition is: CLUSTER_[Tenant]_[Cluster C]_
Analyzing cluster [Cluster B]-inter-broker-listener
Analyzing cluster [Cluster B]-mgmt-api-db
Analyzing cluster [Cluster B]
Analyzing cluster [Cluster A]-inter-broker-listener
Analyzing cluster [Cluster A]
Analyzing cluster [Cluster C]-inter-broker-listener
Analyzing cluster [Cluster C]
Hostname: '[Worker Hostname]'. Using configuration for cluster '[Cluster C]', node ID: '1'
Loading tenant definition: axual
Loading tenant definition: [Tenant]
Loading instance definition: [Tenant]-[Instance A]
Loading instance definition: [Tenant]-[Instance B]
Configuring cluster services for node [Worker Hostname] in cluster [Cluster C]
...
Preparing broker: Done
Param 1: run
Param 2: broker
Param 3: axual/broker:6.3.0
Param 4: Starting broker
Param 5: Done
Param 6: -d --tty=false --restart=always -v /appl/kafka/config/broker:/config/broker ...
Param 7:
...
Done
---

Repeat step 1 and 2 for the other brokers in the cluster

Step 1b - Upgrade inter broker protocol version

Make sure the cluster is functioning properly before you continue with this step. Check the monitoring dashboards and make sure the logs are not showing any errors.

Override the inter.broker.protocol.version and give it the value 2.8

File platform-config/clusters/{cluster-name}/broker.sh
```
INTER_BROKER_PROTOCOL_VERSION=2.8
```
Restart the broker after modifying the configuration. Enter the following command:
```
axual.sh -v restart cluster broker
```
Wait for the broker to restart, and the under replicated partition count on the Broker stats per node dashboard to go back to zero, before you continue with the next broker.
Repeat steps 1 and 2 for the remaining nodes in the cluster

Step 1c - Upgrade log message format version

Make sure the cluster is functioning properly before you continue with this step. Check the monitoring dashboards and make sure the logs are not showing any errors.

Override the log.message.format.version and give it the value 2.8

File platform-config/clusters/{cluster-name}/broker.sh
```
LOG_MESSAGE_FORMAT_VERSION=2.8
```
Restart the broker after modifying the configuration. Enter the following command:
```
axual.sh -v restart cluster broker
```
Wait for the broker to restart, and the under replicated partition count on the Broker stats per node dashboard to go back to zero, before you continue with the next broker.
Repeat steps 1 and 2 for the remaining nodes in the cluster

Step 2 - Upgrade Cluster API to 2.1.1

While the cluster API is not available, topic apply is not possible.

If you want to enable other TLS protocol versions, follow Enabling/Disabling TLS Protocols

Restart the service:

axual.sh restart cluster cluster-api

Confirm that the service restarts successfully by looking at the command output. It should end with:

Starting cluster-api: Done

Also, check the docker logs and make sure there is no error and service is up:

docker logs -f cluster-api

Step 3 - Upgrade Cluster Browse to 1.3.0

Restart the service

axual.sh restart cluster cluster-browse

Confirm that the service restarts successfully by looking at the command output. It should end with:

Starting cluster-browse: Done

Step 4 - Upgrade Instance API to 3.3.0

Restart the service

axual.sh restart instance <instance-name> instance-api

Confirm that the service restarts successfully by looking at the command output. It should end with:

Starting [INSTANCE NAME]-instance-api: Done

Step 5 - Upgrade Stream Browse to 1.2.1

Be sure that platform-config/clusters/{cluster-name}/stream-browse.sh has STREAM_BROWSE_CLIENT_SSL_KEY_PASSWORD variable configured with the client keystore key password.

Restart the service:

axual.sh restart mgmt stream-browse

Step 6 - Upgrade REST Proxy to 1.5.0

Restart the service

axual.sh restart instance <instance-name> rest-proxy

Confirm that the service restarts successfully by looking at the command output. It should end with:

Starting <instance-name>-rest-proxy: Done

Step 7 - Upgrade Schema-Registry to 5.3.0

In the below steps, we are going to set up Schema-Registry to use the 5.3.0 version.

Step 7a - Restarting Schema-Registry slave

Run the following command for each instance where schema-registry slave is running:
```
axual.sh restart instance <instance-name> sr-slave
```
Confirm that the service restarts successfully by looking at the command output. It should end with:
```
Starting [Instance Name]-sr-slave: Done
```
Verification after Schema-Registry slave restart - Don’t continue before all the following criteria met:
1. Verifying every step of the way
2. Check the docker logs and make sure there is no error and service is up.
  docker logs -f <instance-name>-sr-slave

Step 7b - Restarting Schema-Registry master

Run the following command for each instance where schema-registry master is running:
```
axual.sh restart instance <instance-name> sr-master
```
Confirm that the service restarts successfully by looking at the command output. It should end with:
```
Starting [Instance Name]-sr-master: Done
```
Verification after Schema-Registry master restart - Don’t continue before all the following criteria met:
1. Verifying every step of the way
2. Check the docker logs and make sure there is no error and service is up.
  docker logs -f <instance-name>-sr-master

Step 8 - Upgrade Discovery API to 2.4.0

While Discovery API is not available, client reconfiguration will not happen.

Update Configuration

As part of a change in Discovery API, the SSL configuration names have changed. If you are using SSL authentication for Discovery API, please revisit your configuration. Check the table below for the old and new versions.

DISCOVERYAPI_SERVER_SSL_CLIENT_AUTH Configuration Mapping

Previously

From 2021.3

false (default)

none (default)

true

need
Restart the service.
```
axual.sh restart instance <instance-name> discovery-api
```
Confirm that the service restarts successfully by looking at the command output. It should end with:
```
Starting [Instance Name]-discovery-api: Done
```

Step 9 - Upgrade Operation Manager to 1.2.0

Restart the service

axual.sh restart mgmt operation-manager

Confirm that the service restarts successfully by looking at the command output. It should end with:

Starting operation-manager: Done

Step 10 - Upgrade Mgmt API to 6.8.1

Be sure to complete this step update before running Management UI 6.1.0

Restart the service:

axual.sh restart mgmt mgmt-api

Confirm that the service restarts successfully by looking at the command output. It should end with:

Starting mgmt-api: Done

Step 11 - Upgrade Mgmt UI to 6.1.0

Restart the service:

axual.sh restart mgmt mgmt-ui

Confirm that the service restarts successfully by looking at the command output. It should end with:

Starting mgmt-ui: Done

That’s it

No other steps are required to upgrade or configure the platform components. You can read through the release blog to find out what has changed since the last release and forward it to your colleagues.