Performing the upgrade using the CLI

Typical upgrade steps

A lot of the upgrade steps can be performed without impact on your users. Basically, the deployment or upgrade of components is split in two actions:

  1. Configuration changes, such as added or changed configuration parameters, including the new component’s version

  2. Deployment of the upgraded component, by (re)starting

The configuration changes can be done in advance most of the time, limiting downtime for your end users.

In the following upgrade steps, platform-config refers to the location where your platform configuration is stored for that particular environment.

Verifying every step of the way

When performing the upgrade, we strongly advise verifying whether things are working every step of the way. It is pointless to continue the upgrade if halfway 1 of the services fail to start. In general, we can give you the following tips that apply to every service when performing (re)starts after an upgrade:

  • Check whether the new docker image version has been pulled successfully

  • Check whether the container actually starts and is at least up for > 30 seconds, and not in "Restarting" mode

There are also verification steps that depend on the service which is being upgraded. Those steps can be found in the upgrade docs itself.

Performing the upgrade using the CLI

Step 1 - Upgrade Broker to 6.4.0

In the below steps, we are going to set up Broker to use the 6.4.0 version.

Step 1a - Configuring Broker (Optional)

Change INTER_BROKER_PROTOCOL_VERSION only when all brokers have been upgraded to 6.4.0 version.

Change LOG_MESSAGE_FORMAT_VERSION only when all clients have been upgraded to support Kafka 3.2.

With broker version 6.4.0, kafka version 3.2.0 has been introduced. Follow the Updating the protocol versions in Broker (Axual CLI) section to make the optional configuration upgrades.

Step 1b - Restarting Broker

When restarting the brokers, please use a rolling restart approach
  1. Run the following command for each cluster on the machines where Broker is running:

    axual.sh restart cluster broker
  2. Confirm that the Broker restarts successfully by looking at the command output. It should end with:

    Starting broker: Done
  3. Verification after Broker restart - Don’t continue before all the following criteria met:

    1. Verifying every step of the way

    2. Check the docker logs of the Broker and make sure there are no errors.

      docker logs -f broker
    3. Check that the UnderReplicatedPartition count is 0

  4. Proceed to the next Broker

Step 2 - Upgrade Cluster API to 2.2.4

In the below steps, we are going to set up Cluster API to use the 2.2.4 version.

Step 2a - Configuring Cluster API

No additional configuration needs to be passed for running Cluster API 2.2.4 version.

Step 2b - Restarting Cluster API

  1. Run the following command for each cluster where Cluster API is running:

    axual.sh restart cluster cluster-api
  2. Confirm that the service restarts successfully by looking at the command output. It should end with:

    Starting cluster-api: Done
  3. Verification after Cluster API restart - Don’t continue before all the following criteria met:

    1. Verifying every step of the way

    2. Check the docker logs and make sure there is no error and service is up.

      docker logs -f cluster-api

Step 3 - Upgrade Cluster Browse to 1.4.0

In the below steps, we are going to set up Cluster Browse to use the 1.4.0 version.

Step 3a - Configuring Cluster Browse

No additional configuration needs to be passed for running Cluster Browse 1.4.0 version.

Step 3b - Restarting Cluster Browse

  1. Run the following command for each cluster where Cluster Browse is running:

    axual.sh restart cluster cluster-browse
  2. Confirm that the service restarts successfully by looking at the command output. It should end with:

    Starting cluster-browse: Done
  3. Verification after Cluster Browse restart - Don’t continue before all the following criteria met:

    1. Verifying every step of the way

    2. Check the docker logs and make sure there is no error and service is up.

      docker logs -f cluster-browse

Step 4 - Upgrade Discovery API to 2.6.4

In the below steps, we are going to set up Discovery API to use the 2.6.4 version.

Step 4a - Configuring Discovery API

No additional configuration needs to be passed for running Discovery API 2.6.4 version.

Step 4b - Restarting Discovery API

When restarting the Discovery API service, please use a rolling restart approach
  1. Run the following command for each instance on the machines that should run Discovery API:

    axual.sh restart instance <instance-name> discovery-api
  2. Confirm that the service restarts successfully by looking at the command output. It should end with:

    Starting [Instance Name]-discovery-api: Done
  3. Verification after Discovery API restart - Don’t continue before all the following criteria met:

    1. Verifying every step of the way

    2. Check the docker logs and make sure there is no error and service is up.

      docker logs -f <instance-name>-discovery-api

Step 5 - Upgrade Schema Registry to 5.4.1

In the below steps, we are going to set up Schema-Registry to use the 5.4.1 version.

Step 5a - Restarting Schema Registry Slave

When restarting the Schema Registry Slave service, please use a rolling restart approach
  1. Run the following command for each instance where schema-registry slave is running:

    axual.sh restart instance <instance-name> sr-slave
  2. Confirm that the service restarts successfully by looking at the command output. It should end with:

    Starting [Instance Name]-sr-slave: Done
  3. Verification after Schema-Registry slave restart - Don’t continue before all the following criteria met:

    1. Verifying every step of the way

    2. Check the docker logs and make sure there is no error and service is up.

      docker logs -f <instance-name>-sr-slave

Step 5b - Restarting Schema Registry Master

  1. Run the following command for each instance where schema-registry master is running:

    axual.sh restart instance <instance-name> sr-master
  2. Confirm that the service restarts successfully by looking at the command output. It should end with:

    Starting [Instance Name]-sr-master: Done
  3. Verification after Schema-Registry master restart - Don’t continue before all the following criteria met:

    1. Verifying every step of the way

    2. Check the docker logs and make sure there is no error and service is up.

      docker logs -f <instance-name>-sr-master

Step 6 - Upgrade Instance API to 3.4.3

In the below steps, we are going to set up Instance API to use the 3.4.3 version.

Step 6a - Configuring Instance API

No additional configuration needs to be passed for running Instance API 3.4.3 version.

Step 6b - Restarting Instance API

  1. Run the following command on the machine that should run Instance API:

    axual.sh restart instance <instance-name> instance-api
  2. Confirm that the service restarts successfully by looking at the command output. It should end with:

    Starting [Instance Name]-instance-api: Done
  3. Verification after Instance API restart - Don’t continue before all the following criteria met:

    1. Verifying every step of the way

    2. Check the docker logs and make sure there is no error and service is up.

      docker logs -f <instance-name>-instance-api

Step 7 - Upgrade Management API to 6.12.1

In the below steps, we are going to set up Management API to use the 6.12.1 version.

Step 7a - Configuring Management API

No additional configuration needs to be passed for running Management API 6.12.1 version.

Step 7b - Restarting Management API

During the Management API upgrade, the Management Service Portal will not be accessible, but applications will still be able to consume and produce messages
  1. Run the following command on the machine that should run Management API:

    axual.sh restart mgmt mgmt-api
  2. Confirm that the service restarts successfully by looking at the command output. It should end with:

    Starting mgmt-api: Done
  3. Verification after Management API restart - Don’t continue before all the following criteria met:

    1. Verifying every step of the way

    2. Check the docker logs and make sure there is no error and service is up.

      docker logs -f mgmt-api

Step 8 - Upgrade Management UI to 6.5.0

In the below steps, we are going to set up Management UI to use the 6.5.0 version.

Step 8a - Configuring Management UI

No additional configuration needs to be passed for running Management UI 6.5.0 version.

Step 8b - Restarting Management UI

During the Management UI upgrade, the Management Service Portal will not be accessible, but applications will still be able to consume and produce messages
  1. Run the following command on the machine that should run Management UI:

axual.sh restart mgmt mgmt-ui
  1. Confirm that the service restarts successfully by looking at the command output. It should end with:

    Starting mgmt-ui: Done
  2. Verification after Management UI restart - Don’t continue before all the following criteria met:

    1. Verifying every step of the way

    2. Check the docker logs and make sure there is no error and service is up.

      docker logs -f mgmt-ui

Step 9 - Upgrade Operation Manager to 1.3.0

In the below steps, we are going to set up Operation Manager to use the 1.3.0 version.

Step 9a - Configuring Operation Manager

No additional configuration needs to be passed for running Operation Manager 1.3.0 version.

Step 9b - Restarting Operation Manager

  1. Run the following command on the machine that should run Operation Manager:

    axual.sh restart mgmt operation-manager
  2. Confirm that the service restarts successfully by looking at the command output. It should end with:

    Starting operation-manager: Done
  3. Verification after Operation Manager restart - Don’t continue before all the following criteria met:

    1. Verifying every step of the way

    2. Check the docker logs and make sure there is no error and service is up.

      docker logs -f operation-manager

Step 10 - Upgrade Stream Browse to 1.3.0

In the below steps, we are going to set up Stream Browse to use the 1.3.0 version.

Step 10a - Configuring Stream Browse

No additional configuration needs to be passed for running Stream Browse 1.3.0 version.

Step 10b - Restarting Stream Browse

  1. Run the following command on the machine that should run Stream Browse:

    axual.sh restart mgmt stream-browse
  2. Confirm that the service restarts successfully by looking at the command output. It should end with:

    Starting stream-browse: Done
  3. Verification after Stream Browse restart - Don’t continue before all the following criteria met:

    1. Verifying every step of the way

    2. Check the docker logs and make sure there is no error and service is up.

      docker logs -f stream-browse

Step 11 - Upgrade Axual-Connect to 2.3.2

In the below steps, we are going to set up Axual-Connect to use the 2.3.2 version.

Step 11a - Configuring Axual-Connect

No additional configuration needs to be passed for running Axual-Connect 2.3.2 version.

Step 11b - Restarting Axual-Connect

  1. Run the following command on the machine that should run Axual-Connect:

    axual.sh restart client <instance-name> axual-connect
  2. Confirm that the service restarts successfully by looking at the command output. It should end with:

    Starting axual-connect: Done
  3. Verification after Axual-Connect restart - Don’t continue before all the following criteria met:

    1. Verifying every step of the way

    2. Check the docker logs and make sure there is no error and service is up.

      docker logs -f <instance-name>-axual-connect
    3. Check individual connector status by accessing curl -k -u user:pass https://host:port/connectors?expand=status or by checking the individual connector logs on the directory ${LOG_DIR}/axual-connect/connectors/connector_name/.

Step 12 - Upgrade REST Proxy to 1.5.1

Restart the service

axual.sh restart client <instance-name> rest-proxy

Confirm that the service restarts successfully by looking at the command output. It should end with:

Starting <instance-name>-rest-proxy: Done

Step 13 - Upgrade to Keycloak 18.0.1

During Keycloak upgrade the Management Service Portal will not be accessible, but applications will still be able to consume and produce message

In this step we are going to set up Keycloak to use the 18.0.1 version.

Consider that the default version of Keycloak is configured in versions.sh file as 18.0.1 and no need to override it in platform-config/clusters/{cluster-name}/keycloak.sh. Also, there is no change in existing configurations.

Step 13a - Stop Keycloak

Log in to the node where Keycloak is running and stop Keycloak with the following command:

axual.sh stop mgmt mgmt-keycloak

Step 13b - Run MariaDB Scripts (Optional)

If you’re running a MariaDB Server version prior to 10.3.11, you’ll need to run the following script before starting the Keycloak 18 container/pod:

-- Disable foreign key check
SET FOREIGN_KEY_CHECKS=0;

-- KC 12 - 12.1.0-add-realm-localization-table
CREATE TABLE REALM_LOCALIZATIONS
(
    REALM_ID varchar(255) not null,
    LOCALE   varchar(255) not null,
    TEXTS    longtext     not null,
    primary key (REALM_ID, LOCALE)
);

SELECT max(ORDEREXECUTED) INTO @max_order_executed FROM DATABASECHANGELOG;

INSERT INTO DATABASECHANGELOG (ID, AUTHOR, FILENAME, DATEEXECUTED, ORDEREXECUTED,
  EXECTYPE, MD5SUM, DESCRIPTION, COMMENTS, TAG, LIQUIBASE, CONTEXTS, LABELS, DEPLOYMENT_ID)
VALUES ('12.1.0-add-realm-localization-table', 'keycloak', 'META-INF/jpa-changelog-12.0.0.xml', now(), @max_order_executed + 1, 'EXECUTED',
  '8:babadb686aab7b56562817e60bf0abd0', 'createTable tableName=REALM_LOCALIZATIONS; addPrimaryKey tableName=REALM_LOCALIZATIONS','', null, '4.8.0', null, null, '5988138086');

-- KC 13 - 13.0.0-increase-column-size-federated
ALTER TABLE CLIENT_SCOPE_CLIENT MODIFY COLUMN CLIENT_ID VARCHAR(255);
ALTER TABLE CLIENT_SCOPE_CLIENT MODIFY COLUMN SCOPE_ID VARCHAR(255);

SELECT max(ORDEREXECUTED) INTO @max_order_executed FROM DATABASECHANGELOG;

INSERT INTO DATABASECHANGELOG (ID, AUTHOR, FILENAME, DATEEXECUTED, ORDEREXECUTED,
  EXECTYPE, MD5SUM, DESCRIPTION, COMMENTS, TAG, LIQUIBASE, CONTEXTS, LABELS, DEPLOYMENT_ID)
VALUES ('13.0.0-increase-column-size-federated', 'keycloak', 'META-INF/jpa-changelog-13.0.0.xml', now(), @max_order_executed + 1, 'EXECUTED',
  '8:9d11b619db2ae27c25853b8a37cd0dea', 'modifyDataType columnName=CLIENT_ID, tableName=CLIENT_SCOPE_CLIENT; modifyDataType columnName=SCOPE_ID, tableName=CLIENT_SCOPE_CLIENT', '', null, '4.8.0', null, null, '5988138086');

-- KC 13 - json-string-accomodation-fixed
ALTER TABLE REALM_ATTRIBUTE ADD VALUE_NEW LONGTEXT;
UPDATE REALM_ATTRIBUTE SET VALUE_NEW = VALUE;
ALTER TABLE REALM_ATTRIBUTE DROP COLUMN VALUE;
ALTER TABLE REALM_ATTRIBUTE CHANGE COLUMN VALUE_NEW VALUE LONGTEXT;

SELECT max(ORDEREXECUTED) INTO @max_order_executed FROM DATABASECHANGELOG;

INSERT INTO DATABASECHANGELOG (ID, AUTHOR, FILENAME, DATEEXECUTED, ORDEREXECUTED,
  EXECTYPE, MD5SUM, DESCRIPTION, COMMENTS, TAG, LIQUIBASE, CONTEXTS, LABELS, DEPLOYMENT_ID)
VALUES ('json-string-accomodation-fixed', 'keycloak', 'META-INF/jpa-changelog-13.0.0.xml', now(), @max_order_executed + 1, 'EXECUTED',
  '8:dfbee0d6237a23ef4ccbb7a4e063c163', 'addColumn tableName=REALM_ATTRIBUTE; update tableName=REALM_ATTRIBUTE; dropColumn columnName=VALUE, tableName=REALM_ATTRIBUTE; renameColumn newColumnName=VALUE, oldColumnName=VALUE_NEW, tableName=REALM_ATTRIBUTE', '', null, '4.8.0', null, null, '5988138086');

-- KC 15 - 15.0.0-KEYCLOAK-18467
ALTER TABLE REALM_LOCALIZATIONS ADD TEXTS_NEW LONGTEXT;
UPDATE REALM_LOCALIZATIONS SET TEXTS_NEW = TEXTS;
ALTER TABLE REALM_LOCALIZATIONS DROP COLUMN TEXTS;
ALTER TABLE REALM_LOCALIZATIONS CHANGE COLUMN TEXTS_NEW TEXTS LONGTEXT NOT NULL;

SELECT max(ORDEREXECUTED) INTO @max_order_executed FROM DATABASECHANGELOG;

INSERT INTO DATABASECHANGELOG (ID, AUTHOR, FILENAME, DATEEXECUTED, ORDEREXECUTED,
  EXECTYPE, MD5SUM, DESCRIPTION, COMMENTS, TAG, LIQUIBASE, CONTEXTS, LABELS, DEPLOYMENT_ID)
VALUES ('15.0.0-KEYCLOAK-18467', 'keycloak', 'META-INF/jpa-changelog-15.0.0.xml', now(), @max_order_executed + 1, 'EXECUTED',
  '8:ba8ee3b694d043f2bfc1a1079d0760d7', 'addColumn tableName=REALM_LOCALIZATIONS; update tableName=REALM_LOCALIZATIONS; dropColumn columnName=TEXTS, tableName=REALM_LOCALIZATIONS; renameColumn newColumnName=TEXTS, oldColumnName=TEXTS_NEW, tableName=REALM_LOCALIZATIONS; addNotNullConstrai...', '', null, '4.8.0', null, null, '5988138086');

-- Enable foreign key check
SET FOREIGN_KEY_CHECKS=1;

Step 13c - Start Keycloak

Log in to the node where Keycloak is running and restart Keycloak with the following command:

axual.sh start mgmt mgmt-keycloak

After the starting command complete, make sure Keycloak is up and running by:

  • checking logs

  • accessing the Admin Console

  • accessing the Management Service Portal

Step 13d - Access Admin Console on the management port

In the new version of Keycloak the separate port for admin console is eliminated, and you can access it directly as follows:

  1. Log in to Keycloak Admin Console, using the UI, go to https://KEYCLOAK_HOSTNAME:KEYCLOAK_PORT/auth. You will see the following login screen.

  2. Enter the KEYCLOAK_USER and KEYCLOAK_PASSWORD stored on your platform-config, then press login.

    Keycloak Admin login screen

Rollback of Keycloak

Check the Rollback page Rollback Keycloak in case of facing any problem during upgrade steps.

Aftercare for Keycloak upgrade

After confirming the upgrade was successful and that Keycloak 18 is up and running, you may remove axual-keycloak-theme

In Keycloak 18, theme is part of the custom image and there is no need to replace the theme folder and its configuration.

That’s it

No other steps are required to upgrade or configure the platform components. You can read through the release blog to find out what has changed since the last release and forward it to your colleagues.