Performing the upgrade using the CLI
Typical upgrade steps
A lot of the upgrade steps can be performed without impact on your users. Basically, the deployment or upgrade of components is split in two actions:
-
Configuration changes, such as added or changed configuration parameters, including the new component’s version
-
Deployment of the upgraded component, by (re)starting
The configuration changes can be done in advance most of the time, limiting downtime for your end users.
In the following upgrade steps, platform-config refers to the location where your platform configuration is stored for that particular environment.
|
Verifying every step of the way
When performing the upgrade, we strongly advise verifying whether things are working every step of the way. It is pointless to continue the upgrade if halfway 1 of the services fail to start. In general, we can give you the following tips that apply to every service when performing (re)starts after an upgrade:
-
Check whether the new docker image version has been pulled successfully
-
Check whether the container actually starts and is at least up for > 30 seconds, and not in "Restarting" mode
There are also verification steps that depend on the service which is being upgraded. Those steps can be found in the upgrade docs itself. |
Performing the upgrade using the CLI
Step 1 - Upgrade Broker to 6.4.0
In the below steps, we are going to set up Broker to use the 6.4.0 version.
Step 1a - Configuring Broker (Optional)
Change Change |
With broker version 6.4.0, kafka version 3.2.0 has been introduced. Follow the Updating the protocol versions in Broker (Axual CLI) section to make the optional configuration upgrades.
Step 1b - Restarting Broker
When restarting the brokers, please use a rolling restart approach |
-
Run the following command for each cluster on the machines where Broker is running:
axual.sh restart cluster broker
-
Confirm that the Broker restarts successfully by looking at the command output. It should end with:
Starting broker: Done
-
Verification after Broker restart - Don’t continue before all the following criteria met:
-
Check the docker logs of the Broker and make sure there are no errors.
docker logs -f broker
-
Check that the UnderReplicatedPartition count is 0
-
Proceed to the next Broker
Step 2 - Upgrade Cluster API to 2.2.4
In the below steps, we are going to set up Cluster API to use the 2.2.4 version.
Step 2a - Configuring Cluster API
No additional configuration needs to be passed for running Cluster API 2.2.4 version.
Step 2b - Restarting Cluster API
-
Run the following command for each cluster where Cluster API is running:
axual.sh restart cluster cluster-api
-
Confirm that the service restarts successfully by looking at the command output. It should end with:
Starting cluster-api: Done
-
Verification after Cluster API restart - Don’t continue before all the following criteria met:
-
Check the docker logs and make sure there is no error and service is up.
docker logs -f cluster-api
Step 3 - Upgrade Cluster Browse to 1.4.0
In the below steps, we are going to set up Cluster Browse to use the 1.4.0 version.
Step 3a - Configuring Cluster Browse
No additional configuration needs to be passed for running Cluster Browse 1.4.0 version.
Step 3b - Restarting Cluster Browse
-
Run the following command for each cluster where Cluster Browse is running:
axual.sh restart cluster cluster-browse
-
Confirm that the service restarts successfully by looking at the command output. It should end with:
Starting cluster-browse: Done
-
Verification after Cluster Browse restart - Don’t continue before all the following criteria met:
-
Check the docker logs and make sure there is no error and service is up.
docker logs -f cluster-browse
Step 4 - Upgrade Discovery API to 2.6.4
In the below steps, we are going to set up Discovery API to use the 2.6.4 version.
Step 4a - Configuring Discovery API
No additional configuration needs to be passed for running Discovery API 2.6.4 version.
Step 4b - Restarting Discovery API
When restarting the Discovery API service, please use a rolling restart approach |
-
Run the following command for each instance on the machines that should run Discovery API:
axual.sh restart instance <instance-name> discovery-api
-
Confirm that the service restarts successfully by looking at the command output. It should end with:
Starting [Instance Name]-discovery-api: Done
-
Verification after Discovery API restart - Don’t continue before all the following criteria met:
-
Check the docker logs and make sure there is no error and service is up.
docker logs -f <instance-name>-discovery-api
Step 5 - Upgrade Schema Registry to 5.4.1
In the below steps, we are going to set up Schema-Registry to use the 5.4.1 version.
Step 5a - Restarting Schema Registry Slave
When restarting the Schema Registry Slave service, please use a rolling restart approach |
-
Run the following command for each instance where schema-registry slave is running:
axual.sh restart instance <instance-name> sr-slave
-
Confirm that the service restarts successfully by looking at the command output. It should end with:
Starting [Instance Name]-sr-slave: Done
-
Verification after Schema-Registry slave restart - Don’t continue before all the following criteria met:
-
Check the docker logs and make sure there is no error and service is up.
docker logs -f <instance-name>-sr-slave
Step 5b - Restarting Schema Registry Master
-
Run the following command for each instance where schema-registry master is running:
axual.sh restart instance <instance-name> sr-master
-
Confirm that the service restarts successfully by looking at the command output. It should end with:
Starting [Instance Name]-sr-master: Done
-
Verification after Schema-Registry master restart - Don’t continue before all the following criteria met:
-
Check the docker logs and make sure there is no error and service is up.
docker logs -f <instance-name>-sr-master
Step 6 - Upgrade Instance API to 3.4.3
In the below steps, we are going to set up Instance API to use the 3.4.3 version.
Step 6a - Configuring Instance API
No additional configuration needs to be passed for running Instance API 3.4.3 version.
Step 6b - Restarting Instance API
-
Run the following command on the machine that should run Instance API:
axual.sh restart instance <instance-name> instance-api
-
Confirm that the service restarts successfully by looking at the command output. It should end with:
Starting [Instance Name]-instance-api: Done
-
Verification after Instance API restart - Don’t continue before all the following criteria met:
-
Check the docker logs and make sure there is no error and service is up.
docker logs -f <instance-name>-instance-api
Step 7 - Upgrade Management API to 6.12.1
In the below steps, we are going to set up Management API to use the 6.12.1 version.
Step 7a - Configuring Management API
No additional configuration needs to be passed for running Management API 6.12.1 version.
Step 7b - Restarting Management API
During the Management API upgrade, the Management Service Portal will not be accessible, but applications will still be able to consume and produce messages |
-
Run the following command on the machine that should run Management API:
axual.sh restart mgmt mgmt-api
-
Confirm that the service restarts successfully by looking at the command output. It should end with:
Starting mgmt-api: Done
-
Verification after Management API restart - Don’t continue before all the following criteria met:
-
Check the docker logs and make sure there is no error and service is up.
docker logs -f mgmt-api
Step 8 - Upgrade Management UI to 6.5.0
In the below steps, we are going to set up Management UI to use the 6.5.0 version.
Step 8a - Configuring Management UI
No additional configuration needs to be passed for running Management UI 6.5.0 version.
Step 8b - Restarting Management UI
During the Management UI upgrade, the Management Service Portal will not be accessible, but applications will still be able to consume and produce messages |
-
Run the following command on the machine that should run Management UI:
axual.sh restart mgmt mgmt-ui
-
Confirm that the service restarts successfully by looking at the command output. It should end with:
Starting mgmt-ui: Done
-
Verification after Management UI restart - Don’t continue before all the following criteria met:
-
Check the docker logs and make sure there is no error and service is up.
docker logs -f mgmt-ui
Step 9 - Upgrade Operation Manager to 1.3.0
In the below steps, we are going to set up Operation Manager to use the 1.3.0 version.
Step 9a - Configuring Operation Manager
No additional configuration needs to be passed for running Operation Manager 1.3.0 version.
Step 9b - Restarting Operation Manager
-
Run the following command on the machine that should run Operation Manager:
axual.sh restart mgmt operation-manager
-
Confirm that the service restarts successfully by looking at the command output. It should end with:
Starting operation-manager: Done
-
Verification after Operation Manager restart - Don’t continue before all the following criteria met:
-
Check the docker logs and make sure there is no error and service is up.
docker logs -f operation-manager
Step 10 - Upgrade Stream Browse to 1.3.0
In the below steps, we are going to set up Stream Browse to use the 1.3.0 version.
Step 10a - Configuring Stream Browse
No additional configuration needs to be passed for running Stream Browse 1.3.0 version.
Step 10b - Restarting Stream Browse
-
Run the following command on the machine that should run Stream Browse:
axual.sh restart mgmt stream-browse
-
Confirm that the service restarts successfully by looking at the command output. It should end with:
Starting stream-browse: Done
-
Verification after Stream Browse restart - Don’t continue before all the following criteria met:
-
Check the docker logs and make sure there is no error and service is up.
docker logs -f stream-browse
Step 11 - Upgrade Axual-Connect to 2.3.2
In the below steps, we are going to set up Axual-Connect to use the 2.3.2 version.
Step 11a - Configuring Axual-Connect
No additional configuration needs to be passed for running Axual-Connect 2.3.2 version.
Step 11b - Restarting Axual-Connect
-
Run the following command on the machine that should run Axual-Connect:
axual.sh restart client <instance-name> axual-connect
-
Confirm that the service restarts successfully by looking at the command output. It should end with:
Starting axual-connect: Done
-
Verification after Axual-Connect restart - Don’t continue before all the following criteria met:
-
Check the docker logs and make sure there is no error and service is up.
docker logs -f <instance-name>-axual-connect
-
Check individual connector status by accessing
curl -k -u user:pass https://host:port/connectors?expand=status
or by checking the individual connector logs on the directory${LOG_DIR}/axual-connect/connectors/connector_name/
.
Step 12 - Upgrade REST Proxy to 1.5.1
Restart the service
axual.sh restart client <instance-name> rest-proxy
Confirm that the service restarts successfully by looking at the command output. It should end with:
Starting <instance-name>-rest-proxy: Done
Step 13 - Upgrade to Keycloak 18.0.1
During Keycloak upgrade the Management Service Portal will not be accessible, but applications will still be able to consume and produce message |
In this step we are going to set up Keycloak to use the 18.0.1 version.
Consider that the default version of Keycloak is configured in versions.sh file as 18.0.1 and no
need to override it in platform-config/clusters/{cluster-name}/keycloak.sh
. Also, there is no change in
existing configurations.
Step 13a - Stop Keycloak
Log in to the node where Keycloak is running and stop Keycloak with the following command:
axual.sh stop mgmt mgmt-keycloak
Step 13b - Run MariaDB Scripts (Optional)
If you’re running a MariaDB Server version prior to 10.3.11, you’ll need to run the following script before starting the Keycloak 18 container/pod:
-- Disable foreign key check
SET FOREIGN_KEY_CHECKS=0;
-- KC 12 - 12.1.0-add-realm-localization-table
CREATE TABLE REALM_LOCALIZATIONS
(
REALM_ID varchar(255) not null,
LOCALE varchar(255) not null,
TEXTS longtext not null,
primary key (REALM_ID, LOCALE)
);
SELECT max(ORDEREXECUTED) INTO @max_order_executed FROM DATABASECHANGELOG;
INSERT INTO DATABASECHANGELOG (ID, AUTHOR, FILENAME, DATEEXECUTED, ORDEREXECUTED,
EXECTYPE, MD5SUM, DESCRIPTION, COMMENTS, TAG, LIQUIBASE, CONTEXTS, LABELS, DEPLOYMENT_ID)
VALUES ('12.1.0-add-realm-localization-table', 'keycloak', 'META-INF/jpa-changelog-12.0.0.xml', now(), @max_order_executed + 1, 'EXECUTED',
'8:babadb686aab7b56562817e60bf0abd0', 'createTable tableName=REALM_LOCALIZATIONS; addPrimaryKey tableName=REALM_LOCALIZATIONS','', null, '4.8.0', null, null, '5988138086');
-- KC 13 - 13.0.0-increase-column-size-federated
ALTER TABLE CLIENT_SCOPE_CLIENT MODIFY COLUMN CLIENT_ID VARCHAR(255);
ALTER TABLE CLIENT_SCOPE_CLIENT MODIFY COLUMN SCOPE_ID VARCHAR(255);
SELECT max(ORDEREXECUTED) INTO @max_order_executed FROM DATABASECHANGELOG;
INSERT INTO DATABASECHANGELOG (ID, AUTHOR, FILENAME, DATEEXECUTED, ORDEREXECUTED,
EXECTYPE, MD5SUM, DESCRIPTION, COMMENTS, TAG, LIQUIBASE, CONTEXTS, LABELS, DEPLOYMENT_ID)
VALUES ('13.0.0-increase-column-size-federated', 'keycloak', 'META-INF/jpa-changelog-13.0.0.xml', now(), @max_order_executed + 1, 'EXECUTED',
'8:9d11b619db2ae27c25853b8a37cd0dea', 'modifyDataType columnName=CLIENT_ID, tableName=CLIENT_SCOPE_CLIENT; modifyDataType columnName=SCOPE_ID, tableName=CLIENT_SCOPE_CLIENT', '', null, '4.8.0', null, null, '5988138086');
-- KC 13 - json-string-accomodation-fixed
ALTER TABLE REALM_ATTRIBUTE ADD VALUE_NEW LONGTEXT;
UPDATE REALM_ATTRIBUTE SET VALUE_NEW = VALUE;
ALTER TABLE REALM_ATTRIBUTE DROP COLUMN VALUE;
ALTER TABLE REALM_ATTRIBUTE CHANGE COLUMN VALUE_NEW VALUE LONGTEXT;
SELECT max(ORDEREXECUTED) INTO @max_order_executed FROM DATABASECHANGELOG;
INSERT INTO DATABASECHANGELOG (ID, AUTHOR, FILENAME, DATEEXECUTED, ORDEREXECUTED,
EXECTYPE, MD5SUM, DESCRIPTION, COMMENTS, TAG, LIQUIBASE, CONTEXTS, LABELS, DEPLOYMENT_ID)
VALUES ('json-string-accomodation-fixed', 'keycloak', 'META-INF/jpa-changelog-13.0.0.xml', now(), @max_order_executed + 1, 'EXECUTED',
'8:dfbee0d6237a23ef4ccbb7a4e063c163', 'addColumn tableName=REALM_ATTRIBUTE; update tableName=REALM_ATTRIBUTE; dropColumn columnName=VALUE, tableName=REALM_ATTRIBUTE; renameColumn newColumnName=VALUE, oldColumnName=VALUE_NEW, tableName=REALM_ATTRIBUTE', '', null, '4.8.0', null, null, '5988138086');
-- KC 15 - 15.0.0-KEYCLOAK-18467
ALTER TABLE REALM_LOCALIZATIONS ADD TEXTS_NEW LONGTEXT;
UPDATE REALM_LOCALIZATIONS SET TEXTS_NEW = TEXTS;
ALTER TABLE REALM_LOCALIZATIONS DROP COLUMN TEXTS;
ALTER TABLE REALM_LOCALIZATIONS CHANGE COLUMN TEXTS_NEW TEXTS LONGTEXT NOT NULL;
SELECT max(ORDEREXECUTED) INTO @max_order_executed FROM DATABASECHANGELOG;
INSERT INTO DATABASECHANGELOG (ID, AUTHOR, FILENAME, DATEEXECUTED, ORDEREXECUTED,
EXECTYPE, MD5SUM, DESCRIPTION, COMMENTS, TAG, LIQUIBASE, CONTEXTS, LABELS, DEPLOYMENT_ID)
VALUES ('15.0.0-KEYCLOAK-18467', 'keycloak', 'META-INF/jpa-changelog-15.0.0.xml', now(), @max_order_executed + 1, 'EXECUTED',
'8:ba8ee3b694d043f2bfc1a1079d0760d7', 'addColumn tableName=REALM_LOCALIZATIONS; update tableName=REALM_LOCALIZATIONS; dropColumn columnName=TEXTS, tableName=REALM_LOCALIZATIONS; renameColumn newColumnName=TEXTS, oldColumnName=TEXTS_NEW, tableName=REALM_LOCALIZATIONS; addNotNullConstrai...', '', null, '4.8.0', null, null, '5988138086');
-- Enable foreign key check
SET FOREIGN_KEY_CHECKS=1;
Step 13c - Start Keycloak
Log in to the node where Keycloak is running and restart Keycloak with the following command:
axual.sh start mgmt mgmt-keycloak
After the starting command complete, make sure Keycloak is up and running by:
-
checking logs
-
accessing the Admin Console
-
accessing the Management Service Portal
Step 13d - Access Admin Console on the management port
In the new version of Keycloak the separate port for admin console is eliminated, and you can access it directly as follows:
-
Log in to Keycloak Admin Console, using the UI, go to https://KEYCLOAK_HOSTNAME:KEYCLOAK_PORT/auth. You will see the following login screen.
-
Enter the KEYCLOAK_USER and KEYCLOAK_PASSWORD stored on your
platform-config
, then press login.
Rollback of Keycloak
Check the Rollback page Rollback Keycloak in case of facing any problem during upgrade steps.