Keycloak Upgrade

To be able to start upgrading the Keycloak, you need to have upgraded Self-Service to the Governance charts.

Install a new Keycloak instance

Prerequisite

Your Keycloak DB is running on MariaDB 10.3.11 version or later
Your Keycloak DB is running on MySQL 8.0 version or later

In case you are running a prior version, please contact the Support.

We are going to give you advice on how to migrate your Keycloak DB to a newer version.

Objective

Update the governance values.yaml to start a new instance of Keycloak that migrates the Keycloak DB schema from v19 to v22.

Preparation

Backup Keycloak DB (data and table-schemas)

Retrieve existing Keycloak Docker Environment Variables

Example Keycloak Docker Env Variables

KC_HOSTNAME_STRICT:             [keycloak-host-strict-toggle]
KC_HTTP_ENABLED:                [keycloak-http-toggle]
KC_DB:                          [keycloak-db-provider]
KC_DB_URL_HOST:                 [keycloak-db-host]
KC_DB_URL_PORT:                 [keycloak-db-port]
KC_DB_URL_DATABASE:             [keycloak-db-name]
KC_DB_USERNAME:                 [keycloak-db-username]
KC_DB_PASSWORD:                 [keycloak-db-password]
KEYCLOAK_ADMIN:                 [keycloak-admin-username]
KEYCLOAK_ADMIN_PASSWORD:        [keycloak-admin-password]

Collect the actual values for the DB and Admin credentials, the Keycloak k8s-secrets will be removed once uninstalled the old Keycloak

Only the Keycloak Docker Environment Variables mentioned above need to be used in the new Keycloak instance

Execution

Open the values.yaml used to deploy the Management stack with the governance charts.

Enable the Keycloak deployed with Governance Charts

values.yaml

global:
  # existing endpoints should not be changed
  # ...
  keycloak:
    enabled: true

Add a new keycloak section inside the axual-governance section

Use the retrieved Keycloak Docker Environment variables as input

values.yaml

axual-governance:
  ## Keycloak
  keycloak:
    # TODO change
    imagePullSecrets:
      - name: [your-image-pull-secret]

    database:
      # TODO change
      # Old Keycloak Docker Environment Variable = `KC_DB`
      vendor: "[keycloak-db-provider]"
      # TODO change
      # Old Keycloak Docker Environment Variable = `KC_DB_URL_HOST`
      hostname: "[keycloak-db-host]"
      # TODO change
      # Old Keycloak Docker Environment Variable = `KC_DB_URL_DATABASE`
      database: "[keycloak-db-name]"
      # TODO change
      # Old Keycloak Docker Environment Variable = `KC_DB_URL_PORT`
      port: "[keycloak-db-port]"
      # TODO change
      # Old Keycloak Docker Environment Variable = `KC_DB_USERNAME`
      username: "[keycloak-db-username]"
      # TODO change
      # Old Keycloak Docker Environment Variable = `KC_DB_PASSWORD`
      password: "[keycloak-db-password]"
    extraEnv: |
      # TODO change (can be omitted since it's an existing installation)
      # Old Keycloak Docker Environment Variable = `KEYCLOAK_ADMIN`
      - name: KEYCLOAK_ADMIN
        value: "[keycloak-admin-username]"
      # TODO change (can be omitted since it's an existing installation)
      # Old Keycloak Docker Environment Variable = `KEYCLOAK_ADMIN_PASSWORD`
      - name: KEYCLOAK_ADMIN_PASSWORD
        value: "[keycloak-admin-password]"
      - name: JAVA_OPTS_APPEND
        value: -Djgroups.dns.query={{ template "keycloak.serviceDnsName" . }}
      # TODO change
      # Old Keycloak Docker Environment Variable = `KC_HTTP_ENABLED`
      - name: KC_HTTP_ENABLED
        value: "[keycloak-http-toggle]"
      # TODO change
      # Old Keycloak Docker Environment Variable = `KC_HOSTNAME_STRICT`
      - name: KC_HOSTNAME_STRICT
        value: "[keycloak-host-strict-toggle]"
    # This control access to the Keycloak Admin Console
    ingress:
      enabled: false

Upgrade the Governance stack helm application with the new values.yaml, this will deploy a new Keycloak pod next to the old one.

In case your new Keycloak needs to have additional CA added to its truststore, please follow this guide to configure the additional CA

In case your new Keycloak fails to connect with an Azure MySQL Flex Database with SSL due Certificates not conforming to algorithm constraints, follow this extra section.

Refer https://github.com/keycloak/keycloak/issues/19185 for more details.

Extra initContainer to enable sha1 policy

Update the new keycloak section inside the axual-governance section

values.yaml

axual-governance:
  ## Keycloak
  keycloak:

    extraVolumes: |
      - name: keycloak-init-realm
        configMap:
          name: governance-local-realm
      - name: themes
        emptyDir: { }
      - name: java-security
        emptyDir: {}

    extraVolumeMounts: |
      - name: keycloak-init-realm
        mountPath: "/opt/keycloak/data/import"
        readOnly: true
      - name: themes
        mountPath: /opt/keycloak/themes
      - name: java-security
        mountPath: /etc/crypto-policies/back-ends/

    extraInitContainers: |
      - name: theme-provider
        image: registry.axual.io/opensource/images/axual/keycloak-themes:0.1.1
        imagePullPolicy: IfNotPresent
        command:
          - sh
        args:
          - -c
          - |
            echo "Copying themes..."
            cp -R /axual-themes/* /themes
        volumeMounts:
          - name: themes
            mountPath: /themes
      - name: java-security-for-pg-flexible
        image: quay.io/keycloak/keycloak:22.0.4
        imagePullPolicy: IfNotPresent
        command:
          - sh
        args:
          - -c
          - |
            echo "Allowing SHA1 in crypto-policies."
            cp /etc/crypto-policies/back-ends/java.config /java-security/java.config
            cd /java-security/
            sed -i 's/SHA1,//g' java.config
        volumeMounts:
          - name: java-security
            mountPath: /java-security

Upgrade the Governance stack helm application with the new values.yaml, this will update the new Keycloak pod.

Verification

In this step, we are going to verify that the new Keycloak pod has migrated the DB schema and the old Keycloak is still working.

Check that the new Keycloak pod is in a ready state
Open a browser and access your Tenant in the Self-Service

You should be able to access the Self-Service as usual.

In case any of the above verification is failed, proceed to the Rollback Scenarios section.

Configure API Gateway to connect to the new Keycloak

Objective

Update the governance values.yaml to configure the API Gateway to redirect /auth traffic to the new Keycloak instance.

Preparation

Retrieve the new Keycloak Service Name

Execution

Open the values.yaml used to deploy the Management stack with the governance charts.

Navigate to the api-gateway section and update the values.yaml to forward requests to the new Keycloak

values.yaml

axual-governance:
  ## API Gateway
  api-gateway:
    config:
      gateway:
        endpoints:
          # existing endpoints should not be changed
          # ...
          keycloak:
            enabled: true
            # Change to the new Keycloak Service Name
            url: "http://[new-keycloak-service-name]"

        local:
          auth:
            # this endpoint should not be changed
            issuerUrlForValidation: [existing-keycloak-url]
            # Change to the new Keycloak Service Name
            jwkSetUri: http://[new-keycloak-service-name]/auth/realms/local/protocol/openid-connect/certs

        sso:
          keycloak:
            # this endpoint should not be changed
            advertisedBaseUrl: [existing-keycloak-url]
            # Change to the new Keycloak Service Name
            internalBaseUrl: http://[new-keycloak-service-name]

Upgrade the Governance stack helm application with the new values.yaml, this will update the API-Gateway pod with the new configuration.

Verification

In this step, we are going to verify that the Self-Service is able to interact with the new Keycloak pod.

Open a browser and access your Tenant in the Self-Service

You should be able to access the Self-Service as usual.

In case any of the above verification is failed, proceed to the Rollback Scenarios section.

Uninstall the Keycloak using Axual Helm Charts

Objective

To remove the old Keycloak installed with Axual Helm Charts.

Execution

In the old values.yaml you used to configure the Axual Helm Charts installation, disable the component:

values.yaml

global:
  # Global Mgmt configuration
  mgmt:
    keycloak:
      enabled: false

Upgrade the old Management stack helm application with the new values.yaml, this will undeploy the old Keycloak pod.

Verification

The old Keycloak is removed from the kubernetes cluster.

Rollback Scenarios

Self-Service still using the old Keycloak

Disable the new Keycloak by turning off the global.keycloak.enabled toggle

values.yaml

global:
  # existing toggle should not be changed
  # ...
  keycloak:
    enabled: false

Upgrade the Governance stack helm application with the new values.yaml, this will undeploy the new Keycloak pod.
Using the backup Keycloak DB, restore the Keycloak DB Schema to v19
Open a browser and access your Tenant in the Self-Service

You should be able to access the Self-Service as usual.

Self-Service switched to the new Keycloak

Navigate to the api-gateway section and update the values.yaml to forward requests to the old Keycloak

values.yaml

axual-governance:
  ## API Gateway
  api-gateway:
    config:
      gateway:
        endpoints:
          # existing endpoints should not be changed
          # ...
          keycloak:
            enabled: true
            # Restore to the old Keycloak Service Name
            url: "http://[old-keycloak-service-name]"

        local:
          auth:
            # this endpoint should not be changed
            issuerUrlForValidation: [existing-keycloak-url]
            # Restore to the old Keycloak Service Name
            jwkSetUri: http://[old-keycloak-service-name]/auth/realms/local/protocol/openid-connect/certs

        sso:
          keycloak:
            # this endpoint should not be changed
            advertisedBaseUrl: [existing-keycloak-url]
            # Restore to the old Keycloak Service Name
            internalBaseUrl: http://[old-keycloak-service-name]

Upgrade the Governance stack helm application with the new values.yaml, this will update the API-Gateway pod with the new configuration.
Open a browser and access your Tenant in the Self-Service

You should be able to access the Self-Service as usual.