Management Stack Upgrade

Start new Management Stack

Objective

Come up with a new values.yaml file to start new management components with axual-governance-chart alongside existing axual-helm-charts components.

Execution

Define a new Chart.yaml that defines as dependency the governance:0.3.4 charts.

Chart.yaml

apiVersion: v2
appVersion: "2024.1"
description: Management Stack with Governance Charts
name: governance
type: application
version: 0.1.0
dependencies:
  - name: "axual-governance"
    version: "0.3.4"
    repository: "https://dev.axual.io/nexus/repository/axual-helm-stable"

Create a new values.yaml using the existing Management’s values.yaml used by the Axual Helm Charts as reference.

First, disable the components that you are not upgrading:

values.yaml

global:
  platform-manager-mysql:
    enabled: false
  platform-manager-vault:
    enabled: false
  keycloak-mysql:
    enabled: false
  keycloak:
    enabled: false
  organization-mgmt:
    enabled: false
  metrics-exposer:
    enabled: true

In case you had no HashiCorp Vault running as part of the Management stack with Axual Helm Charts, enable platform-manager-vault

values.yaml

global:
  platform-manager-vault:
    enabled: true

Platform Manager

In the platform-manager section, provide the TLS configuration of the existing Platform-Manager.

values.yaml

axual-governance:
  platform-manager:
    tls:
      enabled: true
      automatedKeystores: true
      createServerKeypairSecret: false
      serverKeypairSecretName: [existing-server-key-pair]
      createClientKeypairSecret: false
      clientKeypairSecretName: [existing-client-key-pair]
      createTruststoreCaSecret: false
      truststoreCaSecretName: [existing-truststore]

In the platform-manager section, provide the Spring Datasource & JPA configuration to point to the existing Database inside the config.spring as below:

values.yaml

axual-governance:
  platform-manager:
    config:
      spring:
        # Spring Datasource
        datasource:
          name: "[database-name]"
          url: "jdbc:mysql://[database-url]:[database-port]/[database-name]?[extra-param]"
          username: "[database-username-for-platform-manager-db]"
          password: "[database-password-for-platform-manager-db]"
          # in case of MariaDB Database
          # driver-class-name: "org.mariadb.jdbc.Driver"
          # in case of MySQL Database
          driver-class-name: "com.mysql.cj.jdbc.Driver"
        # Spring JPA
        jpa.database-platform: "org.hibernate.dialect.MySQLDialect"
        # Flyway Configuration
        flyway:
          # in case of MariaDB Database
          # locations: "classpath:db/migration/mariadb"
          # in case of MySQL Database
          locations: "classpath:db/migration/mysql"

Pay attention on which datasource.driver-class-name and flyway.localtions to select based on your existing DB vendor.

In the platform-manager section, provide the Axual configuration to enable Instance-API interaction.

values.yaml

axual-governance:
  platform-manager:
    config:
      # Axual Configuration
      axual:
        # Instance API Configuration
        instance-api:
          available: true
          authorizer: tls

(Optional) In the platform-manager section, provide the Axual configuration to enable Application-Operation-Manager interaction.

values.yaml

axual-governance:
  platform-manager:
    config:
      # Axual Configuration
      axual:
        # Application Operation Manager Configuration
        operation-manager:
          available: true
          authorizer: tls
          base-uri: https://[existing-aom-url]:[existing-aom-port]

Only in case your installation has Connect support

In the platform-manager section, provide the GovernanceVault configuration to store governance secrets.

values.yaml

axual-governance:
  platform-manager:
    config:
      # Governance Vault Configuration
      governance:
        vault:
          enabled: true
          uri: https://[vault-url]:[vault-port]
          path: "governance"
          roleId: "[platform-manager-policy-role-id]"
          secretId: "[platform-manager-policy-secret-id]"
      vault:
        enabled: false

Access the Vault UI to create the governance KV Secrets Engine
Update (or Create) the Platform Manager policy
Follow these pages to:

Configure HashiCorp Vault with the Governance Charts

Set up HashiCorp Vault and Platform Manager for the governance secrets

(Optional) In the platform-manager section, provide the Vault configuration to store connector secrets.

values.yaml

axual-governance:
  platform-manager:
    config:
    # Connector Vault Configuration
      vault:
        enabled: true
        uri: https://[vault-url]:[vault-port]
        roleId: "[existing-platform-manager-policy-role-id]"
        secretId: "[existing-platform-manager-policy-secret-id]"

Only in case your installation has Connect support

In the platform-manager section, provide the Security configuration.

values.yaml

axual-governance:
  platform-manager:
    config:
      # Axual Configuration
      axual:
        # Define the available authentication methods visible in the Self-Service
        api.available.auth.methods: "[existing-auth-methods]"
        # Define the security configuration
        security:
          header-based-auth: true
      #Server Security
      server:
        ssl:
          # This could be turned on
          enabled: false

Platform UI

In the platform-ui section, provide the Authentication and Platform-Manager domain name to be added inside the CSP policy as shown below:

values.yaml

axual-governance:
  platform-ui:
    # AuthenticationService Function Qualified Domain
    auth0:
      fqdn: "[mgmt-ingress-host]"

    # PlatformManager Function Qualified Domain
    platformManager:
      fqdn: "[mgmt-ingress-host]"

In the platform-ui section, provide the FeatureFlags configuration as shown below:

values.yaml

axual-governance:
  platform-ui:
    # Window.ENV Configuration
    config:
      docsVersion: "2024.1"
      mgmtApiUrl: "https://[mgmt-ingress-host]/api"
      # this URL needs no ending `/`
      mgmtUiUrl: "https://[mgmt-ingress-host]"

      # Feature Flags

      # To enable Insights with Metrics Exposer
      insightsEnabled: true
      metricsExposerUrl: "https://[mgmt-ingress-host]/api/metrics"

      # To enable Browse&Search with Stream-Browse
      streamBrowseEnabled: true
      streamBrowseUrl: 'https://[mgmt-ingress-host]/api/browse'

      # To enable Browse&Search with Topic-Browse
      topicBrowseEnabled: false
      topicBrowseUrl: 'https://[mgmt-ingress-host]/api/stream_configs'

      # To enable Connect
      connectEnabled: true

      # Keycloak Configuration
      keycloakEnabled: true
      oidcEndpoint: "https://[mgmt-ingress-host]"
      oidcScopes: "openid profile email"
      configurationType: "remote"
      responseTypes: "code"
      clientId: "self-service"
      clientSecret: "notSecret"

      # Wizard Providers Configuration
      enabledKafkaProviders:
        - apache_kafka

API Gateway

In the api-gateway section, provide the Axual configuration as shown below:

values.yaml

axual-governance:
  api-gateway:

    env:
      - name: SPRING_SSL_BUNDLE_PEM_SERVER_KEYSTORE_CERTIFICATE
        valueFrom:
          secretKeyRef:
            key: tls.crt
            name: [existing-mgmt-ingress-certificate]
      - name: SPRING_SSL_BUNDLE_PEM_SERVER_KEYSTORE_PRIVATE_KEY
        valueFrom:
          secretKeyRef:
            key: tls.key
            name: [existing-mgmt-ingress-certificate]

    config:

      server:
        ssl:
          enabled: true
          bundle: server

      management:
        server:
          ssl:
            enabled: false

      spring:
        cloud:
          gateway:
            httpclient:
              ssl:
                use-insecure-trust-manager: true


      # Authentication Endpoint
      permissions-api:
        url: "http://[platform-manager-service-name]/api/auth"

      # Topic Browse Config Endpoint
      topic-browse-config-api:
        url: "http://[platform-manager-service-name]/api/stream_configs/{id}/browse-config"

      # Metrics Exposer Config Endpoint
      metrics-exposer-config-api:
        url: "http://[platform-manager-service-name]/api/monitoring_information"

      gateway:
        endpoints:
          platformManager:
            enabled: true
            url: "http://[platform-manager-service-name]"
          organizationManager:
            enabled: false
          topicBrowse:
            enabled: false
          billing:
            enabled: false
          metricsExposer:
            enabled: true
            url: "http://[metrics-exposer-service-name]"
          metricsExposerApiDocs:
            enabled: true
            url: "http://[metrics-exposer-service-name]"
          platformUi:
            enabled: true
            url: "http://[platform-UI-service-name]"
          keycloak:
            enabled: true
            url: "http://[keycloak-service-name]"
          streamBrowseLegacy:
            enabled: true
            url: "https://[stream-browse-service-name]:[stream-browse-service-port]"

      # This defines the Local Realm for user registration
      local:
        auth:
          # Outside URL for Authentication Server
          issuerUrlForValidation: https://[mgmt-ingress-host]/auth/realms/local
          # Kubernetes Service Name
          jwkSetUri: http://[keycloak-service-name]/auth/realms/local/protocol/openid-connect/certs

      # Keycloak Authentication Server
      sso:
        keycloak:
          advertisedBaseUrl: https://[mgmt-ingress-host]
          internalBaseUrl: http://selfservice-keycloak-http
          # Based on whom sign the ingress
          useInsecureTrustManager: false

      # Enable logging any call intercepted by API Gateway
      logging:
        filter:
          enabled: true

    # Keep Api Gateway's ingress disable for now
    ingress:
      enabled: false

In case you are using HTTPS protocol in one of the gateway.endpoints, you can configure the API-Gateway with a set of known certificates that it can trust.

You should mount an extra volume that contains the trusted-ca and provide the below configuration:

values.yaml

axual-governance:
  api-gateway:
    extraVolumes: |
      - name: ca-certs
        secret:
          secretName: [existing-mgmt-truststore-ca-certs]
    extraVolumeMounts: |
      - name: ca-certs
        mountPath: /path/to/trusted-ca

    config:
      spring:
        cloud:
          gateway:
            httpclient:
              ssl:
                trustedX509Certificates:
                - /path/to/trusted-ca/cert1.pem
                - /path/to/trusted-ca/cert2.pem

When connecting to an HTTPS backend service, be sure that the server certificate of the backend service has properly configured the DNS entry with the backend service name.

(Optional) Metrics Exposer

In the metrics-exposer section, provide the below configuration:

values.yaml

axual-governance:
  # Axual Metrics Exposer
  metrics-exposer:
    # Metrics Exposer Configuration
    config:
      # Axual Configuration
      metrics-exposer:
        public-address: "http://[mgmt-ingress-host]"
        prometheus-urls:
          default: http://[prometheus-service-name]:[prometheus-service-port]

Verification

In this step, we are going to verify that the new Management stack can perform the same operation as the old Management stack.

Add in your /etc/hosts file a DNS mapping for the [mgmt-ingress-host] to 127.0.0.1
Open k9s in sudo mode
Port forward the API-Gateway pod to local-port 443
Open a browser and access https://[mgmt-ingress-host]/login/[your-realm]

You should be able to access the Self-Service as usual.

Verify the most common operations in the Self-Service like:

Topic deployment with ACL and Schemas
Connector start/stop
Stream Browse
Metrics Exposer shows insights

If all checks are successful, you can proceed to the next step.

Remember to remove the DNS mapping in your etc/hosts file.

Switch Management Ingress

Objective

Replacement of the existing Management’s ingress with the API-Gateway’s ingress to dispatch requests to the new Management stack

Execution

To avoid downtime or customer impact, perform this switch outside office hours

Disable the axual-helm-charts Management’s ingress
values.yaml
platform: mgmt: ingress: enabled: false

Enable the axual-governance-charts API-Gateway’s ingress

values.yaml

axual-governance:
  api-gateway:

    ingress:
      enabled: true
      className: "[ingress-class]"
      annotations:
        nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
      hosts:
        - # -- The fully qualified domain name of a network host.
          host: "[mgmt-ingress-host]"
          paths:
            - # -- Matched against the path of an incoming request.
              path: "/"
              # -- Determines the interpretation of the Path matching.
              # Can be one of the following values: `Exact`, `Prefix`, `ImplementationSpecific`.
              pathType: "ImplementationSpecific"
      tls:
      - secretName: "[existing-mgmt-ingress-certificate]"
        hosts:
          - "[mgmt-ingress-host]"

Verification

In this step, we are going to verify that the new Management stack can perform the same operation as the old Management stack.

Open a browser and access https://[mgmt-ingress-host]/login/[your-realm]

Verify the most common operations in the Self-Service like:

Topic deployment with ACL and Schemas
Connector start/stop
Stream Browse
Metrics Exposer shows insights

If all checks are successful, you can proceed to the next step.

Make Self-Service Cluster(s) ready

To be able to edit any clusters, the operator needs to be a user with SUPER_ADMIN and TENANT_ADMIN roles

Objective

Make the Self-Service cluster(s) ready to be used without the Cluster-API, which covers setting the value of the ProviderType and storing Kafka superuser authentication in the Governance Vault.

The below steps will need to be executed for each cluster available in the self-service

Execution

Before starting the cluster Upgrade, please make sure that:
- The super-user certificate, the Intermediate CA + super-user certificate, and the Root CA + Intermediate CA + super-user certificate you will use for accessing the Kafka cluster are part of the kafka.superUsers list
- The Kafka bootstrap service name of the running Kafka Bootstrap is part of the kafka.kafka.externalListenerConfiguration.bootstrap.alternativeNames list
- The Intermediate CA and Root CA of the super-user certificate are part of the kafka.kafka.security.extraCaCerts
  values.yaml
  platform: core: kafka: kafka: superUsers: - [[0] Root CA , [1] Intermediate CA , [2] super-user certificate] - [[0] Intermediate CA , [1] super-user certificate] - [super-user certificate] externalListenerConfiguration: bootstrap: alternativeNames: - ["the bootstrap service-name"] security: extraCaCerts: root-ca.crt: |- [Root CA] intermediate-ca.crt: |- [Intermediate CA]
  Once the above checks are done, the Upgrade will be performed on the Self-Service.
Access the Self-Service
Open the cluster page
Select the cluster that you want to upgrade and press the Edit Cluster button
On the edit page, press the Migrate cluster button
Provide Kafka bootstrap URL data (e.g. the bootstrap-service-name:port]
Select TLS as the authentication type and upload the super-user certificate and its private-key (these are used to instantiate a Kafka AdminClient)
Press the Verify button
Insert the Topic Pattern, Consumer Group Pattern, and Transactional ID Pattern to the existing values that are set for the cluster (you can use the Discovery-API config)
Change the Shared cluster toggle to yes (This allows multiple Tenant to see and use this Cluster)
Press the Update Cluster button.

You have now made the Cluster ready to be used without the Cluster-API

Verification

In this step, we are going to verify that the new Management stack can perform the same operation as the old Management stack.

Open a browser and access https://[mgmt-ingress-host]/login/[your-realm]

Verify the most common operations in the Self-Service like:

Topic deployment with ACL and Schemas
Connector start/stop
Stream Browse
Metrics Exposer shows insights

If all checks are successful, you can proceed to the next cluster.

Make Self-Service Instance(s) ready

To be able to edit any instances, the operator needs to be a user with SUPER_ADMIN role

To be able to start upgrading the instances, you need to have upgraded all Self-Service clusters.

Objective

Make the Self-Service instance(s) ready to be used without the Instance-API.

The below steps will need to be executed for each instance available in the self-service

Execution

Before starting the instance Upgrade, please make sure that the Schema-Registry-Master NetworkPolicy is allowing traffic coming directly from Platform-Manager
values.yaml
platform: instance: schemaregistry: networkPolicy: podMatchLabels: app.kubernetes.io/name: "platform-manager"
Please note that if you have a custom label or have changed the Platform-Manager to any desired name, you should replace it.

Once the above check is done, the Upgrade will be performed on the Self-Service.
Access the Self-Service
Open the instances page
Edit the instance you are upgrading
Remove the Instance Manager API URL
Remove the existing clusters (they will be added again with the Schema Registry details)
Press Select Cluster
Select the cluster used by this instance and provide the Schema Registry details
Provide the Schema Registry Master service-name (e.g. http://[running-schema-registry-master-service-name:running-schema-registry-master-service-port])
Select NoAUth as the authentication method
Press the Verify button
Press the Select Cluster button.
Once inserted all the Clusters, press the Update Instance button.

You have now made the Instance ready to be used without the Instance-API

Verification

In this step, we are going to verify that the new Management stack can perform the same operation as the old Management stack without interacting with the Instance-API and the Cluster-API.

Open a browser and access https://[mgmt-ingress-host]/login/[your-realm]

Verify the most common operations in the Self-Service like:

Topic deployment with ACL and Schemas
Connector start/stop
Stream Browse
Metrics Exposer shows insights
No logs added to Cluster-API
No logs added to Instance-API

If all checks are successful, you can proceed to the next steps.