Platform Manager

Functionality

Overview

Platform Manager is used to administer the Axual Platform. It is used to perform tasks like:

Apply topic configuration
Allow producer/consumer to access a topic
Synchronize Instance

README & Changelog

More details are in the Platform Manager 13.0.0 Readme and the Platform Manager 13.0.0 Changelog

Example use

As a developer that creates many topics per week, I get tired of clicking through the UI. Instead, I set up a CI/CD pipeline to create topics via API calls.

API

The Platform Manager API is documented here.

Installation

The Platform Manager depends on Database Support and Vault for basic functionality, requires API Gateway, Topic Browse, Apache Kafka

Helm Charts

As part of the Governance Helm charts, the API Gateway can be installed following the guide Axual Streaming & Governance Installation.

Configuration

Vault

Vault configuration is detailed in the Platform Manager README.

KSML Application Deployment Sizes

KSML Applications can be deployed with predefined CPU and memory sizes (e.g., XS, S, M, L, XL). Each size defines Kubernetes resource limits for the application. One size must be set as the default, which is used when no size is specified for the application deployment.

Configuring Deployment Sizes

Edit the values.yaml file to define the available sizes:

axual:
  application-deployment:
    ksml:
      deployment-sizes:
        - name: "XS"
          cpu: 0.5
          memory: 2Gi
        - name: "S"
          default-size: true
          cpu: 1
          memory: 4Gi
        - name: "M"
          cpu: 2
          memory: 8Gi
        - name: "L"
          cpu: 3
          memory: 12Gi
        - name: "XL"
          cpu: 4
          memory: 16Gi

Only one size can be marked as default. If no default is set, the first defined size becomes default automatically.

These sizes are predefined and used by default. Edit values.yaml only if you want to change CPU/memory values or set a different default size. It is recommended to provide the CPU values in cores. For example 0.5, 1, 2.

KSML Application Deployment State Store Sizes

State stores are used for stateful stream processing operations in KSML applications, such as aggregations, joins, and windowing. The state store sizes represent the available persistent volume storage options in megabytes (MB) that can be allocated to KSML Application Deployments.

Configuring State Store Sizes

Edit the values.yaml file to define the available state store sizes:

axual:
  application-deployment:
    ksml:
      streams-state-store:
        sizes-megabytes:
          - 256
          - 512
          - 1024
          - 2048
          - 4096

These sizes define the persistent volume storage capacity available for stateful processing. KSML Application owners will be presented with these options to persist their state through the Self-Service.

Edit values.yaml only if you need to add or modify state store size options for your KSML applications. All values must be specified in megabytes (MB).

KSML Failure Notifications

If Notifications are enabled KSML Applications can be monitored for failure statuses.

Throttling of the number of messages sent to Application owners and the frequency with which the status scanning takes place are configurable.

Below is the default values that indicate that Deployment status scraping takes place every 5 minutes and can result to up to 1 Notification being sent per hour per Deployment.

scheduler:
  deployment:
    report:
      failure:
        ksml:
          cron: "0 */5 * ? * *"  # every 5 minutes
          max-frequency-minutes: 60

Security

Authentication Methods

The authentication methods provided by the platform can be configured through the configuration of Platform Manager. SCRAM_SHA_256, SCRAM_SHA_512, and OAUTH_BEARER are available only after their support is enabled on the broker level. By default, only SSL is configured as supported authentication method follow below steps to modify authentication methods:

Enable additional listeners on the brokers.

Edit the values.yaml of the Platform Manager charts and add availableAuthMethods as such:

mgmt:
  api:
    axual:
      availableAuthMethods: 'SSL, SCRAM_SHA_512, SCRAM_SHA_256, OAUTH_BEARER'

Using TLS/SSL between Platform Manager and Remote DB

To use TLS/SSL between Remote DB and Platform Manager follow the steps below:

Change useSSL=true in platform/charts/mgmt/charts/api/values.yaml for Platform Manager

mgmt:
  api:
    spring:
      datasource:
        urlSuffix:
          useSsl: true

Add enabledTLSProtocols to urlSuffix as a comma-separated list, for example: enabledTLSProtocols=TLSv1.2,TLSv1.3
```
mgmt:
  api:
    spring:
      datasource:
        urlSuffix:
          enabledTLSProtocols: TLSv1.2,TLSv1.3
```
When enabling TLSv1.3 for DB connection, we need to be sure that TLSv1.3 is a valid client and https protocols in the jvmArguments"
```
mgmt:
  api:
    jvmArguments: "-Djdk.tls.client.protocols=TLSv1.2,TLSv1.3 -Dhttps.protocols=TLSv1.2,TLSv1.3"
```

Connect Reconciliation Jobs

When Connect support has been enabled, the Platform Manager uses two jobs to keep plugins and deployments in sync.

The plugins job retrieves the available plugins from an Axual Connect instance and updates the Self-Service.
The deployments job retrieves the connector status in an Axual Connect instance and compares it with the application deployment status, if they do not match, it updates the application deployment status in the Self-Service.

Edit the values.yaml for Platform Manager and add the following configuration to change the Quarts expressions, for example:

axual-governance:
  platform-manager:
    config:
      scheduler:
        reconciliation:
          connect:
            #  every 30 minutes
            deployments:
              cron: 0 */30 * ? * *
            plugins:
              cron: 0 0 */1 ? * *

Logging

Change the logging configuration by following these steps.
The Platform Manager is a primary source for Audit logging within the Axual Platform.

Metrics

The Platform Manager exposes the default Prometheus metrics via the Spring Boot Actuator, without additional custom metrics.
More info about Monitoring & Metrics here.