Platform Manager

Functionality

Overview

Platform Manager is used to administer the Axual Platform. It is used to perform tasks like:

Apply topic configuration
Allow producer/consumer to access a topic
Synchronize Instance

README & Changelog

More details are in the Platform Manager 15.0.0 Readme and the Platform Manager 15.0.0 Changelog

Example use

As a developer that creates many topics per week, I get tired of clicking through the UI. Instead, I set up a CI/CD pipeline to create topics via API calls.

API

The Platform Manager API is documented here.

Download OpenAPI Specification

Installation

The Platform Manager depends on Database Support and Vault for basic functionality, requires API Gateway, Topic Browse, Apache Kafka

Helm Charts

As part of the Governance Helm charts, the API Gateway can be installed following the guide Installation and Configuration of Axual Platform.

Configuration

Vault

Vault configuration is detailed in the Platform Manager README.

KSML Application Deployment Sizes

KSML Applications can be deployed with predefined CPU and memory sizes (e.g., XS, S, M, L, XL). Each size defines Kubernetes resource limits for the application. One size must be set as the default, which is used when no size is specified for the application deployment.

Configuring Deployment Sizes

Edit the values.yaml file to define the available sizes:

axual:
  application-deployment:
    ksml:
      deployment-sizes:
        - name: "XS"
          cpu: 0.5
          memory: 2Gi
        - name: "S"
          default-size: true
          cpu: 1
          memory: 4Gi
        - name: "M"
          cpu: 2
          memory: 8Gi
        - name: "L"
          cpu: 3
          memory: 12Gi
        - name: "XL"
          cpu: 4
          memory: 16Gi

Only one size can be marked as default. If no default is set, the first defined size becomes default automatically.

These sizes are predefined and used by default. Edit values.yaml only if you want to change CPU/memory values or set a different default size. It is recommended to provide the CPU values in cores. For example 0.5, 1, 2.

KSML Application Deployment State Store Sizes

State stores are used for stateful stream processing operations in KSML applications, such as aggregations, joins, and windowing. The state store sizes represent the available persistent volume storage options in megabytes (MB) that can be allocated to KSML Application Deployments.

Configuring State Store Sizes

Edit the values.yaml file to define the available state store sizes:

axual:
  application-deployment:
    ksml:
      streams-state-store:
        sizes-megabytes:
          - 256
          - 512
          - 1024
          - 2048
          - 4096

These sizes define the persistent volume storage capacity available for stateful processing. KSML Application owners will be presented with these options to persist their state through the Self-Service.

Edit values.yaml only if you need to add or modify state store size options for your KSML applications. All values must be specified in megabytes (MB).

KSML Failure Notifications

If Notifications are enabled KSML Applications can be monitored for failure statuses.

Throttling the number of messages sent to Application owners and the frequency with which the status scanning takes place are configurable.

Below are the default values that indicate that Deployment status scraping takes place every 5 minutes and can result in up to 1 Notification being sent per hour per Deployment.

scheduler:
  deployment:
    report:
      failure:
        ksml:
          cron: "0 */5 * ? * *"  # every 5 minutes
          max-frequency-minutes: 60

Security

Authentication Methods

The authentication methods provided by the platform can be configured through the configuration of Platform Manager. SCRAM_SHA_256, SCRAM_SHA_512, and OAUTH_BEARER are available only after their support is enabled on the broker level. By default, only SSL is configured as supported authentication method follow below steps to modify authentication methods:

Enable additional listeners on the brokers.

Edit the values.yaml of the Platform Manager charts and add availableAuthMethods as such:

mgmt:
  api:
    axual:
      availableAuthMethods: 'SSL, SCRAM_SHA_512, SCRAM_SHA_256, OAUTH_BEARER'

Using TLS/SSL between Platform Manager and Remote DB

To use TLS/SSL between Remote DB and Platform Manager follow the steps below:

Change useSSL=true in platform/charts/mgmt/charts/api/values.yaml for Platform Manager

mgmt:
  api:
    spring:
      datasource:
        urlSuffix:
          useSsl: true

Add enabledTLSProtocols to urlSuffix as a comma-separated list, for example: enabledTLSProtocols=TLSv1.2,TLSv1.3
```
mgmt:
  api:
    spring:
      datasource:
        urlSuffix:
          enabledTLSProtocols: TLSv1.2,TLSv1.3
```
When enabling TLSv1.3 for DB connection, we need to be sure that TLSv1.3 is a valid client and https protocols in the jvmArguments"
```
mgmt:
  api:
    jvmArguments: "-Djdk.tls.client.protocols=TLSv1.2,TLSv1.3 -Dhttps.protocols=TLSv1.2,TLSv1.3"
```

Connect Reconciliation Jobs

When Connect support has been enabled, the Platform Manager uses two jobs to keep plugins and deployments in sync.

The plugins job retrieves the available plugins from an Axual Connect instance and updates the Self-Service.
The deployments job retrieves the connector status in an Axual Connect instance and compares it with the application deployment status, if they do not match, it updates the application deployment status in the Self-Service.

Edit the values.yaml for Platform Manager and add the following configuration to change the Quarts expressions, for example:

axual-governance:
  platform-manager:
    config:
      scheduler:
        reconciliation:
          connect:
            #  every 30 minutes
            deployments:
              cron: 0 */30 * ? * *
            plugins:
              cron: 0 0 */1 ? * *

Audit logging

Audit logging is used to create an audit trail on user actions to show who exactly did what on secure resources. Within Axual Platform the Platform Manager writes audit logs with the io.axual.auditing.logging package, these can be filtered out for long-term storage if required.

Externalize Audit Event Logs

Platform Manager can write audit events to a rolling log file in addition to storing them in the database. This makes it possible to ship audit events to an external system without polling the database.

How it works

When auditLog.enabled is set to true in the Platform Manager Helm values, a Logback RollingFileAppender is activated. Every audit event that Platform Manager records is serialized to JSON and appended to the active log file. The appender rolls the file daily and compresses older files; files older than logMaxHistory days are deleted automatically.

The log file is written to an emptyDir volume mounted at /logs. This volume is ephemeral: its contents exist only for the lifetime of the pod and are lost on restart. To persist or forward the events you need to add a sidecar container that reads from the same volume and ships the data externally.

Enabling audit log file writing

Set the following in the Platform Manager values.yaml:

platform-manager:
  auditLog:
    enabled: true                                           (1)
    logFile: /logs/audit-events.log                        (2)
    logFileNamePattern: /logs/audit-events.log.%d{yyyy-MM-dd}.gz  (3)
    logMaxHistory: 30                                      (4)
    logPattern: "%m%n"                                     (5)

1	Activates the `RollingFileAppender`; default is `false`.
2	Path of the active (current) log file inside the container.
3	Logback rolling filename pattern. Files are rolled daily and compressed with gzip.
4	Number of days to retain rolled files before Logback deletes them.
5	Encoder pattern for each log line. The default `%m%n` emits the raw JSON audit entry followed by a newline; change this only if your downstream tooling requires a different format.

When auditLog.enabled is false (the default) the logger io.axual.governance.audit.events is set to OFF and no file is written.

Shipping logs with a sidecar container

Because the /logs volume is an emptyDir, you must add a log shipper sidecar to forward events before the pod is recycled.

Metrics

The Platform Manager exposes the default Prometheus metrics via the Spring Boot Actuator, without additional custom metrics.
More info about Monitoring & Metrics here.