Deploying Axual Streaming and Axual Governance on Kubernetes

Use the following instructions to deploy Axual Governance and Streaming on Kubernetes, so you can use Axual Platform.

Prerequisites

To install Axual Governance and Axual Platform in your own infrastructure, you need:

Credentials for the Axual Registry (https://docker.axual.io)

If you do not have credentials yet, please request them here: https://insights.axual.com/credentials
A Kubernetes cluster (version 1.24 or above) with an Ingress controller
Helm (>= version 3)
Your favorite terminal

Installation procedure

By following the steps below, you will deploy Axual Streaming first, after which you will deploy Axual Governance and integrate the two.

Step 1: preparations

To install the Axual Streaming and Governance Helm Charts we need to go through some preparation steps.

Create a namespace and Docker Registry Secret

Creating a namespace is optional but recommended. A namespace provides an isolation layer within a Kubernetes cluster.

Create a namespace.
```
kubectl create namespace kafka
```

Create a secret for the Axual Docker registry.

kubectl -n kafka                                    \
    create secret docker-registry axualdockercred   \
    --docker-server=docker.axual.io               \
    --docker-username=[your-username]               \
    --docker-password=[your-password]

Make sure to replace the username and password with your credentials for Axual Harbor.

Download the dependencies

To pull images from the Axual Nexus Registry with Helm, you need to log in with the command:

helm repo add axual-stable https://dev.axual.io/nexus/repository/axual-helm-stable --username [your_username]
helm repo update

Optionally, create a Chart.yml file locally and download the dependencies. Execute the helm dependency update command in the folder where the Chart.yml file is located

helm dependency update

Step 2: install Strimzi Operator

Installing the Strimzi Operator using Helm simplifies the deployment of Kafka and its associated Custom Resource Definitions (CRDs) on a Kubernetes cluster.

To be able to install Strimzi, we need to:

Add the Strimzi repository to helm:

helm repo add strimzi https://strimzi.io/charts/

Update the repo:
```
helm repo update
```

If you want to deploy the platform supporting multi-tenant installations (as described in Configure Strimzi), you need to install the Strimzi Operator using a custom Axual Kafka image. In that scenario, use the command below:

helm install strimzi strimzi/strimzi-kafka-operator \
--version=0.34.0 \
--namespace kafka \
--set kafka.image.registry=docker.axual.io \
--set kafka.image.repository=axual/streaming/strimzi \
--set image.imagePullSecrets='axualdockercred'

If multi-tenancy support is not required, you can install the Strimzi Operator with standard images:

helm install strimzi strimzi/strimzi-kafka-operator \
--namespace kafka

In the first case, you can install the latest version of Strimzi that supports released Axual Kafka images (Strimzi 0.34.0), in the second scenario, you can install the latest version of Strimzi available.

Step 3: deploying Axual Streaming

Download the example axual-streaming.values.yaml

Click to open axual-streaming.values.yaml

global:
  clusterDomain: "cluster.local"

  imageRegistry: "docker.axual.io"
  imagePullSecrets:
    - name: axualdockercred

  platform-manager:
    enabled: true

  platform-ui:
    enabled: true

  api-gateway:
    enabled: true

  organization-mgmt:
    enabled: true

  topic-browse:
    enabled: true

api-gateway:
  config:
    gateway:
      endpoints:
        platformManager:
          enabled: true
          url: "http://axual-governance-platform-manager"
        organizationManager:
          enabled: true
          url: "http://axual-governance-organization-mgmt"
        topicBrowse:
          enabled: true
          url: "http://axual-governance-topic-browse"
        billing:
          enabled: false
        metricsExposer:
          enabled: false
        platformUi:
          enabled: true
          url: "http://axual-governance-platform-ui"
        keycloak:
          enabled: true
          url: "http://keycloak:8080"

    topic-browse-config-api:
      url: "http://axual-governance-platform-manager/api/stream_configs/{id}/browse-config"
    permissions-api:
      url: "http://axual-governance-platform-manager/api/auth"

    local:
      auth:
        issuerUrlForValidation: https://axual-governance.apps-crc.testing/auth/realms/local
        jwkSetUri: http://keycloak:8080/auth/realms/local/protocol/openid-connect/certs
    sso:
      keycloak:
        advertisedBaseUrl: https://axual-governance.apps-crc.testing
        internalBaseUrl: http://keycloak:8080
        useInsecureTrustManager: true
    logging:
      filter:
        enabled: true

  route:
    enabled: true
    annotations: {}
    labels: {}
    host: "axual-governance.apps-crc.testing"
    path: "/"
    tls:
      termination: "edge"

platform-manager:
  debug:
    enabled: false
  image:
    pullPolicy: "Always"
  mtls:
    enabled: false
  config:
    spring:
      datasource:
        name: "fluxdb"
        url: "jdbc:mysql://axual-platform-manager-mysql:3306/selfservicedb?useSSL=false&useLegacyDatetimeCode=false&serverTimezone=UTC"
        username: "fluxmaster"
        password: "Passw0rd"
        driver-class-name: "com.mysql.cj.jdbc.Driver"
      jpa.database-platform: "org.hibernate.dialect.MySQLDialect"
      flyway:
        locations: "classpath:db/migration/mysql"
    axual:
      api.available.auth.methods: "SSL, SCRAM_SHA_512"
      instance-api:
        available: false
      operation-manager:
        available: false
      connect:
        available: false
      client:
        socket-timeout: 60000
      security:
        header-based-auth: true # Disable Keycloak and rely on headers passed from API Gateway
    governance:
      vault:
        enabled: false
        uri: "http://vault:8200"
        path: "governance"
        # Required: change to your roleId for PlatformManager, see step 9 above
        roleId: "744062d7-1d86-3496-bfdc-76f4d7352a2e"
        # Required: change to your secretId for PlatformManager, see step 9 above
        secretId: "634799b5-3ac7-ff6e-071f-a620243318fe"
    # Vault Configuration for Connectors
    vault:
      enabled: false
    azure:
      keyVault:
        enabled: false
    subscription-management:
      enabled: false
    server:
      ssl:
        enabled: false
      forward-headers-strategy: framework

platform-ui:
  platformManager:
    fqdn: "axual-governance.apps-crc.testing"

  # Window.ENV Configuration
  config:
    docsVersion: "2023.3"
    mgmtApiUrl: "https://axual-governance.apps-crc.testing/api"
    mgmtUiUrl: "https://axual-governance.apps-crc.testing/"
    organizationManagerUrl: "https://axual-governance.apps-crc.testing/api/organizations"
    topicBrowseUrl: 'https://axual-governance.apps-crc.testing/api/stream_configs'
    stripeBillingManagementUrl: "https://billing.stripe.com/p/login/test_00g15x1UV4hKcI8eUU"

    # Feature Flags
    billingEnabled: false
    insightsEnabled: false
    dataClassificationEnabled: false
    connectEnabled: false

    clientId: "self-service"
    clientSecret: "notSecret"
    configurationType: "remote"

    # Stream Browsing
    streamBrowseEnabled: false
    # This is for deciding on SB/CB(false) or TB(true)
    topicBrowseEnabled: true

    # Hide multiple environments
    singleEnvironmentEnabled: true
    organizationShortNameEditEnabled: true

    # Stripe Subscription
    subscriptionEnabled: false

    # Keycloak Configuration
    keycloakEnabled: true
    oidcScopes: "openid profile email"
    responseTypes: "code"
    oidcEndpoint: "https://axual-governance.apps-crc.testing"

    # Wizard Allowed Providers
    enabledKafkaProviders:
      - aiven
      - confluent_cloud
      - apache_kafka

organization-mgmt:
  config:
    authStrategy: "keycloak"
    keycloakDomain: "http://keycloak:8080"
    keycloakCLIUsername: "admin"
    keycloakCLIPassword: "admin"
    tlsVerification: false

Determine the domain you want to use throughout the deployment. This domain will be used to access the interfaces of the services you deploy. Set the domain in your shell, it will be used to template a values.yaml. Replace [YOUR_DOMAIN] with your domain in the example below.
```
export DOMAIN=[YOUR_DOMAIN]
```
Apply your domain to the axual-streaming.values.yaml you downloaded
```
sed -i '' -e "s/<domain>/$DOMAIN/g" axual-streaming.values.yaml
```
Note: the example sed command above works with bash on MacOS. Use the correct sed command for your shell/OS.

Install Axual Streaming

helm install streaming axual-stable/axual-streaming --version 1.0.0 -f ./axual-streaming.values.yaml -n kafka

As soon as the platform is deployed successfully, you can continue with the next step.

Axual Streaming certificates

In a default deployment of Axual Streaming, the following certificates should be used during the onboarding in Axual Governance.

Download each of the following files into the same folder, you will need them later on in the configuration of the platform.

cluster-ca.crt: the CA certificate (root + intermediate), used to sign the broker certificate.Axual Governance and other connecting clients need it to trust the connection from the brokers.

View cluster-ca.crt

platform-manager.key: private key which (in this example) is used by Platform Manager to connect to the Kafka cluster

View platform-manager.key

platform-manager.crt: certificate which (in this example) is used by Platform Manager to connect to the Kafka cluster. This certificate is already authorized as superuser in our Kafka cluster.

View platform-manager.crt

Step 4: deploying Axual Governance

Create a new namespace for Axual Governance
```
kubectl create namespace axual
```

Download the example axual-governance.values.yaml

Click to open axual-governance.values.yaml

global:
  imageRegistry: "docker.axual.io"
  imagePullSecrets:
    - name: axualdockercred

  platform-manager-mysql:
    enabled: true

  platform-manager-vault:
    enabled: true

  keycloak-mysql:
    enabled: true

  keycloak:
    enabled: true

  # -- Axual Components toggles
  platform-manager:
    enabled: true

  platform-ui:
    enabled: true

  api-gateway:
    enabled: true

  organization-mgmt:
    enabled: true

  topic-browse:
    enabled: true

# Component specific configuration
## Keycloak DB
keycloak-mysql:
  fullnameOverride: "axual-keycloak-mysql"
  auth:
    rootPassword: "rootpassword"
    database: "keycloak-db"
    username: "keycloak"
    password: "Passw0rd1!"

## Keycloak
keycloak:
  autoscaling:
    enabled: false
  database:
    vendor: "mysql"
    hostname: "axual-keycloak-mysql"
    database: "keycloak-db"
    port: "3306"
    username: "keycloak"
    password: "Passw0rd1!"
  extraEnv: |
    - name: KEYCLOAK_ADMIN
      value: "admin"
    - name: KEYCLOAK_ADMIN_PASSWORD
      value: "admin123"
    - name: JAVA_OPTS_APPEND
      value: -Djgroups.dns.query={{ template "keycloak.serviceDnsName" . }}
    - name: KC_HTTP_ENABLED
      value: "true"
    - name: KC_HOSTNAME_STRICT
      value: "false"

  # This control access to the Keycloak Admin Console
  ingress:
    enabled: false
    ingressClassName: "nginx"
    rules:
      - # -- The fully qualified domain name of a network host.
        host: "axual.<domain>"
        paths:
          - # -- Matched against the path of an incoming request.
            path: "/auth"
            # -- Determines the interpretation of the Path matching.
            # Can be one of the following values: `Exact`, `Prefix`, `ImplementationSpecific`.
            pathType: "ImplementationSpecific"
    tls: [ ]

## Platform Manager DB
platform-manager-mysql:
  fullnameOverride: "axual-platform-manager-mysql"
  auth:
    rootPassword: "rootpassword"
    database: "selfservice-db"
    username: "fluxmaster"
    password: "Passw0rd"

## Platform Manager
platform-manager:
  # Enable Remote Debug with Platform Manager
  debug:
    enabled: false

  mtls:
    enabled: false

  config:
    spring:
      # Spring Datasource
      datasource:
        name: "fluxdb"
        url: "jdbc:mysql://axual-platform-manager-mysql:3306/selfservice-db?useSSL=false&useLegacyDatetimeCode=false&serverTimezone=UTC"
        username: "fluxmaster"
        password: "Passw0rd"
        driver-class-name: "com.mysql.cj.jdbc.Driver"
      # Spring JPA
      jpa.database-platform: "org.hibernate.dialect.MySQLDialect"
      # Flyway Configuration
      flyway:
        locations: "classpath:db/migration/mysql"

    # Axual Platform Manager
    axual:
      api.available.auth.methods: "SSL, SCRAM_SHA_512"
      # Instance Manager Configuration
      instance-api:
        available: false
      # Application Operation Manager Configuration
      operation-manager:
        available: false
      # Connect Configuration
      connect:
        instance-connect-credentials:
          axual-poc:
            authorizer: "basic"
            username: "me"
            password: "notsecret"
        available: true

      #Client Configuration
      client:
        socket-timeout: 60000

      # Security Configuration
      security:
        header-based-auth: true # Disable Keycloak and rely on headers passed from API Gateway

    # Governance Vault Configuration
    governance:
      vault:
        enabled: false
        uri: "http://axual-governance-platform-manager-vault:8200"
        path: "governance"
        roleId: "885d808c-6c58-33fe-fe4f-592b32eaa763"
        secretId: "efb8dc0a-f16a-913c-6f7c-99dbf77ccce1"

    # Vault Configuration for Connectors
    vault:
      enabled: false
    connector-vault:
      enabled: true
      instances:
        axual-poc:
          uri: "http://axual-governance-platform-manager-vault:8200"
          roleId: "a76c3bf3-2e9a-eb14-968a-bd65c112b5c4"
          secretId: "ff19752d-7ba5-83d7-d281-8497fa919e52"
          certChainKeyname: "certificate.chain"
          privateKeyname: "private.key"
          connectorsPath: "connectors"
    # Azure KeyVault Configuration
    azure:
      keyVault:
        enabled: false

    # Subscription Management Configuration
    subscription-management:
      enabled: false

    #Server Security
    server:
      ssl:
        enabled: false
      forward-headers-strategy: framework

## Self-Service UI
platform-ui:
  # Auth0 Function Qualified Domain
  auth0:
    fqdn: "axual.<domain>"

  # PlatformManager Function Qualified Domain
  platformManager:
    fqdn: "axual-governance-platform-manager"

  # Window.ENV Configuration
  config:
    docsVersion: "2024.1"
    mgmtApiUrl: "https://axual.<domain>/api"
    mgmtUiUrl: "https://axual.<domain>/"
    stripeBillingManagementUrl: "https://billing.stripe.com/p/login/test_00g15x1UV4hKcI8eUU"
    topicBrowseUrl: 'https://axual.<domain>/api/stream_configs'

    # Feature Flags
    billingEnabled: false
    insightsEnabled: false
    streamBrowseEnabled: false
    # This is for deciding on SB/CB(false) or TB(true)
    topicBrowseEnabled: true
    dataClassificationEnabled: false
    connectEnabled: true

    subscriptionEnabled: false

    # Keycloak Configuration
    keycloakEnabled: true
    oidcEndpoint: "https://axual.<domain>"
    oidcScopes: "openid profile email"
    configurationType: "remote"
    responseTypes: "code"
    clientId: "self-service"
    clientSecret: "notSecret"

    # OM Configuration
    organizationManagerUrl: 'https://axual.<domain>/api/organizations'
    organizationShortNameEditEnabled: true

    # Wizard Providers Configuration
    enabledKafkaProviders:
      - confluent_cloud
      - aiven
      - apache_kafka

  # CSP Extra Values
  csp:
    extra:
      # To allow loading UserFlow script
      scriptSrc: "https://*.userflow.com"
      mediaSrc: "https://*.userflow.com https://storage.googleapis.com/studio1-prod-blob/"
      imageSrc: "https://*.userflow.com https://storage.googleapis.com/studio1-prod-blob/"
      styleSrc: "https://*.userflow.com"
      connectSrc: "https://*.userflow.com wss://e.userflow.com"

## Organization Manager
organization-mgmt:
  config:
    # Keycloak Configuration
    authStrategy: "keycloak"
    # Issue with API GW -> Keycloak
    #      keycloakDomain: "https://platform.local"
    keycloakDomain: "http://axual-governance-keycloak-http"
    keycloakCLIUsername: "admin"
    keycloakCLIPassword: "admin123"
    # -- To disable tls verification in the http-client
    tlsVerification: false

## Api GateWay
api-gateway:
  debug:
    enabled: false
  # -- Configuration passed to the container.
  # Contents get injected to a ConfigMap, which gets mounted as an `application.yml` file.
  config:
    spring:
      cloud:
        gateway:
          httpclient:
            ssl:
              use-insecure-trust-manager: true

    permissions-api:
      url: "http://axual-governance-platform-manager/api/auth"

    topic-browse-config-api:
      url: "http://axual-governance-platform-manager/api/stream_configs/{id}/browse-config"

    gateway:
      endpoints:
        platformManager:
          enabled: true
          url: "http://axual-governance-platform-manager"
        organizationManager:
          enabled: true
          url: "http://axual-governance-organization-mgmt"
        topicBrowse:
          enabled: true
          url: "http://axual-governance-topic-browse"
        billing:
          enabled: false
        metricsExposer:
          enabled: false
        platformUi:
          enabled: true
          url: "http://axual-governance-platform-ui"
        keycloak:
          enabled: true
          url: "http://axual-governance-keycloak-http"

    # This defines the Local Realm for user registration
    # when UI doesn't provide a Realm header -> LOCAL
    local:
      auth:
        # Outside URL for Authentication Server
        issuerUrlForValidation: https://axual.<domain>/auth/realms/local
        # Kubernetes Service Name if running Keycloak
        jwkSetUri: http://axual-governance-keycloak-http/auth/realms/local/protocol/openid-connect/certs

    # when UI provides a Realm header -> SSO
    # Keycloak Authentication Server
    sso:
      keycloak:
        advertisedBaseUrl: https://axual.<domain>
        internalBaseUrl: http://axual-governance-keycloak-http
        useInsecureTrustManager: true
    logging:
      filter:
        enabled: false

  # Only Api Gateway has an active ingress
  ingress:
    enabled: true
    className: "nginx"
    hosts:
      - # -- The fully qualified domain name of a network host.
        host: "axual.<domain>"
        paths:
          - # -- Matched against the path of an incoming request.
            path: "/"
            # -- Determines the interpretation of the Path matching.
            # Can be one of the following values: `Exact`, `Prefix`, `ImplementationSpecific`.
            pathType: "ImplementationSpecific"
    tls: [ ]

Determine the domain you want to use throughout the deployment. This domain will be used to access the interfaces of the services you deploy. Set the domain in your shell, it will be used to template a values.yaml. Replace [YOUR_DOMAIN] with your domain in the example below.
```
export DOMAIN=[YOUR_DOMAIN]
```
Apply your domain to the axual-governance.values.yaml you downloaded
```
sed -i '' -e "s/<domain>/$DOMAIN/g" axual-governance.values.yaml
```
Note: the example sed command above works with bash on MacOS. Use the correct sed command for your shell/OS.

Install Axual Governance

helm install axual-governance axual-stable/axual-governance --version 0.5.0 -f ./axual-governance.values.yaml -n axual

Axual Governance needs HashiCorp Vault to securely store the credentials of the Kafka cluster you are onboarding. The following steps will make the installation of Vault a smooth experience.

First, create an alias for the Vault CLI which will be reused later and initialize Vault
```
alias v='kubectl -n axual exec --stdin=true axual-governance-platform-manager-vault-0 -- '
v vault operator init -key-shares=1 -key-threshold=1
```
Keep the values for Unseal Key and Root Token in a safe place, you will need them later

v vault operator unseal [UNSEAL_KEY]
v vault login [ROOT_TOKEN]

Prepare Vault to be used by the platform

v vault secrets enable -path=governance kv-v2
v vault auth enable approle
echo 'path "governance/*" {capabilities = ["read","create","update","delete"]}' | v vault policy write platform-manager -
v vault write auth/approle/role/platform-manager token_policies="platform-manager"
v vault read auth/approle/role/platform-manager/role-id
v vault write -force auth/approle/role/platform-manager/secret-id

Find the role_id and secret_id in the output of the command above and store them in a safe place.

Update the configuration for platform-manager in axual-governance.values.yaml to use the role_id and secret_id generated above and set vault.enabled to true

platform-manager:
  governance:
    vault:
      enabled: true
      uri: "http://axual-governance-platform-manager-vault:8200"
      path: "governance"
      roleId: "[ROLE_ID]"     # Insert the value of `role_id` here (between double quotes)
      secretId: "[SECRET_ID]" # Insert the value of `secret_id` here (between double quotes)

Apply the changes to the axual-governance.values.yaml file

helm upgrade --install axual-governance axual-stable/axual-governance --version 0.5.0 -f ./axual-governance.values.yaml -n axual

Finally, log into the Self-Service interface via the following URL: https://axual.{domain}. The following screen will be shown
Click "Register user" to register a tenant admin user. This user has administrative privileges on the platform. The following screen will be shown
Register the tenant admin on your platform. Enter details for the following fields
1. First name
2. Last name
3. Email
4. Username → you will use this to log in next time
5. Password
6. Confirm password
Click "Register" to add the user, the following screen will be shown.
Add the details of your organization and click "Continue". You will be redirected to the dashboard.

By completing the steps above, you have deployed Axual Governance and prepared Self-Service to log in as a tenant admin.

You can now continue with Step 5: onboarding Axual Streaming

Step 5: onboarding Axual Streaming

In this step, you are going to onboard the Kafka cluster you deployed as part of Axual Streaming, so you can create and configure topics and authorize applications in a self-service fashion.

Onboarding the cluster

Log in to Self-Service using the tenant admin credentials
Expand the menu to see all items. The following screen will show:
Click "Clusters", followed by "Add cluster". The following screen will show:
Fill in the following information for your cluster:
1. Name: use a name to refer to your cluster
2. Description: provide a description for this cluster (e.g. dev)
3. Location: additional metadata which helps you to recognize it
Select "Apache Kafka" as the provider.
Click "Continue".
Enter details for the following
1. Kafka bootstrap URL: cluster01-kafka-bootstrap.kafka.svc.cluster.local:9093^*
  
  ^* OR your own <domain> if you specified one
2. Publicly Trusted CA: unchecked
3. CA (PEM): see Axual Streaming certificates » file "cluster-ca.crt"
4. Choose your authentication method.
  - For TLS, upload the following files:
    
    Certificate (PEM): see Axual Streaming certificates » file "platform-manager.crt"
    
    Private Key: see Axual Streaming certificates » file "platform-manager.key"
Click Verify to see whether the provided details are working.Continue when the verify action returns "Broker connection verified"
Leave "Shared cluster" as is
Use the following patterns for multi-environment support (see Topic, Consumer Group and Transactional ID Patterns for more information)
1. Topic pattern: {instance}-{environment}-{topic}
2. Consumer Group pattern: {instance}-{environment}-{group}
3. Transactional ID pattern: {instance}-{environment}-{transactional.id}
Finally, click "Add cluster" to persist all information

Preparing Self-Service

Topics and Applications exist in Environments. In the following steps, you will create an Instance associated with the cluster you just created. Next, you will assign a new environment to this new instance. It is good to know that in the future, you can decide to add more environments to this instance.

Creating an instance

In the menu click "Instances", followed by "New instance"
Provide the following information:
1. Name: a name to describe your instance (e.g. DTA for Dev, Test, Acceptance or Production)
2. Short name: a short name for your instance. Depending on the topic pattern, this name might end up in your topic name
3. Description: add your own description for this instance.
4. Clusters: click "Select cluster", the following screen will show
  1. Select the cluster you created before
  2. Enter the Schema registry URL
  3. Select the Authentication method for Schema Registry (No authentication, Basic authentication and TLS are supported)
  4. Click "Verify" to check whether the URL and connection details are working
  5. Check Publicly trusted CA whenever the certificate of Schema Registry is issued by a publicly trusted CA OR upload the CA used to sign Schema Registry’s certificate.
  6. Click "Select cluster" to assign the cluster to the instance
Enable “Environment mapping” to allow anyone with the ENVIRONMENT_AUTHOR role to create, update and edit environments.
Leave “Properties” as is.
Click “Add Instance” to add the instance.

Creating an environment

In the menu click "Environments", followed by "New environment"
Provide the following information:
1. Name: a name to describe your environment (e.g. "Development" or "Acceptance")
2. Short name: a short name for your environment. Depending on the topic pattern, this name might end up in your topic name
3. Description: Describe the environment
4. Color: select a color which helps users to identify the environment quickly throughout Self-Service
5. Instance: select the instance you created in the previous step
6. Visibility:
  1. Select "Private" if this environment should only be visible by the owner
  2. Select "Public" if this environment should be visible by all users
7. Authorization issuer: determine who is responsible for authorizing topic access requests:
  1. Select "Auto" if access can be granted automatically (e.g. in Test environments)
  2. Select "Stream owner" if access needs to be granted by the topic owner
8. Owner: click "Select cluster", the following screen will show
Leave “Properties” as is.
Click "Add Environment" to add the environment

Congratulations! you have now concluded the first time setup of Self-Service for topic management. Before you invite anyone in the organization to start using it, complete Step 6: Functional verification.

Step 6: Functional verification

In this step, you are going to:

create a topic using the Self-Service interface
authorize an application to produce data to it
produce messages to the topic you created
verify the messages have arrived on the topic

Creating and configuring a topic

In the menu, click "Topics", followed by "New topic"
Provide the following details for the topic:
1. Name: use "mytopic" as the name
2. Description: use something like "Topic for functional verification"
3. Owner: select the team you are member of, "Admins"
4. Key Type: to keep things simple, use "String" as the key type
5. Value Type: to keep things simple, use "String" as the value type
6. Retention Policy: select "Delete"
Leave "Properties" as is
Click "Add Topic" to add the topic
Next, configure the topic, so it is created in the Kafka cluster with appropriate settings. Click the "Configure topic" icon to open the modal with configuration options.
In the window that opens, define the following details:
1. Retention time: choose any retention time for this test topic, e.g. "1 day"
2. Number of partitions: use "4"
Leave "Properties" as is".
Click "Save" to create the topic on the Kafka cluster.

Creating and authorizing the application

Next, you will create an application and authorize it to produce to the topic.

In the menu, click "Applications", followed by "New application"
Provide the following details for the application:
1. ID: use "io.testing.producer"
2. Name: use "Test producer"
3. Short name: use "test_producer"
4. Owner: select "Admins"
5. Application type: select "Custom", followed by "Java"
6. Visibility: select "Public"
7. Description: use "Simple producer app"
Click "Add application" to add the application
Select the environment you created before as part of Preparing Self-Service
Click the "Authentication" icon to add credentials for the producer app
Upload a certificate (PEM) for an application which you want to authorize to produce.

Make sure the application certificate is signed by a Certificate Authority (CA) which is trusted by the broker
Click "Add request" to request produce access on your topic
Select "Producer" for produce access
Select topic "mytopic"
Click "Request approval" to make sure the app with the uploaded certificate can produce

Your app is ready and authorized to produce. Complete your verification by producing some data on the topic.

Producing some data

We are going to use the tool kcat to produce some data to the topic.

Install kcat on your machine. Click here for installation instructions.

Create a file named kcat.conf with the following contents

# Bootstrap server URL and port
bootstrap.servers=kafka-0.kafka.local:443
security.protocol=SSL
# For ssl.key.location and ssl.certificate.location, use the private key and certificate of the app (PEM)
# It is the same certificate you uploaded in step 6. "Creating and authorizing the application" above
ssl.key.location=platform-manager.key
ssl.certificate.location=platform-manager.crt
# For ssl.ca.location, use the certificate of the CA which signed the broker's certificate
ssl.ca.location=cluster-ca.crt

Produce some messages to the topic with the following command:
```
echo "Test message contents" | kcat -F kcat.conf -P -t loc-test-mytopic -k "Test key"
```
Be sure to use the topic name which respects the pattern. In the example above you can see that you are producing a message to topic mytopic in environment test, instance loc
If you managed to produce messages successfully, you will not see any errors.

The final verification is done in Self-Service in the next paragraph.

Self-Service Verification

The last step in the verification process is done in Self-Service, by using topic browse & search.

Find the topic in Self-Service and navigate to the topic detail page
Click the "Messages" tab
Click "Search" to accept the default search options and browse for messages on the topic right away. If everything is configured correctly, the messages should become visible below the search controls.

Click a row to expand it and show details for a given message

Next steps

🎉 Congratulations! you have now concluded the first time setup of Self-Service for topic management, and you have verified the integration between Axual Governance and Axual Streaming.

Preparing for a production like setup

For a production like setup, you need to do more advanced configuration of the platform. This falls beyond the scope of this "How-To", but is explained under the following documentation sections:

Connecting a consumer application

Analog to the producer app, you can create a consumer app to process the messages on the topic. Use the following command to consume messages from the topic.

kcat -F kcat.conf -C -t loc-test-mytopic -o -1 -e

For more information about kcat parameters, check the official kcat documentation.

Need support?

Please navigate to our Support portal and select Additional » Product trial questions to submit a support request specifically for your trial.