Deploy Axual services

Introduction to Axual services

The Axual services consist of all the services that are required to start up a kafka-cluster, an instance layer on top of that for client connections, and the self-service stack to manage your topics and applications.

Install Axual Operator

Axual Operator (forked version of Strimzi Operator) is being used to deploy a Kafka cluster.

This will NOT install Kafka Clusters! Only the Axual Operator capable of deploying Kafka clusters.

  1. Add the Axual helm repository

    You need the Axual helm repository to be able to use Axual helm charts and operator.

    helm repo add axual-stable https://dev.axual.io/nexus/repository/axual-helm-stable --username your_username
    Make sure to replace the username and password with your credentials for Axual Nexus. If your username or password contains special characters, you might have to 'URL Encode' them to prevent clashes with your terminal: 
 !   #   $   %   &   '   (   )   *   +   ,   /   :   ;   <   =   >   ?   @   [   \   ]   ^   _   `   {   |   }   ~
%21 %23 %24 %25 %26 %27 %28 %29 %2A %2B %2C %2F %3A %3B %3C %3D %3E %3F %40 %5B %5C %5D %5E %5F %60 %7B %7C %7D %7E

  1. Update the Axual helm repository to always download the latest charts available.

    helm repo update

  2. Install Axual Operator with Helm.

    helm upgrade --install strimzi --set watchAnyNamespace=true axual-stable/axual-operator -n kafka

Install Axual Platform in Kubernetes

Helm Chart with preconfigured Tenant

To install the latest version of Axual Platform with preconfigured Tenant and Realm that you can use to log in, you will need to create a custom values.yaml file like the following

values.yaml
global:
  mgmt:
    keycloak:
      importInitialRealm: true

core:
  kafka:
    kafka:
      externalListenerConfiguration:
        bootstrap:
          annotations: {}
          alternativeNames:
            - "platform.local"
        brokers:
          - broker: 0
            advertisedHost: "platform.local"
            advertisedPort: 31757
            nodePort: 31757
            annotations: {}

mgmt:
  api:
    initSelfServiceJob:
      enabled: true

If you are deploying Axual on a machine with an ARM-based CPU, please update the values.yaml to use a compatible version of the database:

values.yaml
mgmt:
  api:
    initSelfServiceJob:
      enabled: true
  mariadb:
    image:
      repository: library/mariadb
      tag: 10.7.1
    metrics:
      image:
        registry: docker.axual.io
        repository: prom/mysqld-exporter
        tag: v0.13.0
    securityContext:
      enabled: true
      fsGroup: 999
      runAsUser: 999
    master:
      persistence:
        mountPath: /var/lib/mysql

Then you can perform the installation command with the custom values as input

helm install platform axual-stable/platform -f values.yaml -n kafka

This will install the Axual Platform with existing Realm and Tenant and with the broker advertised in your local host.

Helm chart without preconfigured Tenant

These commands will install the Axual Platform. However, there won’t be any Tenant in Self-Service and there won’t be a Keycloak realm that we can use to log in.

To search for available chart versions

helm search repo -l axual-stable/platform

To install the Axual platform with a specific Chart version my-version:

helm install platform axual-stable/platform --version=my-version -n kafka

To install the latest version of Axual Platform, run the command as shown below.

helm install platform axual-stable/platform -n kafka

To upgrade from an older version of the chart to the latest version of Axual Platform, run the command as shown below.

helm upgrade --install platform axual-stable/platform -n kafka

(Optional) Add Metrics Exposer to Axual Platform

Metrics Exposer is a new component that allows users to get insight about their applications or streams.

Prerequisites

To have a functional Metrics Exposer there are some prerequisites that need to be done.

Prometheus Operator

The Axual Helm chart can install a Prometheus server, by relying on the Prometheus Operator. If you want to use the included Prometheus server, the Prometheus Operator must first be present on the Kubernetes cluster.

You can run a Prometheus stack by following these steps:

  1. Add the Prometheus Community helm charts

    helm repo add prometheus-community https://prometheus-community.github.io/helm-charts && helm repo update

  2. Now it’s time to install the Prometheus Stack

    helm upgrade --install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
--set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \
--set prometheus.prometheusSpec.ruleSelectorNilUsesHelmValues=false \
--set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
--set prometheus-node-exporter.hostRootFsMount.enabled=false -n kafka

It is also possible to bring your own Prometheus server. Disable the included Prometheus server by setting mgmt.metricsexposer.prometheus.enabled=false, and set mgmt.metricsexposer.axual.prometheusUrls.default to your Prometheus server.

Broker Metrics

Be sure that your prometheus stack is running before starting your kafka brokers.

Available configured Kafka brokers to expose their metrics.

If not configured yet, please upgrade your Axual Platform with the following values.yaml

values.yaml
core:
  kafka:
    kafka:
      #Enable Broker Metrics
      metrics: true
helm upgrade --install platform axual-stable/platform -n kafka -f values.yaml

This will restart the broker with an attached PodMonitor resource. This resource will be scraped by Prometheus and brokers' metrics will be available on Prometheus

Keycloak configuration

Keycloak needs to be properly configured before enabling metrics-exposer:

  • A metrics-exposer Client Scope is required

If you are already logged in to the Axual Self Service, please log out and log in again to have a refreshed JWT token with the updated scopes

Enable Metrics Exposer

After you have done the prerequisite steps, you can restart Axual Platform with the following values.yaml

values.yaml
global:
  mgmt:
    # Toggle for Metrics Exposer Deployment
    metricsexposer:
      enabled: true

If you are performing the installation on Docker Desktop, please be sure to specify the storage class name as hostpath in your values.yaml. Use the following values.yaml

values.yaml
mgmt:
  # Metrics Exposer configuration
  metricsexposer:
    prometheus:
      persistentVolume:
        # Storage Class Name for Docker Desktop
        storageClassName: hostpath
 
helm upgrade --install platform axual-stable/platform -n kafka -f values.yaml

The Metrics Exposer is now available with a dedicated Prometheus instance.

URL: https://platform.local/api/metrics/query
OpenAPI Spec: https://platform.local/api/metrics/api-docs.yaml

Turn on Insights feature on Self-Service

Once Metrics Exposer is running, we need to update the Self-Service configuration to use it and show insights.

values.yaml
mgmt:
  ui:
    # Enables Metrics Exposer scopes
    oidc:
      scopes: "openid profile email metrics-exposer"
    # Enables Insight with local Metrics Exposer
    insights:
      enabled: true
      metricsExposerUrl: "https://platform.local/api/metrics"
helm upgrade --install platform axual-stable/platform -n kafka -f values.yaml

The insight feature is now available in your Self-Service, you will now be able to see metrics like:

  • messages produced to a topic

  • consumer lag for an active consumer on a topic

Try out Axual Platform

Login into Self-Service

  1. Login to Self Service with the following URL:

    URL: https://platform.local/login/axual
    Self-Service Login

  2. Use the first login credentials:

      Username: kubernetes@axual.com
  Password: password
    Self-Service Axual Home

  3. If you would like to log into Keycloak in order to customize authentication settings, add new users or realms, please navigate to: https://platform.local/auth

      Username: admin
  Password: secret

    === Keep your configuration in mind This is the configuration you’ll have to pass to your AxualClient. You can edit the axual-client-examples with the following values:

  Tenant: axual
  Endpoint: https://platform.local:29000

Follow getting started with your configuration

As Axual Platform is now running fine, you can follow our Getting Started guide to produce/consume some events.

Remember to change the tenant, endpoint, and environment from examples

Next step: Install Axual-Connect

This concludes the deployment of the Axual services on k8s. You may continue to Install Axual-Connect.