How to Deploy the Axual Distributor

This guide shows you how to install the Axual Distributor chart for one Kafka cluster, fill in its three configuration sections, and start its connectors in the order that avoids distributing offsets ahead of the records they point at.

Type

How-to guide

Goal

Get an Axual Distributor running for one Kafka cluster and its connectors distributing to the remote clusters.

Audience

Platform Operator with Helm access to the cluster and permission to create Strimzi resources in the Axual Distributor’s namespace.

When to use

Use this guide when adding distribution to an instance, once the distribution model is decided.

Deploy one Axual Distributor per Kafka cluster of a tenant instance. An instance spanning three clusters runs three Axual Distributors, each reading locally and writing to the others.

Contents

The sections below cover each task in this guide:

Prerequisites

Confirm the following before you begin.

Access and permissions required

You need the following access and permissions:

  • An account for the Axual chart registry. The Axual Distributor charts and images are in a private registry, so contact Axual support to request access.

  • Permission to create Strimzi KafkaConnect and KafkaConnector resources in the target namespace.

Tools and versions required

You need the following tools:

  • helm >= 3.12.

  • kubectl >= 1.28.

  • A Strimzi Operator version the Axual Distributor release supports. The Axual Distributor image has to match the operator, and the supported combinations are listed in Distributor Helm Readme.

Resources that must exist before starting

The following must already exist:

Kafka cluster

The local cluster this Axual Distributor reads from and distributes out of.

Strimzi Operator

The operator watches for the KafkaConnect and KafkaConnector definitions the chart creates and performs the actual deployment. Nothing starts without it.

Runtime client certificate

A client certificate for connecting to the Kafka cluster and reaching the right topics and consumer groups. It does not need superuser rights.

Initialisation client TLS Secret

A TLS Secret whose certificate is allowed to create topics and Access Control List (ACL) entries, which is what the initialisation job does. It needs Alter on the cluster and Create on the timestamps topic. It does not need superuser rights. See Permissions the init job needs.

Distribution model

Which clusters are the static targets at each level. Decide this before configuring anything, because the cluster names it fixes cannot be changed later without leaving data undistributed. See Distribution Between Kafka Clusters.

Install the chart

Log in to the registry, then install the chart with your values.yaml. Nothing distributes yet: the chart deliberately lets you install Kafka Connect with every distributor disabled, so you deploy first and switch the connectors on later. Take the chart version from Distributor Helm Readme, which is generated from the chart and states the version it documents.

Replace every <VALUE> placeholder with your own value before running a command. 
helm registry login -u <YOUR_USER> registry.axual.io/axual-charts
helm upgrade --install local-distributor oci://registry.axual.io/axual-charts/distributor --version <CHART_VERSION> -f values.yaml

Confirm the Kafka Connect pod reaches Running before configuring any connectors.

Fill in the three configuration sections

The chart’s values split into three areas, and they are written in this order because each reads names from the one before it.

init

Creates the topics and ACL entries the rest needs, deriving names from the two sections below. See init: the initialisation job.

connect

Configures the Kafka Connect deployment itself: bootstrap servers, TLS material, the three internal topics and the worker consumer group. See connect: the Kafka Connect deployment.

distribution

Defines the local and remote clusters and configures the four connectors. See distribution: local and remote clusters.

Each connector is enabled independently, so a connector can be configured now and switched on at the point the ordering below calls for it.

To keep remote cluster private keys out of values.yaml, mount them instead. See How to Load Remote Cluster TLS Material from Kubernetes Secrets.

Start the connectors in the right order

The order matters for the reason given in step 4: a distributed offset is only meaningful once the record it points at has arrived on the target cluster. Enable each connector, confirm it is running, then move to the next.

  1. Start the Schema Distributor, on the cluster where Self-Service registers the schema. Skip this step entirely with Apicurio Registry, which is shared between clusters rather than distributed to.

  2. Start the Offset Committer on every cluster. It looks up and commits offsets locally, and waits harmlessly while no remote Offset Distributor is sending it anything.

  3. Start the Message Distributor, which distributes records to the target cluster.

  4. Wait until the Message Distributor has worked through every topic and is keeping up with new production. Do not skip this. Starting offset distribution early distributes timestamps pointing at records the target cluster has not received.

  5. Start the Offset Distributor, which resolves the timestamp and distributes it to the remote clusters.

Confirm each connector before enabling the next, and verify the result end to end once all of them are running. See How to Verify Distribution.