Axual Distributor

This reference introduces the Axual Distributor, the three distribution processes it runs, what it needs before it can be installed, and the pages that document each part of it.

Type

Reference

Goal

Find the Axual Distributor page that covers the process, chart value or metric you need.

Audience

Platform Operators and architects running distribution between Kafka clusters.

When to use

Use this guide as the entry point to the Axual Distributor documentation.

Contents

The sections below cover each area of the component:

What the Axual Distributor does

Axual Distributor is a cluster-level service that synchronises (distributes) messages, offsets and schemas between multiple Kafka clusters.

Axual Distributor is built on top of Kafka Connect and is deployed using Strimzi KafkaConnect and KafkaConnector Custom Resources (CRs). Message and Schema distributors run as Kafka Sink Connectors. The Message Distributor receives messages from all instance topics in the cluster, and the Schema Distributor from the schemas topic. The received messages are distributed to other clusters in the instance. Axual Distributor takes possible differences in internal (technical) topic names on different clusters into account.

Georeplication and MirrorMaker 2

Distribution (which covers the Message Distributor and the Offset Distributor) is a way to create near-identical geo-replicated Kafka clusters.
MirrorMaker 2 does similar things but is less suitable for a true bilateral synchronisation with an Active-Active setup.

Distribution processes

Distribution synchronises data across Kafka clusters. It has multiple components and shapes.

Distribution Between Kafka Clusters

This process synchronises Kafka data to one or multiple clusters.

It consists of a single component, the Message Distributor.

Offset Distribution

This process synchronises the committed offsets for consumer groups to one or multiple clusters. It consists of two components:

  • The Offset Distributor synchronises offsets based on timestamps to one or multiple clusters.

  • The Offset Committer receives the offsets the Offset Distributor sends, and sets them on its own cluster.

Schema Distribution

This legacy process synchronises AVRO schemas across Confluent-based Legacy Schema Registries.

If you use Apicurio Registry, do not use this component. Instead use one shared Apicurio Registry instance that all clusters can reach.

offset distributor

Installation and configuration

The Axual Distributor depends on the Strimzi Operator to create KafkaConnect and KafkaConnector resources on Kubernetes.
To function, it needs at least two running Kafka clusters to sync messages and offsets.

For installation, start with How to Deploy the Axual Distributor.

Component details

The pages below document the chart, its metrics and its release history: