Axual Distributor
Functionality
Overview
Distributor is a cluster level service that synchronizes (distributes) messages, offsets and schemas between multiple Kafka clusters.
Distributor is built on top of Kafka Connect and is deployed using Strimzi KafkaConnect and KafkaConnector CRDs. Message and Schema distributors run as Kafka Sink Connectors which receive messages on all instance topics and schemas topic respectively in the cluster. The received messages are distributed to other clusters in the instance. Distributor takes possible differences in internal (technical) topic names on different clusters into account.
Georeplication and MirrorMaker2
Distribution (that includes MessageDistributor and OffsetDistributor) is a way to create near-identical geo-replicated Kafka clusters.
MirrorMaker 2 does similar things but is less ideal for a true bilateral synchronization with an Active-Active setup.
Distribution
What we call "Distribution" is the synchronization of data over Kafka clusters, it can have multiple components and shapes.
- Message Distribution
-
This process synchronizes Kafka data to one or multiple clusters.
It consists of a single component, the Message Distributor
- Offset Distribution
-
This process synchronizes the committed offsets for consumer groups to one or multiple clusters. It consists of two components:
-
The Offset Distributor synchronizes offsets based on timestamps to one or multiple clusters
-
The Offset Committer to receive and set offsets that are distributed by the Offset Distributor
-
- Schema Distribution
-
is a legacy component used to synchronize AVRO schemas across Confluent Schema Registries.
Installation and configuration
The Distributor is dependent on the Strimzi Operator for the creation of KafkaConnect and KafkaConnector resources on Kubernetes.
To function, it needs at least two running Kafka clusters to sync messages and offsets.
For installation, start with Deploying Distributor, or the Helm Readme for specific configuration options