Offset Distribution
This guide explains why committed consumer offsets are distributed between clusters, how the Offset Distributor and Offset Committer divide that work between them, where each is configured, and how to tell that both sides are running.
Type |
Explanation |
Goal |
Understand how a consumer can move between clusters without reprocessing weeks of records. |
Audience |
Platform Operators and architects planning a multi-cluster instance. |
When to use |
Read before enabling offset distribution, and when diagnosing a consumer that reprocesses after a migration. |
How offset distribution works
Kafka consumer applications can store their position for one or more topic partitions inside Kafka. The consumer application then continues consuming where it left off after it stops or fails. Offset Distribution copies those committed offsets to every Kafka cluster of the tenant instance, so a consumer application can migrate from one Kafka cluster to another. An application moving from an on-premises installation to the cloud, and using the cloud Kafka cluster, is one example.
Distribution Between Kafka Clusters describes whether committed offsets are distributed, and to which clusters.
The Offset Distributor syncs consumer group offsets from the source cluster to a target cluster, so consumer applications can start on the target cluster without processing all the data on the topic first. When the Message Distributor copies messages to a new Kafka cluster, that cluster holds no consumer group offset, so a new consumer application using the same group ID would process potentially weeks of messages again. The sync runs frequently, and the two connectors split the work between the clusters:
| Connector | Runs on | What it does |
|---|---|---|
Offset Distributor |
Source cluster |
Determines the timestamps of all consumer groups on all Kafka topics, gathered from the |
Offset Committer |
Target cluster |
Receives those offset timestamps and sets the consumer group offsets on its own cluster from them. |
The process carries some delay (below 1 minute), so a limited amount of double processing remains possible.
Enabling the Offset Distributor and Offset Committer
Both connectors are configured in the Axual Distributor chart. The Offset Distributor’s values sit under its level in levels: message and offset distributors, and the Offset Committer has its own section, offsetCommitter.
The order they are started in matters, because a distributed offset is only meaningful once the record it points at has arrived. See Start the connectors in the right order.
Checking that both sides are running
Offsets only arrive when both connectors are up: the Offset Distributor on the source cluster and the Offset Committer on the target. One side running on its own distributes nothing, and neither side reports the other as missing.
For the commands that confirm each side is keeping up, and for what else runs alongside them, see Check that both connectors are keeping up.