How to Verify Distribution

This guide shows you how to bring distribution up between two clusters one connector at a time, confirming at each stage that records arrive exactly once and that committed offsets follow them.

Type

How-to guide

Goal

Confirm that records and committed offsets are distributed in both directions, with no duplicates.

Audience

Platform Operator who can apply Axual Distributor values, and produce to and consume from a test topic on both clusters.

When to use

Use this guide when commissioning distribution between two clusters, before any application depends on it.

The stages below follow the order in Start the connectors in the right order. Verify each stage before enabling the next connector, because a fault found two stages later is much harder to place.

Contents

The sections below cover each task in this guide:

Prerequisites

Confirm the following before you begin.

Access and permissions required

You need the following access and permissions:

  • Permission to apply Axual Distributor values on both clusters.

  • Permission to open a shell into a broker pod, or another way to run kafka-consumer-groups.sh against both clusters.

  • A producer and consumer able to reach a test topic on both clusters.

Tools and versions required

You need the following tools:

  • kubectl >= 1.28, for reading connector state and logs.

  • kafka-consumer-groups.sh, which ships in the broker image at /opt/kafka/bin/.

Resources that must exist before starting

The following must already exist:

  • Two Axual Distributor deployments, one per cluster, installed but with distribution.enabled set to false. See How to Deploy the Axual Distributor.

  • A test topic that exists on both clusters and matches the distributed topic pattern.

Start both Distributors with distribution disabled

This stage proves the initialisation job works before any connector runs. Do it on cluster A, then repeat it identically on cluster B.

  1. Set distribution.enabled to false and init.enabled to true.

  2. Confirm init.principals lists the principals of the Distributors on the other clusters, not only this one. Each cluster’s initialisation job creates the Access Control Lists (ACLs) the remote Distributors need to write here. A missing principal surfaces later as a permission error on the remote side.

  3. Apply the configuration.

  4. Confirm the Kafka Connect deployment came up and its logs are clean. No connectors should be running, and the ACLs for the remote Distributors should exist.

Verify message distribution in both directions

Enable the Message Distributor on one cluster first, confirm one-way flow, then enable the other side and confirm that a record produced on either cluster arrives exactly once on both. The duplicate check is the point of this stage.

  1. On cluster A, enable only the Message Distributor for the level you are testing, leaving init.enabled true so the connector’s groups get created. Apply, then confirm the connector state and logs.

  2. Produce a record to the test topic on cluster A, and consume from the same topic on cluster B. The record should arrive.

  3. On cluster B, enable only its Message Distributor for the matching level. Apply, then confirm its connector state and logs.

  4. Start a consumer on the test topic on cluster A and another on cluster B.

  5. Produce a record to the test topic on cluster A. Both consumers should see exactly one copy.

  6. Produce a second record on cluster B. Again both consumers should see exactly one copy.

    More than one copy on either side means records are looping between the clusters. Stop here and check the distribution model and the level configuration before enabling any offset connector.

  7. Wait for the distribution lag on the test topics to fall back to near zero, reading it from the Message Distributor’s consumer group as Check that both connectors are keeping up describes. Offset distribution is only meaningful once the records the offsets point at have arrived.

Verify offset distribution in both directions

Bring up the Offset Committer on the receiving cluster before the Offset Distributor on the sending one, so there is something to consume the timestamps that get produced.

  1. On cluster B, enable only the Offset Committer, keeping init.enabled true. Apply, then confirm the timestamps topic was created and the committer’s tasks are running, from the logs and the connector state Kubernetes reports.

  2. On cluster A, disable the Offset Committer and enable only the Offset Distributor for the level. Apply, then confirm its tasks are running.

  3. Record the current offset for the test topic and partition on cluster B.

  4. Start a consumer on the timestamps topic on cluster B.

  5. Set the offset for the test topic and group on cluster A, which is the commit the Offset Distributor picks up.

  6. Confirm that a timestamps record for that group is consumed on cluster B once the window period has passed, and that the offset recorded in step 3 has changed.

  7. Repeat steps 1 to 6 with the clusters swapped: Offset Committer on cluster A, Offset Distributor on cluster B, and the offset checks on cluster A.

Both directions verified means distribution is commissioned. Leave every connector that should be permanently enabled switched on in the values, rather than only in the live cluster.

Once distribution is live, keep an eye on it with How to Check Distribution Health, which reads the connectors' consumer group lag from a broker pod.