Distributor Changelog

Add self stall-detection for message distributor tasks: the task reports when it is RUNNING but not making progress while work is pending, through new config stall.detection.enabled / stall.detection.threshold.ms / stall.detection.check.interval.ms, new metrics message_distributor_batch_processor_stalled, message_distributor_batch_processor_no_progress_ms and message_distributor_batch_processor_assigned_partitions, and a new DistributorTaskStalled alert. Detection only reports (metric plus log line); it never fails or restarts the task.

Count every stop-progress failure per partition: a batch-level failure now increments message_distributor_batch_processor_partition_exception_count_total (previously only per-record produce failures were counted), so the exceptions view is trustworthy.

Dashboard: add stall, progress (copy rate vs arrival), per-partition lag and topic-partitioning panels to the Message Distributor dashboard.

Dashboard: add a "Time without progress (ms) per task" panel that plots message_distributor_batch_processor_no_progress_ms against the stall threshold, so the gauge now has a consumer.

Dashboard: empty-result health stats (Running/Failed Tasks, exception counts, "Distributing?") now fall back to 0/OK instead of showing an ambiguous "No data" when nothing has happened yet. The "Distributing?" panel also groups by connector so it matches the DistributorTaskStalled alert.

Integration tests: run out of the box on macOS Docker Desktop (Docker Engine 29+, API >= 1.40) with no manual ~/.docker-java.properties / ~/.testcontainers.properties. A macOS-only Maven profile in distributor-integrationtest points Testcontainers at the real engine socket, sets a supported Docker API version, and fixes Ryuk startup so it cleans up the test containers. Linux CI is unaffected (the profile activates only on macOS).

Unit tests: pass on newer JDKs (JDK 25 / latest LTS) with mvn test, no JDK switch and no extra flags. Byte Buddy is pinned to 1.17.8 (overriding the 1.17.5 that Mockito 5.18.0 bundles), which could not instrument mocks under JDK 25 ("Could not modify all classes", e.g. RemoteProducerTest). Still builds and tests on JDK 17/21.

Dependency updates: Kafka connect-api/connect-runtime to 4.3.1, Avro to 1.12.2, Lombok to 1.18.48, slf4j to 2.0.19, Guava to 33.7.1-jre, Commons Lang3 to 3.20.0, Commons Text to 1.15.0, Dropwizard Metrics JMX to 4.2.40, Jackson Databind to 2.22.2, and several Maven plugin/CI tooling versions. JUnit is upgraded to the JUnit 6 major line (6.1.3). CI: gitlab-pipeline-tools to 2.22.1, which runs the Helm lint jobs on GitLab-hosted arm64 runners and fixes the alpine/helm entrypoint for them.

Fix two tests that broke under the newer Kafka connect-api (it now requires bootstrap.servers with no default): OffsetDistributorTest.metrics and OffsetDistributorTaskTest.cacheTimeoutIsConfigurable now set local.bootstrap.servers / target.bootstrap.servers in their test configs.

Pin jackson-annotations to 2.22 alongside jackson-databind in distributor-common, so it matches everywhere. Without this, distributor-integrationtest's test-scope connect-runtime pulled in an older jackson-annotations that did not match the newer jackson-databind, and Kafka Connect’s plugin scanner failed to load JsonConverter with a NoClassDefFoundError.

Update kafka-synchronisation-connectors to 2.0.0. Fixes a data race in the upstream KafkaSinkRunner (an unsynchronized offsets map shared between the runner thread and the Kafka Connect commit thread, which could throw ConcurrentModificationException or lose an offset commit) and a thread leak in KafkaSinkTask.stop(). Both apply to schema-distributor, which uses this library’s sink runner. No config or code changes needed here: schema-distributor never used the removed deprecated topic.selector string config, and its connector class does not go through the library’s own connector-level validation where the new default-on remote-connectivity check lives.

Fix MessageDistributionChaosIT flaking on a loaded machine: its initial pre-chaos sanity check used a fixed 20s window expecting an exact record count, the same pattern that used to make the post-chaos check flaky. It now waits until every produced record is seen remotely, with early exit, like the post-chaos check already did.

Run the CI coverage-gitlab job on the arm64 image jacoco2cobertura:1.0.11-arm64, so it works on the arm64 runners. The amd64-only tags (1.0.10 / 1.0.11) hang during the runner’s shell detection (detect_shell_script: applet not found) and only fail after the 1-hour job timeout. Also added a short job timeout as a safety net so any future hang fails fast; the job stays allow_failure.

Add Strimzi 1.0.0, 1.0.1, 1.1.0 support (Kafka up to 4.3.0)

Helm chart selects the CRD apiVersion from strimzi.version (kafka.strimzi.io/v1 for Strimzi >= 1.0.0, kafka.strimzi.io/v1beta2 for older); adds strimzi.apiVersion to opt into v1 on Strimzi 0.49 - 0.51. Existing installations on Strimzi 0.40 - 0.51 are unaffected.

Fix silent producer deadlock: RemoteProducer.tryClose() no longer closes the producer while holding the isClosed lock (message and offset distributors)

Add Strimzi 0.50.0, 0.50.1, 0.51.0 support

Add Strimzi 0.49.0 and 0.49.1 support with Kafka 4.1.1

Fix issue where Distributors fail to load when using SERVICE_LOAD plugin discovery mode

Add Strimzi 0.48.0 support with Kafka 4.1.0

Fix OffsetDistributor failing with Kafka 4.x due to removed GroupMetadataManager class

Add Strimzi 0.46.1 and 0.47.0 support

Remove previously deprecated Strimzi versions

Deprecate Strimzi versions 0.40.0 up to but not including 0.44.0

Update implementation to be compatible with Kafka 4.0.x

Upgrade dependencies

Update Helm charts to determine image tag from provided Strimzi version

Fix incorrect conversion of null keys and values to empty array

Cert and key names can be configured

Add Strimzi 0.44.0 and 0.45.0 support, deprecate versions up to but not including 0.40.0

Move default registry from docker.axual.io to registry.axual.io

Move default registry from docker.axual.io to registry.axual.io

Fix issues where producing message on source cluster with null key/value results in empty string on target cluster

Fixed a bug where invalid offset commit metadata could crash offset distribution. Invalid data is now dropped.

Fix OffsetCommitter issue that target fields are validated but provided to client code with the wrong values, causes the committer deployment to fail

Fix issue failing config provider support for offset committer

Upgrade dependencies to fix reported CVEs

Upgrade base images

Introduce released image tags based on Strimzi operator version for compatibility

Upgraded expose rules for JMX metrics

Use client ids with the connector name and task for internal Kafka communication.

Added initial dashboards to project

Fix issue with Helm Charts that default patterns values were not set in connector configuration

Implement Offset Distributor and Offset Committer to synchronize offsets between clusters

Add static target configuration for Message Distributor

Add Kafka topic/acl init container

Create a separate helm chart for Kafka Connect Distributor/Distributions

Replace queue based Kafka Synchronisation Connectors with new implementation for Message Distributor

Implement backpressure

Add explicit distribution metrics

Partition Exception metric was updated only as a produce exception, ignoring other exceptions like topic resolve exceptions

Upgrade to Kafka 3.4.0

Upgrade to Java 11

Upgrade to Strimzi Kafka Connect container 0.35.1

Upgrade kafka-synchronisation-connectors to version 1.0.7

fix: resolving concurrency context issue

Upgrade kafka-synchronization-connectors to version 1.0.5

Upgrade clients to Kafka version 3.2.3

Upgrade kafka-synchronization-connectors to version 1.0.3

For message distributor and schema distributor, ClientId is composed by “tenant-instance” (before it was “instance”)

Upgrade Strimzi to version 0.31.1 with Kafka 3.2.3

Upgrade Axual Client to version 6.0.4 with Kafka 3.2.3

Message Distributor: dynamic environment support

Added support for system topics (_tenant-instance-metrics-provider and _tenant-instance-connect.*) to be distributed without topic resolving

Added remote producer clientId to message distributor. Format messageDistributor-$insance-$taskId

Moved Schema Distributor to separate module

Added remote producer clientId to schema distributor. Format schemaDistributor-$insance-$taskId

Supported CPU architectures: linux/amd64,linux/arm64

First implementation using Kafka Connect Principals

Added support to retry for discovery api if it’s unavailable on startup.

Dockerfile now based on axual/strimzi/kafka

Negative and corner cases ITs:

Added support to retry for discovery api if it’s unavailable on startup

Removed: Offset Distribution

Supported CPU architectures: linux/amd64,linux/arm64

Made producer metadata.max.age.ms configurable

TimeoutCache lost GroupTopicPartition bugfix

Distribute unresolved _<tenant>-<instance>-metric-provider topic

Fixed ssl.endpoint.identification.algorithm config

Simplified the build process, no copy & replace

Upgraded to kafka-connect:2.6.0

Upgraded to axual-client:5.5.0

Simplified integration tests using connect-runtime

Configurable producer delivery.timeout.ms

Configurable producer connections.max.idle.ms