Distributor Metrics Reference

This reference lists every metric the Axual Distributor connectors expose, with its type and JMX MBean ObjectName, and the attributes each metric type carries.

Type

Reference

Goal

Find an Axual Distributor metric’s name, type and JMX ObjectName when building a query or an alert.

Audience

Platform Operators and anyone building Prometheus queries or Grafana dashboards against the Axual Distributor.

When to use

While writing a query, an alert rule or a dashboard panel for the Axual Distributor.

The metrics are the best way to monitor Axual Distributor performance, because they are designed never to fail except under extreme conditions. Each connector owns its own metrics, so a metric name repeated across distributor types is a separate metric with its own ObjectName rather than a shared one.

For enabling metrics collection, see Metrics and Alerts.

Contents

The sections below cover each area in this reference:

Message Distributor

The Message Distributor copies records to a target cluster. Its connector reports lifecycle events, and its tasks report the batching and producing work.

Connector metrics

The Message Distributor’s connector emits these while starting, stopping and reconfiguring itself.

Name Type Description JMX MBean ObjectName

connector-start

Timer

Times the duration of the SinkConnector start method

axual.message-distributor:type=connector-start,connector={connectorName},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

connector-stop

Timer

Times the duration of the SinkConnector stop method

axual.message-distributor:type=connector-stop,connector={connectorName},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

connector-task-configs

Timer

Times the duration of the SinkConnector taskConfigs method

axual.message-distributor:type=connector-task-configs,connector={connectorName},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

connector-context-request-reconfiguration

Counter

Counts the number of invocations of the SinkConnectorContext requestTaskReconfiguration method

axual.message-distributor:type=counters,name=connector-context-request-reconfiguration,connector={connectorName},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

connector-context-raise-error

Counter

Counts the number of invocations of the SinkConnectorContext raiseError method

axual.message-distributor:type=counters,name=connector-context-raise-error,connector={connectorName},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

discovery-changed

Counter

Counts the number of registered changes received from the target discovery service

axual.message-distributor:type=counters,name=discovery-changed,connector={connectorName},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Task metrics

Each Message Distributor task emits these while batching and producing records to the target cluster.

Name Type Description JMX MBean ObjectName

task-start

Timer

Times the duration of the SinkTask start method

axual.message-distributor:type=task-start,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

task-stop

Timer

Times the duration of the SinkTask stop method

axual.message-distributor:type=task-stop,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

task-put

Timer

Times the duration of the SinkTask put method

axual.message-distributor:type=task-put,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

task-flush

Timer

Times the duration of the SinkTask flush method

axual.message-distributor:type=task-flush,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

task-pre-commit

Timer

Times the duration of the SinkTask preCommit method

axual.message-distributor:type=task-pre-commit,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

task-open

Timer

Times the duration of the SinkTask open method

axual.message-distributor:type=task-open,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

task-close

Timer

Times the duration of the SinkTask close method

axual.message-distributor:type=task-close,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

distribution-wait-state

Gauge

Provides the current state of the task. Can be UNKNOWN, WAITING or ACTIVE

axual.message-distributor:type=gauges,name=distribution-wait-state,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

batch-processor-age-ms

Gauge

Provides the age of the batch processor in milliseconds.

axual.message-distributor:type=gauges,name=batch-processor-age-ms,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

batch-processor-restart

Counter

Provides the amount of times the sender has restarted.

axual.message-distributor:type=counters,name=batch-processor-restart,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

batch-processor-batch-offered

Counter

Provides the number of batches that were offered for processing. Empty poll results are not counted

axual.message-distributor:type=counters,name=batch-processor-batch-offered,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

batch-processor-batch-accepted

Counter

Provides the number of batches that were accepted for processing. Empty poll results are not counted

axual.message-distributor:type=counters,name=batch-processor-batch-accepted,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

batch-processor-batch-rejected

Counter

Provides the number of batches that were rejected for processing. Empty poll results are not counted

axual.message-distributor:type=counters,name=batch-processor-batch-rejected,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

batch-processor-batch-processing

Timer

Times the duration of the full batch processing.

axual.message-distributor:type=batch-processor-batch-processing,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

batch-processor-batch-record-size

Histogram

Provides statistics of the number of records in a batch

axual.message-distributor:type=histograms,name=batch-processor-batch-record-size,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

batch-processor-batch-partition-size

Histogram

Provides statistics of the number of partitions in a batch

axual.message-distributor:type=histograms,name=batch-processor-batch-partition-size,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

batch-processor-producer-created

Counter

Provides the number of times a new producer was created.

axual.message-distributor:type=counters,name=batch-processor-producer-created,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

batch-processor-producer-closed

Counter

Provides the number of times a new producer was closed.

axual.message-distributor:type=counters,name=batch-processor-producer-closed,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

batch-processor-producer-send-invoked

Counter

Provides the number of times a record was sent using the producer.

axual.message-distributor:type=counters,name=batch-processor-producer-send-invoked,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

batch-processor-producer-send-completed

Counter

Provides the number of times a record was successfully sent using the producer.

axual.message-distributor:type=counters,name=batch-processor-producer-send-completed,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

batch-processor-producer-send-failed

Counter

Provides the number of times a record was NOT successfully sent using the producer.

axual.message-distributor:type=counters,name=batch-processor-producer-send-failed,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

batch-processor-partition-processing

Timer

Times the duration of the processing for a specific topic partition in a batch.

axual.message-distributor:type=batch-processor-partition-processing,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster},topic={topicName},partition={partitionNr}

batch-processor-partition-record-size

Histogram

Provides statistics of the number of records for a specific topic partition in a batch

axual.message-distributor:type=histograms,name=batch-processor-partition-record-size,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster},topic={topicName},partition={partitionNr}

batch-processor-partition-exception

Counter

Counts the number of times a specific exception occurred for a specific topic partition in a batch.

axual.message-distributor:type=counters,name=batch-processor-partition-exception,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster},topic={topicName},partition={partitionNr},exception={exceptionClassName}

Offset Distributor

The Offset Distributor resolves consumer group timestamps and sends them to the target cluster.

Connector metrics

The connector reports these while starting, stopping and reconfiguring.

Name Type Description JMX MBean ObjectName

connector-start

Timer

Times the duration of the SinkConnector start method

axual.offset-distributor:type=connector-start,connector={connectorName},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

connector-stop

Timer

Times the duration of the SinkConnector stop method

axual.offset-distributor:type=connector-stop,connector={connectorName},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

connector-task-configs

Timer

Times the duration of the SinkConnector taskConfigs method

axual.offset-distributor:type=connector-task-configs,connector={connectorName},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Task metrics

Each task reports these while doing its work.

Name Type Description JMX MBean ObjectName

task-offsets-in

Meter

Count and rate of received offset records

axual.offset-distributor:type=meters,name=task-offsets-in,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

waiting-cache-keys

Gauge

Number of keys (group/topic/partition) waiting in the cache to be processed

axual.offset-distributor:type=gauges,name=waiting-cache-keys,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

cache-misses

Meter

Count and rate of cache misses, due to no waiting key made available (yet).

axual.offset-distributor:type=meters,name=cache-misses,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

cache-hits

Meter

Count and rate of cache hits

axual.offset-distributor:type=meters,name=cache-hits,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

cache-pushbacks-intransit

Meter

Count and rate of number of times a key (group/topic/partition) is pushed back into the cache because the previous offset update was not confirmed yet on the remote

axual.offset-distributor:type=meters,name=cache-pushbacks-intransit,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

cache-pushbacks-send-error

Meter

Count and rate of number of times a key (group/topic/partition) is pushed back into the cache because sending the offset produced an error, and should be retried

axual.offset-distributor:type=meters,name=cache-pushbacks-send-error,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

send-exceptions

Meter

Count and rate of number of exceptions on produce

axual.offset-distributor:type=meters,name=send-exceptions,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

produce-times

Timer

Times the duration between offset produce and resulting callback (regardless of success or error)

axual.offset-distributor:type=produce-times,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

keys-in-transit

Gauge

Number of keys (group/topic/partition) in transit; that is, offset has been produced to the remote but no result (success or error) was received yet

axual.offset-distributor:type=gauges,name=keys-in-transit,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Offset Committer

The Offset Committer receives those timestamps and commits the matching offsets on the local cluster.

Connector metrics

The connector reports these while starting, stopping and reconfiguring.

Name Type Description JMX MBean ObjectName

connector-start

Timer

Times the duration of the SinkConnector start method

axual.offset-committer:type=connector-start,connector={connectorName},tenant={tenantShortName},instance={instanceShortName}

connector-stop

Timer

Times the duration of the SinkConnector stop method

axual.offset-committer:type=connector-stop,connector={connectorName},tenant={tenantShortName},instance={instanceShortName}

connector-task-configs

Timer

Times the duration of the SinkConnector taskConfigs method

axual.offset-committer:type=connector-task-configs,connector={connectorName},tenant={tenantShortName},instance={instanceShortName}

Task metrics

Each task reports these while doing its work.

Name Type Description JMX MBean ObjectName

task-start

Timer

Times the duration of the SinkTask start method

axual.offset-committer:type=task-start,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

task-stop

Timer

Times the duration of the SinkTask stop method

axual.offset-committer:type=task-stop,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

task-put

Timer

Times the duration of the SinkTask put method

axual.offset-committer:type=task-put,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

offsets-group-processing

Timer

Times the duration of the processing of offset commits for a single consumer group

axual.offset-committer:type=offsets-group-processing,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

offset-processor-reinitialized

Counter

Counts how often the offset processor is reinitialized

axual.offset-committer:type=counters,name=offset-processor-reinitialized,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

timestamps-dropped

Counter

Counts how many times a timestamp is dropped and for what cause. Known causes are combined, active-group and missing-partition

axual.offset-committer:type=counters,name=timestamps-dropped,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},cause=\{cause of drop}

offset-processor-missing-partitions-processing

Timer

Times how long it takes to determine if partitions are missing

axual.offset-committer:type=offset-processor-missing-partitions-processing,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

offset-processor-missing-partitions-partition-size

Histogram

Tracks the number of missing partitions encountered

axual.offset-committer:type=histograms,name=offset-processor-missing-partitions-partition-size,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

offset-processor-missing-partitions-exception

Counter

Tracks the number of times a specific exception occurred when trying to determine if partitions are missing

axual.offset-committer:type=counters,name=offset-processor-missing-partitions-exception,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},exception={exceptionClassName}

offset-processor-latest-offsets-processing

Timer

Times how long it takes to retrieve the latest offsets for partitions

axual.offset-committer:type=offset-processor-latest-offsets-processing,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

offset-processor-latest-offsets-partition-size

Histogram

Tracks the number of partitions for which the latest offsets are retrieved

axual.offset-committer:type=histograms,name=offset-processor-latest-offsets-partition-size,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

offset-processor-latest-offsets-exception

Counter

Tracks the number of times a specific exception occurred when trying to determine the latest offsets for partitions

axual.offset-committer:type=counters,name=offset-processor-latest-offsets-exception,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},exception={exceptionClassName}

offset-processor-offsets-from-timestamp-processing

Timer

Times how long it takes to retrieve the offsets of partition for a certain timestamp

axual.offset-committer:type=offset-processor-offsets-from-timestamp-processing,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

offset-processor-offsets-from-timestamp-partition-size

Histogram

Tracks the number of partitions for which the offsets of partition for a certain timestamp are retrieved

axual.offset-committer:type=histograms,name=offset-processor-offsets-from-timestamp-partition-size,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

offset-processor-offsets-from-timestamp-exception

Counter

Tracks the number of times a specific exception occurred when trying to determine the offsets for partitions from a timestamp

axual.offset-committer:type=counters,name=offset-processor-offsets-from-timestamp-exception,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},exception={exceptionClassName}

offset-processor-commit-offsets-processing

Timer

Times how long it takes to commit offsets for partitions

axual.offset-committer:type=offset-processor-commit-offsets-processing,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

offset-processor-commit-offsets-partition-size

Histogram

Tracks the number of partitions for which offsets are committed

axual.offset-committer:type=histograms,name=offset-processor-commit-offsets-partition-size,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

offset-processor-commit-offsets-exception

Counter

Tracks the number of times a specific exception occurred when trying to commit offsets for partitions

axual.offset-committer:type=counters,name=offset-processor-commit-offsets-exception,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},exception={exceptionClassName}

offset-processor-active-consumer-groups-processing

Timer

Times how long it takes to determine which of the consumer groups are active

axual.offset-committer:type=offset-processor-active-consumer-groups-processing,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

offset-processor-active-consumer-groups-group-size

Histogram

Tracks the number of groups for which a consumer group state is requested

axual.offset-committer:type=histograms,name=offset-processor-active-consumer-groups-group-size,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

offset-processor-active-consumer-groups-exception

Counter

Tracks the number of times a specific exception occurred when trying to determine the active consumer groups

axual.offset-committer:type=counters,name=offset-processor-active-consumer-groups-exception,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},exception={exceptionClassName}

offset-processor-admin-create

Timer

Times how long it takes to instantiate a new Kafka Admin Client

axual.offset-committer:type=offset-processor-admin-create,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

offset-processor-admin-close

Timer

Times how long it takes to close a Kafka Admin Client

axual.offset-committer:type=offset-processor-admin-close,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

offset-processor-consumer-create

Timer

Times how long it takes to instantiate a new Kafka Consumer

axual.offset-committer:type=offset-processor-consumer-create,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

offset-processor-consumer-close

Timer

Times how long it takes to close a Kafka Consumer

axual.offset-committer:type=offset-processor-consumer-close,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

Metric Type Attributes

Every metric above is one of five types, and the type decides which attributes its MBean exposes. The tables below list the attributes per type.

Counters

A counter is one of the basic metric entities, used to count how often something occurs.

ObjectName <JMX domain-name>:type=<AxualMetricName.name>,[JMX commonTags],[AxualMetricName.tags]

JMX Attribute

Description

Count

the value of the counter

Gauges

A gauge is used to provide the current value of a component. This can be a numeric value, but also a string, boolean or other object. When registering a gauge a value supplier is needed which provides the current value to the metrics exposure.

This method is called everytime a JMX Metric is queried, so the value supplier should not contain blocking features.

ObjectName <JMX domain-name>:type=<AxualMetricName.name>,[JMX commonTags],[AxualMetricName.tags]

JMX Attribute

Description

Count

the value of the counter

Number

is set if a numeric value supplier is used

Value

is set if a non-numeric value supplier is used, like String/Boolean

Histograms

Histograms provide statistical distribution metrics. The metric is updated with a value, and the histogram will compute the new statistical values.

ObjectName <JMX domain-name>:type=<AxualMetricName.name>,[JMX commonTags],[AxualMetricName.tags]

JMX Attribute

Description

50thPercentile

75thPercentile

95thPercentile

98thPercentile

99thPercentile

999thPercentile

Count

how often the metric has been updated

Max

the largest value measured

Min

the smallest value measured

Mean

the arithmetic mean or average value of the measured values

StdDev

the standard deviation of the measured values

SnapShotSize

the current sample size for statistics calculation, maximum is 1028 samples

Timers

Timers are used to track the duration of certain actions, and provide statistical duration and throughput metrics.

ObjectName <JMX domain-name>:type=<AxualMetricName.name>,[JMX commonTags],[AxualMetricName.tags]

JMX Attribute

Description

50thPercentile

75thPercentile

95thPercentile

98thPercentile

99thPercentile

999thPercentile

DurationUnit

the unit used to track the duration, default is milliseconds

Count

how often the metric has been updated

Max

the largest value measured

Min

the smallest value measured

Mean

the arithmetic mean or average value of the measured values

MeanRate

the mean rate at which events have occurred since the meter was created

FifteenMinuteRate

the fifteen-minute moving average rate at which events have occurred since the meter was created

FiveMinuteRate

the five-minute moving average rate at which events have occurred since the meter was created

OneMinuteRate

the one-minute moving average rate at which events have occurred since the meter was created

RateUnit

the unit in which the rate is measured, default is measurements/second

StdDev

the standard deviation of the measured values

Meters

Meters track how often an event occurs, and provide the total count and moving-average rates for it.

ObjectName <JMX domain-name>:type=<AxualMetricName.name>,[JMX commonTags],[AxualMetricName.tags]

JMX Attribute

Description

Count

how often the metric has been updated

MeanRate

the mean rate at which events have occurred since the meter was created

FifteenMinuteRate

the fifteen-minute moving average rate at which events have occurred since the meter was created

FiveMinuteRate

the five-minute moving average rate at which events have occurred since the meter was created

OneMinuteRate

the one-minute moving average rate at which events have occurred since the meter was created

RateUnit

the unit in which the rate is measured, default is measurements/second