Monitoring Distributor

The Axual Distributors provide metrics about their functioning. The metrics are the best way to monitor distributors performance, as they are designed to never fail except for extreme conditions.

This document contain all the metrics and JMX MBean ObjectNames, which are exposed through Prometheus.

The available attributes of each metric type is listed at the end of this page.

See the Helm Readme on how to enable metrics.

Monitoring Distributors

This section contains the metrics exposed by the different distributors. Some metrics might be repeated across distributor types, but since each distributor is responsible for its own metrics they get their own copy of the metric definitions.

This document provides the metric names, types and JMX MBean ObjectNames. The attributes of the MBeans can be found in the Metric Type Attributes documentation.

Message Distributor

Connector Metrics

Name connector-start

Type

Timer

Description

Times the duration of the SinkConnector start method

ObjectName

axual.message-distributor:type=connector-start,connector={connectorName},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name connector-stop

Type

Timer

Description

Times the duration of the SinkConnector stop method

ObjectName

axual.message-distributor:type=connector-stop,connector={connectorName},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name connector-task-configs

Type

Timer

Description

Times the duration of the SinkConnector taskConfigs method

ObjectName

axual.message-distributor:type=connector-task-configs,connector={connectorName},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name connector-context-request-reconfiguration

Type

Counter

Description

Counts the number of invocations of the SinkConnectorContext requestTaskReconfiguration method

ObjectName

axual.message-distributor:type=counters,name=connector-context-request-reconfiguration,connector={connectorName},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name connector-context-raise-error

Type

Counter

Description

Counts the number of invocations of the SinkConnectorContext raiseError method

ObjectName

axual.message-distributor:type=counters,name=connector-context-raise-error,connector={connectorName},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name discovery-changed

Type

Counter

Description

Counts the number of registered changes received from the target discovery service

ObjectName

axual.message-distributor:type=counters,name=discovery-changed,connector={connectorName},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Task Metrics

Name task-start

Type

Timer

Description

Times the duration of the SinkTask start method

ObjectName

axual.message-distributor:type=task-start,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name task-stop

Type

Timer

Description

Times the duration of the SinkTask stop method

ObjectName

axual.message-distributor:type=task-stop,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name task-put

Type

Timer

Description

Times the duration of the SinkTask put method

ObjectName

axual.message-distributor:type=task-put,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name task-flush

Type

Timer

Description

Times the duration of the SinkTask flush method

ObjectName

axual.message-distributor:type=task-flush,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name task-pre-commit

Type

Timer

Description

Times the duration of the SinkTask preCommit method

ObjectName

axual.message-distributor:type=task-pre-commit,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name task-open

Type

Timer

Description

Times the duration of the SinkTask open method

ObjectName

axual.message-distributor:type=task-open,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name task-close

Type

Timer

Description

Times the duration of the SinkTask close method

ObjectName

axual.message-distributor:type=task-close,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name distribution-wait-state

Type

Gauge

Description

Provides the current state of the task. Can be UNKNOWN, WAITING or ACTIVE

ObjectName

axual.message-distributor:type=gauges,name=distribution-wait-state,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name batch-processor-age-ms

Type

Gauge

Description

Provides the age of the batch processor in milliseconds.

ObjectName

axual.message-distributor:type=gauges,name=batch-processor-age-ms,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name batch-processor-restart

Type

Counter

Description

Provides the amount of times the sender has restarted.

ObjectName

axual.message-distributor:type=counters,name=batch-processor-restart,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name batch-processor-batch-offered

Type

Counter

Description

Provides the number of batches that were offered for processing. Empty poll results are not counted

ObjectName

axual.message-distributor:type=counters,name=batch-processor-batch-offered,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name batch-processor-batch-accepted

Type

Counter

Description

Provides the number of batches that were accepted for processing. Empty poll results are not counted

ObjectName

axual.message-distributor:type=counters,name=batch-processor-batch-accepted,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name batch-processor-batch-rejected

Type

Counter

Description

Provides the number of batches that were rejected for processing. Empty poll results are not counted

ObjectName

axual.message-distributor:type=counters,name=batch-processor-batch-rejected,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name batch-processor-batch-processing

Type

Timer

Description

Times the duration of the full batch processing.

ObjectName

axual.message-distributor:type=batch-processor-batch-processing,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name batch-processor-batch-record-size

Type

Histogram

Description

Provides statistics of the number of records in a batch

ObjectName

axual.message-distributor:type=histograms,name=batch-processor-batch-record-size,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name batch-processor-batch-partition-size

Type

Histogram

Description

Provides statistics of the number of partitions in a batch

ObjectName

axual.message-distributor:type=histograms,name=batch-processor-batch-partition-size,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name batch-processor-producer-created

Type

Counter

Description

Provides the number of times a new producer was created.

ObjectName

axual.message-distributor:type=counters,name=batch-processor-producer-created,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name batch-processor-producer-closed

Type

Counter

Description

Provides the number of times a new producer was closed.

ObjectName

axual.message-distributor:type=counters,name=batch-processor-producer-closed,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name batch-processor-producer-send-invoked

Type

Counter

Description

Provides the number of times a record was send using the producer.

ObjectName

axual.message-distributor:type=counters,name=batch-processor-producer-send-invoked,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name batch-processor-producer-send-completed

Type

Counter

Description

Provides the number of times a record was successfully send using the producer.

ObjectName

axual.message-distributor:type=counters,name=batch-processor-producer-send-completed,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name batch-processor-producer-send-failed

Type

Counter

Description

Provides the number of times a record was NOT successfully send using the producer.

ObjectName

axual.message-distributor:type=counters,name=batch-processor-producer-send-failed,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name batch-processor-partition-processing

Type

Timer

Description

Times the duration of the processing for a specific topic partition in a batch.

ObjectName

axual.message-distributor:type=batch-processor-partition-processing,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster},topic={topicName},partition={partitionNr}

Name batch-processor-partition-record-size

Type

Histogram

Description

Provides statistics of the number of records for a specific topic partition in a batch

ObjectName

axual.message-distributor:type=histograms,name=batch-processor-partition-record-size,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster},topic={topicName},partition={partitionNr}

Name batch-processor-partition-exception

Type

Counter

Description

Counts the number of times a specific exception occurred for a specific topic partition in a batch.

ObjectName

axual.message-distributor:type=counters,name=batch-processor-partition-exception,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster},topic={topicName},partition={partitionNr},exception={exceptionClassName}

Offset Distributor

Connector metrics

Name connector-start

Type

Timer

Description

Times the duration of the SinkConnector start method

ObjectName

axual.offset-distributor:type=connector-start,connector={connectorName},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name connector-stop

Type

Timer

Description

Times the duration of the SinkConnector stop method

ObjectName

axual.offset-distributor:type=connector-stop,connector={connectorName},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name connector-task-configs

Type

Timer

Description

Times the duration of the SinkConnector taskConfigs method

ObjectName

axual.offset-distributor:type=connector-task-configs,connector={connectorName},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Task metrics

Name task-offsets-in

Type

Meter

Description

Count and rate of received offset records

ObjectName

axual.offset-distributor:type=meters,name=task-offsets-in,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name waiting-cache-keys

Type

Gauge

Description

Number of keys (group/topic/partition) waiting in the cache to be processed

ObjectName

axual.offset-distributor:type=gauges,name=waiting-cache-keys,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name cache-misses

Type

Meter

Description

Count and rate of cache misses, due to no waiting key made available (yet).

ObjectName

axual.offset-distributor:type=meters,name=cache-misses,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name cache-hits

Type

Meter

Description

Count and rate of cache hits

ObjectName

axual.offset-distributor:type=meters,name=cache-hits,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name cache-pushbacks-intransit

Type

Meter

Description

Count and rate of number of times a key (group/topic/partition) is pushed back into the cache because the previous offset update was not confirmed yet on the remote

ObjectName

axual.offset-distributor:type=meters,name=cache-pushbacks-intransit,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name cache-pushbacks-send-error

Type

Meter

Description

Count and rate of number of times a key (group/topic/partition) is pushed back into the cache because sending the offset produced an error, and should be retried

ObjectName

axual.offset-distributor:type=meters,name=cache-pushbacks-send-error,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name send-exceptions

Type

Meter

Description

Count and rate of number of exceptions on produce

ObjectName

axual.offset-distributor:type=meters,name=send-exceptions,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name produce-times

Type

Timer

Description

Times the duration between offset produce and resulting callback (regardless of success or error)

ObjectName

axual.offset-distributor:type=produce-times,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Name keys-in-transit

Type

Gauge

Description

Number of keys (group/topic/partition) in transit; that is, offset has been produced to the remote but no result (success or error) was received yet

ObjectName

axual.offset-distributor:type=gauges,name=keys-in-transit,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},from={sourceCluster},to={targetCluster}

Offset Committer

Connector metrics

Name connector-start

Type

Timer

Description

Times the duration of the SinkConnector start method

ObjectName

axual.offset-committer:type=connector-start,connector={connectorName},tenant={tenantShortName},instance={instanceShortName}

Name connector-stop

Type

Timer

Description

Times the duration of the SinkConnector stop method

ObjectName

axual.offset-committer:type=connector-stop,connector={connectorName},tenant={tenantShortName},instance={instanceShortName}

Name connector-task-configs

Type

Timer

Description

Times the duration of the SinkConnector taskConfigs method

ObjectName

axual.offset-committer:type=connector-task-configs,connector={connectorName},tenant={tenantShortName},instance={instanceShortName}

Task metrics

Name task-start

Type

Timer

Description

Times the duration of the SinkTask start method

ObjectName

axual.offset-committer:type=task-start,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

Name task-stop

Type

Timer

Description

Times the duration of the SinkTask stop method

ObjectName

axual.offset-committer:type=task-stop,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

Name task-put

Type

Timer

Description

Times the duration of the SinkTask put method

ObjectName

axual.offset-committer:type=task-put,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

Name offsets-group-processing

Type

Timer

Description

Times the duration of the processing of offset commits for a single consumer group

ObjectName

axual.offset-committer:type=offsets-group-processing,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

Name offset-processor-reinitialized

Type

Counter

Description

Counts how often the offset processor is reinitialized

ObjectName

axual.offset-committer:type=counters,name=offset-processor-reinitialized,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

Name timestamps-dropped

Type

Counter

Description

Counts how many times a timestamp is dropped and for what cause. Known causes are combined , active-group and missing-partition

ObjectName

axual.offset-committer:type=counters,name=timestamps-dropped,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},cause=\{cause of drop}

Name offset-processor-missing-partitions-processing

Type

Timer

Description

Times how long it takes to determine if partitions are missing

ObjectName

axual.offset-committer:type=offset-processor-missing-partitions-processing,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

Name offset-processor-missing-partitions-partition-size

Type

Histogram

Description

Tracks the number of missing partitions encountered

ObjectName

axual.offset-committer:type=histograms,name=offset-processor-missing-partitions-partition-size,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

Name offset-processor-missing-partitions-exception

Type

Counter

Description

Tracks the number of times a specific exception occurred when trying to determine if partitions are missing

ObjectName

axual.offset-committer:type=counters,name=offset-processor-missing-partitions-exception,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},exception={exceptionClassName}

Name offset-processor-latest-offsets-processing

Type

Timer

Description

Times how long it takes to retrieve the latest offsets for partitions

ObjectName

axual.offset-committer:type=offset-processor-latest-offsets-processing,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

Name offset-processor-latest-offsets-partition-size

Type

Histogram

Description

Tracks the number of partitions for which the latest offsets are retrieved

ObjectName

axual.offset-committer:type=histograms,name=offset-processor-latest-offsets-partition-size,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

Name offset-processor-latest-offsets-exception

Type

Counter

Description

Tracks the number of times a specific exception occurred when trying to determine the latest offsets for partitions

ObjectName

axual.offset-committer:type=counters,name=offset-processor-latest-offsets-exception,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},exception={exceptionClassName}

Name offset-processor-offsets-from-timestamp-processing

Type

Timer

Description

Times how long it takes to retrieve the offsets of partition for a certain timestamp

ObjectName

axual.offset-committer:type=offset-processor-offsets-from-timestamp-processing,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

Name offset-processor-offsets-from-timestamp-partition-size

Type

Histogram

Description

Tracks the number of partitions for which the offsets of partition for a certain timestamp are retrieved

ObjectName

axual.offset-committer:type=histograms,name=offset-processor-offsets-from-timestamp-partition-size,connector={connectorName},task={taskId},tenant={tenantShortName},instance=\{instanceShortName

Name offset-processor-offsets-from-timestamp-exception

Type

Counter

Description

Tracks the number of times a specific exception occurred when trying to determine the offsets for partitions from a timestamp

ObjectName

axual.offset-committer:type=counters,name=offset-processor-offsets-from-timestamp-exception,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},exception={exceptionClassName}

Name offset-processor-commit-offsets-processing

Type

Timer

Description

Times how long it takes to commit offsets for partitions

ObjectName

axual.offset-committer:type=offset-processor-commit-offsets-processing,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

Name offset-processor-commit-offsets-partition-size

Type

Histogram

Description

Tracks the number of partitions for which offsets are committed

ObjectName

axual.offset-committer:type=histograms,name=offset-processor-commit-offsets-partition-size,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

Name offset-processor-commit-offsets-exception

Type

Counter

Description

Tracks the number of times a specific exception occurred when trying to commit offsets for partitions

ObjectName

axual.offset-committer:type=counters,name=offset-processor-commit-offsets-exception,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},exception={exceptionClassName}

Name offset-processor-active-consumer-groups-processing

Type

Timer

Description

Times how long it takes to determine which of the consumer groups are active

ObjectName

axual.offset-committer:type=offset-processor-active-consumer-groups-processing,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

Name offset-processor-active-consumer-groups-group-size

Type

Histogram

Description

Tracks the number of groups for which a consumer group state is requested

ObjectName

axual.offset-committer:type=histograms,name=offset-processor-active-consumer-groups-group-size,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

Name offset-processor-active-consumer-groups-exception

Type

Counter

Description

Tracks the number of times a specific exception occurred when trying to determine the active consumer groups

ObjectName

axual.offset-committer:type=counters,name=offset-processor-active-consumer-groups-exception,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName},exception={exceptionClassName}

Name offset-processor-admin-create

Type

Timer

Description

Times how long it takes to instantiate a new Kafka Admin Client

ObjectName

axual.offset-committer:type=offset-processor-admin-create,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

Name offset-processor-admin-close

Type

Timer

Description

Times how long it takes to close a Kafka Admin Client

ObjectName

axual.offset-committer:type=offset-processor-admin-close,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

Name offset-processor-consumer-create

Type

Timer

Description

Times how long it takes to instantiate a new Kafka Consumer

ObjectName

axual.offset-committer:type=offset-processor-consumer-create,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

Name offset-processor-consumer-close

Type

Timer

Description

Times how long it takes to close a Kafka Consumer

ObjectName

axual.offset-committer:type=offset-processor-consumer-close,connector={connectorName},task={taskId},tenant={tenantShortName},instance={instanceShortName}

Metric Type Attributes

Counters

A counter is one of the basic metric entities, used to count how often something occurs.

ObjectName <JMX domain-name>:type=<AxualMetricName.name>,[JMX commonTags],[AxualMetricName.tags]

JMX Attribute

Description

Count

the value of the counter

Gauges

A gauge is used to provide the current value of a component. This can be a numeric value, but also a string, boolean or other object. When registering a gauge a value supplier is needed which provides the current value to the metrics exposure.

This method is called everytime a JMX Metric is queried, so the value supplier should not contain blocking features.

ObjectName <JMX domain-name>:type=<AxualMetricName.name>,[JMX commonTags],[AxualMetricName.tags]

JMX Attribute

Description

Count

the value of the counter

Number

is set if a numeric value supplier is used

Value

is set if a non-numeric value supplier is used, like String/Boolean

Histograms

Histograms provide statistical distribution metrics. The metric is updated with a value, and the histogram will compute the new statistical values.

ObjectName <JMX domain-name>:type=<AxualMetricName.name>,[JMX commonTags],[AxualMetricName.tags]

JMX Attribute

Description

50thPercentile

75thPercentile

95thPercentile

98thPercentile

99thPercentile

999thPercentile

Count

how ofter the metric has been updated

Max

the largest value measured

Min

the smallest value measured

Mean

the arithmetic mean or average value of the measured values

StdDev

the standard deviation of the measured values

SnapShotSize

the current sample size for statistics calculation, maximum is 1028 samples

Timers

Timers are used to track the duration of certain actions, and provide statistical duration and throughput metrics.

ObjectName <JMX domain-name>:type=<AxualMetricName.name>,[JMX commonTags],[AxualMetricName.tags]

JMX Attribute

Description

50thPercentile

75thPercentile

95thPercentile

98thPercentile

99thPercentile

999thPercentile

DurationUnit

the unit used to track the duration, default is milliseconds

Count

how ofter the metric has been updated

Max

the largest value measured

Min

the smallest value measured

Mean

the arithmetic mean or average value of the measured values

MeanRate

the mean rate at which events have occurred since the meter was created

FifteenMinuteRate

the fifteen-minute moving average rate at which events have occurred since the meter was created

FiveMinuteRate

the five-minute moving average rate at which events have occurred since the meter was created

OneMinuteRate

the one-minute moving average rate at which events have occurred since the meter was created

RateUnit

the unit in which the rate is measured, default is measurements/second

StdDev

the standard deviation of the measured values

Meters

Timers are used to track the duration of certain actions, and provide statistical duration and throughput metrics.

ObjectName <JMX domain-name>:type=<AxualMetricName.name>,[JMX commonTags],[AxualMetricName.tags]

JMX Attribute

Description

Count

how ofter the metric has been updated

MeanRate

the mean rate at which events have occurred since the meter was created

FifteenMinuteRate

the fifteen-minute moving average rate at which events have occurred since the meter was created

FiveMinuteRate

the five-minute moving average rate at which events have occurred since the meter was created

OneMinuteRate

the one-minute moving average rate at which events have occurred since the meter was created

RateUnit

the unit in which the rate is measured, default is measurements/second