Size in Bytes
The metric provides the value in bytes that a partition of desired stream uses on each Kafka broker at any given time.
Basic usage
Please refer to the example partition size in bytes metric provided in the API docs
This request is asking for the size in bytes of the
partition 2 of stream payment-events-stream on environment dev for last 30 minutes with the step-size of 10 minutes.
Basic Request
{
"metric": "io.axual.partition/size_bytes",
"stepSize": "PT10M",
"timeWindow": "PT30M",
"filter": {
"type": "AND",
"filters": [
{
"type": "FIELD",
"field": "environment",
"operation": "EQUALS",
"value": "dev"
},
{
"type": "FIELD",
"field": "stream",
"operation": "EQUALS",
"value": "payment-events-stream"
},
{
"type": "FIELD",
"field": "partition",
"operation": "EQUALS",
"value": "2"
}
]
}
}
The below part of sample response, represents the size of partition 2 on each kafka broker (pod) in bytes.
Basic Response
{
"type": "UNGROUPED",
"dataPoints": [
{
"timestamp": "2022-10-18T12:30:00",
"value": 0,
"labels": {
"__name__": "kafka_log_log_size",
"axual_cluster": "jupiter",
"partition": "2",
"pod": "jupiter-kafka-2",
"topic": "axual-qa-dev-payment-events-stream"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:40:00",
"value": 144098,
"labels": {
"__name__": "kafka_log_log_size",
"axual_cluster": "jupiter",
"partition": "2",
"pod": "jupiter-kafka-2",
"topic": "axual-qa-dev-payment-events-stream"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:50:00",
"value": 400366,
"labels": {
"__name__": "kafka_log_log_size",
"axual_cluster": "jupiter",
"partition": "2",
"pod": "jupiter-kafka-2",
"topic": "axual-qa-dev-payment-events-stream"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:30:00",
"value": 0,
"labels": {
"__name__": "kafka_log_log_size",
"axual_cluster": "jupiter",
"partition": "2",
"pod": "jupiter-kafka-1",
"topic": "axual-qa-dev-payment-events-stream"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:40:00",
"value": 149754,
"labels": {
"__name__": "kafka_log_log_size",
"axual_cluster": "jupiter",
"partition": "2",
"pod": "jupiter-kafka-1",
"topic": "axual-qa-dev-payment-events-stream"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:50:00",
"value": 400366,
"labels": {
"__name__": "kafka_log_log_size",
"axual_cluster": "jupiter",
"partition": "2",
"pod": "jupiter-kafka-1",
"topic": "axual-qa-dev-payment-events-stream"
},
"unit": null
}
]
}
This metric could be used to know how much data is stored in each partition, and how it’s distributed among partitions and how much data growths per time unit.
Advanced usage
Using aggregator
By adding aggregator to the request, the size of the partition will be aggregated over all kafka brokers.
For instance asking for the sum aggregation function, will lead to get the total size of the partition in all existing kafka brokers.
Request using sum aggregator
{
"metric": "io.axual.partition/size_bytes",
"stepSize": "PT10M",
"timeWindow": "PT30M",
"aggregator": "sum",
"filter": {
"type": "AND",
"filters": [
{
"type": "FIELD",
"field": "environment",
"operation": "EQUALS",
"value": "dev"
},
{
"type": "FIELD",
"field": "stream",
"operation": "EQUALS",
"value": "payment-events-stream"
},
{
"type": "FIELD",
"field": "partition",
"operation": "EQUALS",
"value": "2"
}
]
}
}
The below response represents the size in bytes of the partition on a Kafka cluster, including replicas.
Response using sum aggregator
{
"type": "UNGROUPED",
"dataPoints": [
{
"timestamp": "2022-10-18T12:30:00",
"value": 0,
"labels": {},
"unit": null
},
{
"timestamp": "2022-10-18T12:40:00",
"value": 293852,
"labels": {},
"unit": null
},
{
"timestamp": "2022-10-18T12:50:00",
"value": 800732,
"labels": {},
"unit": null
}
]
}
It’s also possible to get max or average size of one replica using max or avg aggregators instead of sum
Using groupBy
If you want to get response grouped by some label - you can use groupBy
Request using groupBy
{
"metric": "io.axual.partition/size_bytes",
"groupBy": [
"pod"
],
"stepSize": "PT10M",
"timeWindow": "PT30M",
"filter": {
"type": "AND",
"filters": [
{
"type": "FIELD",
"field": "environment",
"operation": "EQUALS",
"value": "dev"
},
{
"type": "FIELD",
"field": "stream",
"operation": "EQUALS",
"value": "payment-events-stream"
},
{
"type": "FIELD",
"field": "partition",
"operation": "EQUALS",
"value": "2"
}
]
}
}
The below response represents the size in bytes of the partition on each Kafka broker (pod), with data grouped by pod.
Response using groupBy
{
"type": "GROUPED",
"groups": [
{
"labels": {
"pod": "jupiter-kafka-1"
},
"dataPoints": [
{
"timestamp": "2022-10-18T12:30:00",
"value": 0,
"labels": {
"__name__": "kafka_log_log_size",
"axual_cluster": "jupiter",
"partition": "2",
"pod": "jupiter-kafka-1",
"topic": "axual-qa-dev-payment-events-stream"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:40:00",
"value": 149754,
"labels": {
"__name__": "kafka_log_log_size",
"axual_cluster": "jupiter",
"partition": "2",
"pod": "jupiter-kafka-1",
"topic": "axual-qa-dev-payment-events-stream"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:50:00",
"value": 400366,
"labels": {
"__name__": "kafka_log_log_size",
"axual_cluster": "jupiter",
"partition": "2",
"pod": "jupiter-kafka-1",
"topic": "axual-qa-dev-payment-events-stream"
},
"unit": null
}
]
},
{
"labels": {
"pod": "jupiter-kafka-2"
},
"dataPoints": [
{
"timestamp": "2022-10-18T12:30:00",
"value": 0,
"labels": {
"__name__": "kafka_log_log_size",
"axual_cluster": "jupiter",
"partition": "2",
"pod": "jupiter-kafka-2",
"topic": "axual-qa-dev-payment-events-stream"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:40:00",
"value": 144098,
"labels": {
"__name__": "kafka_log_log_size",
"axual_cluster": "jupiter",
"partition": "2",
"pod": "jupiter-kafka-2",
"topic": "axual-qa-dev-payment-events-stream"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:50:00",
"value": 400366,
"labels": {
"__name__": "kafka_log_log_size",
"axual_cluster": "jupiter",
"partition": "2",
"pod": "jupiter-kafka-2",
"topic": "axual-qa-dev-payment-events-stream"
},
"unit": null
}
]
}
]
}