Size in Bytes
The metric provides the size ofa desired stream on each Kafka broker at any given time.
Use cases
How much storage space is used by my topic?
It could be useful to know how much storage space is used for topic, for example:
-
to estimate space left and if new disk storage is needed
-
does topic grow too fast, and maybe it should be configured with smaller retention time or replication factor?
To do that, just use sum aggregator with the smallest time window and time step, it will show you size in bytes for the topic. Sometimes, aggregator is not needed, if you want to estimate space on each Broker (pod)
Request example
{
"metric": "io.axual.stream/size_bytes",
"stepSize": "PT1M",
"timeWindow": "now/-PT1M",
"aggregator": "sum",
"filter": {
"type": "AND",
"filters": [
{
"type": "FIELD",
"field": "environment",
"operation": "EQUALS",
"value": "dev"
},
{
"type": "FIELD",
"field": "stream",
"operation": "EQUALS",
"value": "payment-events-stream"
}
]
}
}
Response example
{
"type": "UNGROUPED",
"dataPoints": [
{
"timestamp": "2022-10-18T16:26:00",
"value": 2228732,
"labels": {},
"unit": null
}
]
}
Basic usage
Please refer to the example stream size in bytes metric provided in API docs
This request is asking for the size in bytes of the stream payment-events-stream on environment dev between 2022-10-18T12:20:00Z and 2022-10-18T13:00:00Z with the step-size of 10 minutes.
Basic Request
{
"metric": "io.axual.stream/size_bytes",
"stepSize": "PT10M",
"timeWindow": "2022-10-18T12:20:00Z/2022-10-18T13:00:00Z",
"filter": {
"type": "AND",
"filters": [
{
"type": "FIELD",
"field": "environment",
"operation": "EQUALS",
"value": "dev"
},
{
"type": "FIELD",
"field": "stream",
"operation": "EQUALS",
"value": "payment-events-stream"
}
]
}
}
The below part of sample response, represents the size of a stream on each kafka broker (pod) in bytes.
Basic Response
{
"type": "UNGROUPED",
"dataPoints": [
{
"timestamp": "2022-10-18T12:30:00",
"value": 0,
"labels": {
"pod": "jupiter-kafka-1"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:40:00",
"value": 149754,
"labels": {
"pod": "jupiter-kafka-1"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:50:00",
"value": 546366,
"labels": {
"pod": "jupiter-kafka-1"
},
"unit": null
},
{
"timestamp": "2022-10-18T13:00:00",
"value": 772366,
"labels": {
"pod": "jupiter-kafka-1"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:30:00",
"value": 0,
"labels": {
"pod": "jupiter-kafka-0"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:40:00",
"value": 2000,
"labels": {
"pod": "jupiter-kafka-0"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:50:00",
"value": 200000,
"labels": {
"pod": "jupiter-kafka-0"
},
"unit": null
},
{
"timestamp": "2022-10-18T13:00:00",
"value": 412000,
"labels": {
"pod": "jupiter-kafka-0"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:30:00",
"value": 0,
"labels": {
"pod": "jupiter-kafka-2"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:40:00",
"value": 146098,
"labels": {
"pod": "jupiter-kafka-2"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:50:00",
"value": 592366,
"labels": {
"pod": "jupiter-kafka-2"
},
"unit": null
},
{
"timestamp": "2022-10-18T13:00:00",
"value": 912366,
"labels": {
"pod": "jupiter-kafka-2"
},
"unit": null
}
]
}
This metric could be used to know how much storage space needed for the stream, how it’s distributed among brokers and how fast storage space is growing.
Advanced usage
Using aggregator
By adding aggregator to the request, the size of the stream will be aggregated over all kafka brokers replicas.
For instance asking for the sum aggregation function, will lead to get the total size of the stream in messages.
Request using sum aggregator
{
"metric": "io.axual.stream/size_bytes",
"stepSize": "PT10M",
"timeWindow": "2022-10-18T12:20:00Z/2022-10-18T13:00:00Z",
"aggregator": "sum",
"filter": {
"type": "AND",
"filters": [
{
"type": "FIELD",
"field": "environment",
"operation": "EQUALS",
"value": "dev"
},
{
"type": "FIELD",
"field": "stream",
"operation": "EQUALS",
"value": "payment-events-stream"
}
]
}
}
The below response represents the size in bytes of the stream on a Kafka cluster, with replicas included.
Response using sum aggregator
{
"type": "UNGROUPED",
"dataPoints": [
{
"timestamp": "2022-10-18T12:30:00",
"value": 0,
"labels": {},
"unit": null
},
{
"timestamp": "2022-10-18T12:40:00",
"value": 297852,
"labels": {},
"unit": null
},
{
"timestamp": "2022-10-18T12:50:00",
"value": 1338732,
"labels": {},
"unit": null
},
{
"timestamp": "2022-10-18T13:00:00",
"value": 2096732,
"labels": {},
"unit": null
}
]
}
Using groupBy
If you want to get response grouped by some label - you can use groupBy
Request using groupBy
{
"metric": "io.axual.stream/size_bytes",
"stepSize": "PT10M",
"timeWindow": "2022-10-18T12:20:00Z/2022-10-18T13:00:00Z",
"groupBy": [
"pod"
],
"filter": {
"type": "AND",
"filters": [
{
"type": "FIELD",
"field": "environment",
"operation": "EQUALS",
"value": "dev"
},
{
"type": "FIELD",
"field": "stream",
"operation": "EQUALS",
"value": "payment-events-stream"
}
]
}
}
The below response represents the size in bytes on each Kafka broker (pod), with data grouped by pod.
Response using groupBy
{
"type": "GROUPED",
"groups": [
{
"labels": {
"pod": "jupiter-kafka-0"
},
"dataPoints": [
{
"timestamp": "2022-10-18T12:30:00",
"value": 0,
"labels": {
"pod": "jupiter-kafka-0"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:40:00",
"value": 2000,
"labels": {
"pod": "jupiter-kafka-0"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:50:00",
"value": 200000,
"labels": {
"pod": "jupiter-kafka-0"
},
"unit": null
},
{
"timestamp": "2022-10-18T13:00:00",
"value": 412000,
"labels": {
"pod": "jupiter-kafka-0"
},
"unit": null
}
]
},
{
"labels": {
"pod": "jupiter-kafka-1"
},
"dataPoints": [
{
"timestamp": "2022-10-18T12:30:00",
"value": 0,
"labels": {
"pod": "jupiter-kafka-1"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:40:00",
"value": 149754,
"labels": {
"pod": "jupiter-kafka-1"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:50:00",
"value": 546366,
"labels": {
"pod": "jupiter-kafka-1"
},
"unit": null
},
{
"timestamp": "2022-10-18T13:00:00",
"value": 772366,
"labels": {
"pod": "jupiter-kafka-1"
},
"unit": null
}
]
},
{
"labels": {
"pod": "jupiter-kafka-2"
},
"dataPoints": [
{
"timestamp": "2022-10-18T12:30:00",
"value": 0,
"labels": {
"pod": "jupiter-kafka-2"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:40:00",
"value": 146098,
"labels": {
"pod": "jupiter-kafka-2"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:50:00",
"value": 592366,
"labels": {
"pod": "jupiter-kafka-2"
},
"unit": null
},
{
"timestamp": "2022-10-18T13:00:00",
"value": 912366,
"labels": {
"pod": "jupiter-kafka-2"
},
"unit": null
}
]
}
]
}