Size in Records
The metric provides the number of messages present in a desired stream on each Kafka broker at any given time.
Use cases
How many messages are all applications producing to my topic?
It could be beneficial to understand from a business perspective how much is happening within the organization, for example:
-
how many payments are processed on a daily basis
-
how many users are logging in on a daily basis
-
how many fraud cases are reported per month
To do that, just use sum aggregator with time window of your choice.
How many messages are there on a topic?
As an owner of a new application consuming messages from a topic, it is good to understand how much data you can expect to be consumed as soon as you start the app from the first time.
To do that, just use sum aggregator with the smallest time window and time step, it will show you number of messages.
Request example
{
"metric": "io.axual.stream/size_records",
"stepSize": "PT1M",
"timeWindow": "now/-PT1M",
"aggregator": "sum",
"filter": {
"type": "AND",
"filters": [
{
"type": "FIELD",
"field": "environment",
"operation": "EQUALS",
"value": "dev"
},
{
"type": "FIELD",
"field": "stream",
"operation": "EQUALS",
"value": "payment-events-stream"
}
]
}
}
Response example
{
"type": "UNGROUPED",
"dataPoints": [
{
"timestamp": "2022-10-18T15:33:00",
"value": 11104,
"labels": {},
"unit": null
}
]
}
Basic usage
Please refer to the example stream size in records metric provided in the API docs
This request is asking for the size in messages of the stream payment-events-stream on environment dev between 2022-10-18T12:20:00Z and 2022-10-18T13:00:00Z with the step-size of 10 minutes.
Basic Request
{
"metric": "io.axual.stream/size_records",
"stepSize": "PT10M",
"timeWindow": "2022-10-18T12:20:00Z/2022-10-18T13:00:00Z",
"filter": {
"type": "AND",
"filters": [
{
"type": "FIELD",
"field": "environment",
"operation": "EQUALS",
"value": "dev"
},
{
"type": "FIELD",
"field": "stream",
"operation": "EQUALS",
"value": "payment-events-stream"
}
]
}
}
The below part of sample response, represents the size of a stream on each kafka broker (pod) in messages.
Basic Response
{
"type": "UNGROUPED",
"dataPoints": [
{
"timestamp": "2022-10-18T12:30:00",
"value": 0,
"labels": {
"pod": "jupiter-kafka-1"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:40:00",
"value": 739,
"labels": {
"pod": "jupiter-kafka-1"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:50:00",
"value": 2712,
"labels": {
"pod": "jupiter-kafka-1"
},
"unit": null
},
{
"timestamp": "2022-10-18T13:00:00",
"value": 3842,
"labels": {
"pod": "jupiter-kafka-1"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:30:00",
"value": 0,
"labels": {
"pod": "jupiter-kafka-0"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:40:00",
"value": 10,
"labels": {
"pod": "jupiter-kafka-0"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:50:00",
"value": 1000,
"labels": {
"pod": "jupiter-kafka-0"
},
"unit": null
},
{
"timestamp": "2022-10-18T13:00:00",
"value": 2060,
"labels": {
"pod": "jupiter-kafka-0"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:30:00",
"value": 0,
"labels": {
"pod": "jupiter-kafka-2"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:40:00",
"value": 722,
"labels": {
"pod": "jupiter-kafka-2"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:50:00",
"value": 2942,
"labels": {
"pod": "jupiter-kafka-2"
},
"unit": null
},
{
"timestamp": "2022-10-18T13:00:00",
"value": 4542,
"labels": {
"pod": "jupiter-kafka-2"
},
"unit": null
}
]
}
This metric could be used to know how many messages stored in the stream, how they distributed among brokers and how many messages are there.
Advanced usage
Using aggregator
By adding aggregator to the request, the size of the stream will be aggregated over all kafka brokers replicas.
For instance asking for the sum aggregation function, will lead to get the total size of the stream in messages.
Request using sum aggregator
{
"metric": "io.axual.stream/size_records",
"stepSize": "PT10M",
"timeWindow": "2022-10-18T12:20:00Z/2022-10-18T13:00:00Z",
"aggregator": "sum",
"filter": {
"type": "AND",
"filters": [
{
"type": "FIELD",
"field": "environment",
"operation": "EQUALS",
"value": "dev"
},
{
"type": "FIELD",
"field": "stream",
"operation": "EQUALS",
"value": "payment-events-stream"
}
]
}
}
The below response represents the size in messages of the stream on a Kafka cluster, with replicas included.
Response using sum aggregator
{
"type": "UNGROUPED",
"dataPoints": [
{
"timestamp": "2022-10-18T12:30:00",
"value": 0,
"labels": {},
"unit": null
},
{
"timestamp": "2022-10-18T12:40:00",
"value": 1471,
"labels": {},
"unit": null
},
{
"timestamp": "2022-10-18T12:50:00",
"value": 6654,
"labels": {},
"unit": null
},
{
"timestamp": "2022-10-18T13:00:00",
"value": 10444,
"labels": {},
"unit": null
}
]
}
Using groupBy
If you want to get response grouped by some label - you can use groupBy
Request using groupBy
{
"metric": "io.axual.stream/size_records",
"stepSize": "PT10M",
"timeWindow": "2022-10-18T12:20:00Z/2022-10-18T13:00:00Z",
"groupBy": [
"pod"
],
"filter": {
"type": "AND",
"filters": [
{
"type": "FIELD",
"field": "environment",
"operation": "EQUALS",
"value": "dev"
},
{
"type": "FIELD",
"field": "stream",
"operation": "EQUALS",
"value": "payment-events-stream"
}
]
}
}
The below response represents the size in messages on each Kafka broker (pod), with data grouped by pod.
Response using groupBy
{
"type": "GROUPED",
"groups": [
{
"labels": {
"pod": "jupiter-kafka-0"
},
"dataPoints": [
{
"timestamp": "2022-10-18T12:30:00",
"value": 0,
"labels": {
"pod": "jupiter-kafka-0"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:40:00",
"value": 10,
"labels": {
"pod": "jupiter-kafka-0"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:50:00",
"value": 1000,
"labels": {
"pod": "jupiter-kafka-0"
},
"unit": null
},
{
"timestamp": "2022-10-18T13:00:00",
"value": 2060,
"labels": {
"pod": "jupiter-kafka-0"
},
"unit": null
}
]
},
{
"labels": {
"pod": "jupiter-kafka-1"
},
"dataPoints": [
{
"timestamp": "2022-10-18T12:30:00",
"value": 0,
"labels": {
"pod": "jupiter-kafka-1"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:40:00",
"value": 739,
"labels": {
"pod": "jupiter-kafka-1"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:50:00",
"value": 2712,
"labels": {
"pod": "jupiter-kafka-1"
},
"unit": null
},
{
"timestamp": "2022-10-18T13:00:00",
"value": 3842,
"labels": {
"pod": "jupiter-kafka-1"
},
"unit": null
}
]
},
{
"labels": {
"pod": "jupiter-kafka-2"
},
"dataPoints": [
{
"timestamp": "2022-10-18T12:30:00",
"value": 0,
"labels": {
"pod": "jupiter-kafka-2"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:40:00",
"value": 722,
"labels": {
"pod": "jupiter-kafka-2"
},
"unit": null
},
{
"timestamp": "2022-10-18T12:50:00",
"value": 2942,
"labels": {
"pod": "jupiter-kafka-2"
},
"unit": null
},
{
"timestamp": "2022-10-18T13:00:00",
"value": 4542,
"labels": {
"pod": "jupiter-kafka-2"
},
"unit": null
}
]
}
]
}