Broker

Starting the Kafka broker

A Kafka cluster consists of one or more brokers (servers). Producers push the records to the topics within broker that stores them on disk keyed by unique offset. It allows consumers to fetch messages by topic, partition and offset.

The broker is a cluster level component and can be (re)started as follows:

Usage:

./axual.sh [start|restart] cluster broker

Axual Broker - Kafka version mapping

Valid values of certain broker configurations can differ between Kafka releases. You can use this map to determine the Kafka version used by a specific Axual Broker release.

Axual Broker	Kafka Version
6.4.0	3.2.0
6.3.0	2.8.1
5.4.0 - 6.2.0	2.6.0
5.3.0	2.4.0

Adding or changing Broker configurations (Axual CLI)

As of broker release 6.0.0 it has become easier to override Kafka Configurations for the broker. For an overview of Kafka configurations, please check https://kafka.apache.org/32/documentation.html#brokerconfigs

To add new broker configuration, you have to make changes in Axual CLI on the server where you are starting the broker.

cd to the directory where Axual CLI is unpacked. This usually is something like ~/platform-deploy or ~/axual-cli.
Edit service/cluster-services.sh file and navigate to function start_service_broker_current.

Locate below code under same function.

local BROKER_ARGS=$( (
    echo "BROKER_BROKER_ID=$NODE_ID"
    echo "BROKER_BROKER_RACK=rack-$RACK_ID"
    echo "NODE_HOSTNAME=$HOSTNAME"
    echo "BROKER_SSL_TRUSTSTORE_LOCATION=/security/$SERVER_TRUSTSTORE"
    echo "BROKER_SSL_KEYSTORE_LOCATION=/security/$SERVER_KEYSTORE"
    ...
    ...
    ...
) | as_docker_environment_variables )

Add the new configuration under BROKER_ARGS variable (same like other configuration in the above code snippet) with prefix BROKER_ following the configuration name in capital letters.

Example: Suppose you want to add new connections.max.idle.ms configuration, then you can add it as: echo "BROKER_CONNECTIONS_MAX_IDLE_MS=300000"

This will look something like:

local BROKER_ARGS=$( (
    echo "BROKER_BROKER_ID=$NODE_ID"
    echo "BROKER_BROKER_RACK=rack-$RACK_ID"
    echo "NODE_HOSTNAME=$HOSTNAME"
    echo "BROKER_SSL_TRUSTSTORE_LOCATION=/security/$SERVER_TRUSTSTORE"
    echo "BROKER_SSL_KEYSTORE_LOCATION=/security/$SERVER_KEYSTORE"
    ...
    ...
    ...
    echo "BROKER_CONNECTIONS_MAX_IDLE_MS=300000"
) | as_docker_environment_variables )

After you have made the changes, restart the Axual Broker with the following command:
```
./axual.sh restart cluster broker
```

Updating the protocol versions in Broker (Axual CLI)

There might be different configuration values (versions) that you might be using in your Brokers according to different requirements. Namely, there are two configurations involved in this.

INTER_BROKER_PROTOCOL_VERSION : Specifies which version of the inter-broker protocol will be used.
LOG_MESSAGE_FORMAT_VERSION : Specify the message format version the broker will use to append messages to the logs.

With Broker version upgrade 6.4.0, you have the option to update above configurations.You can make this configuration updates in the particular platform-config/clusters/{cluster-name}/broker.sh.

Please follow the rolling upgrade section of kafka documentation https://kafka.apache.org/32/documentation.html#upgrade_3_2_0.

The properties inter.broker.protocol.version and log.message.format.version mentioned in kafka documentation can be mapped to INTER_BROKER_PROTOCOL_VERSION and LOG_MESSAGE_FORMAT_VERSION in Broker configuration respectively.