Apache Kafka
Functionality
Overview
Axual Platform is all about improving the security, ownership and workflows on Apache Kafka. To learn more about Kafka itself, read the official documentation.
Different Kafka vendors are available for Axual Platform, but for a complete installation of the platform on Kubernetes we prefer to use the Strimzi Operator to deploy and manage Kafka.
More on Strimzi within the Axual Platform here: Strimzi Operator
API
Apache Kafka does not work through an API, to interact with Kafka you need
-
a library that works on the Kafka protocols
-
a Kafka endpoint: a listener. These can be mTLS, SASL, Oauth and more
-
valid credentials and access to a specific topic, Axual Platform disables any auto-creation of topics and strictly enforces access restrictions.
Follow the Getting Started section for more.
Installation Reference
Kafka installation via the Axual provided Helm charts rely on the Strimzi Operator.
Production ready Kafka
Only if all Infrastructure Requirements are in fulfilled, a production-ready Kafka cluster can be designed.
Availability Zones or Racks
| The Resilience section is a good read for context. |
Kafka works by spreading data over multiple brokers that are expected to be on different Kubernetes Nodes in separated Availability Zones (AZs) or Racks that have their own power and internet connections inside a datacenter.
If one AZ goes down, other AZs should still be functional, so it is common to have 2 or 3 AZs for optimal availability.
-
Kafka on 3 AZs: every AZ will have a Node running a broker and zookeeper, resulting in 3 brokers and 3 zookeepers. Topics need Replication Factor (RF) 3 and 2 minimum In-Sync Replicas (ISRs) ensure a single AZ can become unavailable without service interruption.
-
Kafka on 2 AZs: every AZ will have two nodes running a broker, resulting in 4 brokers. Topics need RF 4 and 2 minimum ISRs ensure a single AZ can become unavailable without service interruption. Zookeeper should still run with 3 Pods, ideally one Pod outside the 2 AZs, on a witness site.
Kafka broker scheduling
It is good practice to set up Taints and Tolerations on the dedicated Kafka worker Nodes to allow the brokers to utilize all its resources. Adding a taint to Kubernetes Nodes may involve an infrastructure team, adding the tolerations to Kafka is exemplified below.
axual-streaming:
kafka:
kafka:
template:
pod:
tolerations:
- key: "axual.io/dedicated"
operator: "Equal"
value: "kafka"
effect: "NoSchedule"Adding a NodeAffinity can also be used if taints can not be set and can also be used in combination with taints.
axual-streaming:
kafka:
kafka:
template:
pod:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "axual.io/dedicated"
operator: In
values:When multiple AZs are available, use PodAntiAffinity to 'push away' brokers and zookeepers from each other to ensure they spread out over all Nodes during scheduling.
axual-streaming:
kafka:
kafka:
template:
pod:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
strimzi.io/name: kafka
topologyKey: "kubernetes.io/hostname"
weight: 50Disks
Disks should be as fast as possible for optimal Kafka performance.
Size should be determined based on historical or estimated data.
Investigate whether Volumes can be increased dynamically.
| Replication or backups of data at volume/disk level should be disabled, for Kafka already replicates data. |
Monitoring
Especially for Production use, the infrastructure used for Monitoring & Metrics and alerting needs to be set up properly and tested.
Additionally, there are more aspects of Observability to be fine-tuned for Production.
Helm Charts
As mentioned before, the Helm charts used for Kafka deployments are developed in-house by Axual. Strimzi does provide the Kubernetes resources but does not develop Helm charts.
To get more details on the Kafka Helm charts, refer to Working with Helm charts to obtain and browse the charts in detail.
Configuration
|
false |
Enable auto creation of topic on the server Type: boolean Default: true Valid Values: Importance: high Update Mode: read-only |
|
false |
Indicates whether to enable replicas not in the ISR set to be elected as leader as a last resort, even though doing so may result in data loss Type: boolean Default: false Valid Values: Importance: high Update Mode: cluster-wide |
|
16 |
The number of threads to use for various background processing tasks Type: int Default: 10 Valid Values: [1,…] Importance: high Update Mode: cluster-wide |
|
4 |
Number of fetcher threads used to replicate messages from a source broker. Increasing this value can increase the degree of I/O parallelism in the follower broker. Type: int Default: 1 Valid Values: Importance: high Update Mode: cluster-wide |
|
20000 |
If a follower hasn’t sent any fetch requests or hasn’t consumed up to the leaders log end offset for at least this time, the leader will remove the follower from isr Type: long Default: 30000 (30 seconds) Valid Values: Importance: high Update Mode: read-only |
|
1000012 |
The maximum amount of data per-partition the server will return. Records are fetched in batches by the consumer. If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch will still be returned to ensure that the consumer can make progress. The maximum record batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config). See fetch.max.bytes for limiting the consumer request size. Type: int Default: 1048576 (1 mebibyte) Valid Values: [0,…] Importance: high |
|
1048576 |
The number of bytes of messages to attempt to fetch for each partition. This is not an absolute maximum, if the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch will still be returned to ensure that progress can be made. The maximum record batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config). Type: int Default: 1048576 (1 mebibyte) Valid Values: [0,…] Importance: medium Update Mode: read-only |
|
65536 |
The socket receive buffer for network requests Type: int Default: 65536 (64 kibibytes) Valid Values: Importance: high Update Mode: read-only |
|
20160 |
After a consumer group loses all its consumers (i.e. becomes empty) its offsets will be kept for this retention period before getting discarded. For standalone consumers (using manual assignment), offsets will be expired after the time of last commit plus this retention period. Type: int Default: 10080 Valid Values: [1,…] Importance: high Update Mode: read-only |
|
1 |
The replication factor for the offsets topic (set higher to ensure availability). Internal topic creation will fail until the cluster size meets this replication factor requirement. Type: short Default: 3 Valid Values: [1,…] Importance: high Update Mode: read-only |
|
1 |
The replication factor for the transaction topic (set higher to ensure availability). Internal topic creation will fail until the cluster size meets this replication factor requirement. Type: short Default: 3 Valid Values: [1,…] Importance: high Update Mode: read-only |
|
1 |
Overridden min.insync.replicas config for the transaction topic. Type: int Default: 2 Valid Values: [1,…] Importance: high Update Mode: read-only |
|
3 |
The number of partitions for the transaction topic (should not change after deployment). Type: int Default: 50 Valid Values: [1,…] Importance: high Update Mode: read-only |
|
- |
The maximum time a message will remain ineligible for compaction in the log. Only applicable for logs that are being compacted. Type: long Default: 9223372036854775807 Valid Values: Importance: medium Update Mode: cluster-wide |
Observability
As Kafka performance and reliability are of critical importance, there are plenty of options to monitor Kafka, via logs, metrics and tracing.
Logging
Kafka logging can be configured by adding an extendSpec section in the values.yaml, for example:
kafka:
config: {}
extendSpec:
logging:
type: inline
loggers:
kafka.root.logger.level: WARN
log4j.logger.kafka: INFO
log4j.logger.kafka.authorizer.logger: INFO
log4j.logger.kafka.request.logger: WARN, CONSOLE
log4j.logger.kafka.network.Processor: OFF
log4j.logger.kafka.server.KafkaApis: OFF
log4j.logger.kafka.controller: INFO
log4j.logger.kafka.log.LogCleaner: INFO
log4j.logger.state.change.logger: INFO
log4j.logger.org.apache.kafka.common.network: INFO
log4j.logger.org.apache.zookeeper: INFO
log4j.logger.org.apache.kafka: INFO
io.strimzi.kafka.agent.KafkaAgent: INFO, CONSOLEMetrics
Enable the exposure of many Kafka JMX metrics via PodMonitor in the values.yaml:
kafka: metrics: true
Enable additional metrics of Kafka Exporter in the values.yaml:
kafka:
kafkaExporter:
enabled: true
Security
|
From |
Enabling SASL SCRAM SHA-512 listener
Enable Kafka cluster SCRAM SHA-512 listener (Strimzi supports SCRAM-SHA-512 only), add a SCRAM SHA-512 listener configuration to every broker
axual-streaming:
kafka:
kafka:
scramsha512listener:
enabled: true # default false
listenerType: loadbalancer # default nodeport
listenerConfiguration:
bootstrap:
# To pass custom annotations on the bootstrap listener service
annotations: {}
brokers:
- broker: 0 # broker ID
# Advertised host the broker for the external listener
# In case of external loadbalancer, use your SASL broker host
# this is unique per broker
advertisedHost: "[advertisedHost]"
# Advertised port, default is 9095
advertisedPort: [advertisedPort]
# Only needed if `scramsha512listener.listenerType` is set to `nodeport`
# You must ensure that the port is free and available for use.
nodePort: [nodePort]
# To pass custom annotations on the external broker listener service
annotations: {}
- broker: 1 # same structure as the broker-0
# ...
- broker: 2 # same structure as the broker-0
# ...| For more information about the different types of listeners, check out this Strimzi documentation |
After enabling SASL on the brokers, enable the SASL SCRAM-512 functionality in self-service.