How to Deploy a Kafka Connect Cluster

This guide shows you how to deploy a Kafka Connect cluster end to end: writing its values file, creating the broker credential Secrets, wiring the Connector Vault, installing the chart, verifying the worker, and registering the cluster in Self-Service.

Type

How-to guide

Goal

Deploy a Kafka Connect cluster and register it so App Owners can create connectors on it.

Audience

Platform Operator with Helm access to the target namespace and a Vault admin token.

When to use

Use this guide when adding a Kafka Connect cluster to an existing Axual instance.

Contents

The sections below cover each task in this guide:

Where this guide fits

This guide is the operator procedure for standing up one cluster and handing it to a Tenant Admin. Two companion pages carry the material it does not repeat.

For background on how Kafka Connect is structured, see Kafka Connect: Concepts and Architecture. For every chart value, see the Kafka Connect Helm Readme, and Kafka Connect Chart Values Reference for the constraints and Self-Service mapping it doesn’t cover.

Prerequisites

Confirm the following before you begin.

Access and permissions required

You need the following access and permissions:

  • cluster-admin (or equivalent) on the target Kubernetes namespace.

  • Read access to the kafka-connect-helm chart repository.

  • Harbor credentials for the internal project (development images only; not required for production).

Tools and versions required

You need the following tools and versions:

  • helm >= 3.12

  • kubectl >= 1.28

  • Kubernetes >= 1.21 (>= 1.31 for the default imageVolumes plugin delivery mode)

  • Strimzi cluster operator >= 0.47 (>= 0.51.0 recommended) installed and watching the target namespace; it must serve the kafka.strimzi.io/v1 API, which the chart renders the KafkaConnect resource with

Prerequisite how-tos to complete first

Complete these two prerequisite guides before Step 1:

  1. How to Prepare the Kafka Broker for Kafka Connect creates the internal topics and grants the worker Access Control List (ACL) permissions.

  2. How to Set Up the Connector Vault for Kafka Connect creates the worker read policy and AppRole, the worker credentials Kubernetes Secret, and confirms Platform Manager’s write access.

Resources that must exist before starting

The following resources must already exist before you start:

  • The worker mutual TLS (mTLS) client-cert Secret or the Simple Authentication and Security Layer (SASL)/SCRAM-512 password Secret in the target namespace.

  • The broker CA certificate Secret in the target namespace.

  • The Harbor image pull Secret (regcred-harbor) in the target namespace (development only).

  • The worker credentials Kubernetes Secret created in step 9 of the Vault setup how-to.

Replace every <VALUE> placeholder with your actual value before running a command. <tenant>, <instance>, and <cluster-name> are placeholders for the cluster’s real short names (for example axual, dev, my-cluster). Substitute them in every command, policy, AppRole name, and path; they are stored and matched exactly as written. <connector-vault-path> is the Vault path holding this cluster’s connector credentials, chosen in step 2 of How to Set Up the Connector Vault for Kafka Connect.

Step 1: Create your cluster values file

Create a connect-values.yaml file. Build it section by section from the blocks below.

Set cluster naming and identity labels

Name the cluster and set the identity labels that tie it to its tenant and instance.

fullnameOverride: "<cluster-name>"   (1)

tenant:      "<tenant-short-name>"   (2)
instance:    "<instance-short-name>"
clusterName: "<cluster-name>"        (3)
1 Pins the KafkaConnect Custom Resource (CR) name and the REST API service name. Use the same value as clusterName.
2 Use the short name from Self-Service, not the display name.
3 Must be lowercase and match exactly what you will register in Self-Service.
Keep tenant, instance, and clusterName in the values file, not as --set flags. Under GitOps (ArgoCD, Flux) the values file is the desired state, so --set flags are not tracked. The identity labels then drift on the next reconcile, which breaks Log Provisioner pod discovery. Use --set for these values only in a throwaway manual install.

Check the internal topic names and group ID

Leave groupId, configStorageTopic, offsetStorageTopic, and statusStorageTopic out of the values file. The chart derives all four from the identity labels, so two clusters on the same Kafka broker never share them:

groupId             _<tenant>-<instance>-<clusterName>-connect
configStorageTopic  _<tenant>-<instance>-<clusterName>-connect-configs
offsetStorageTopic  _<tenant>-<instance>-<clusterName>-connect-offsets
statusStorageTopic  _<tenant>-<instance>-<clusterName>-connect-status

Set a key only to adopt state that already lives under other names; the chart then uses that value verbatim. These four values must be unique across every Kafka Connect (KC) cluster on the same Kafka broker. On a running cluster, helm upgrade refuses any change to tenant, instance, clusterName, or these four keys, because renaming the topics loses the worker’s connector configurations, offsets, and status.

Configure the broker connection

Set only the bootstrapServers key that matches your worker authentication type.

mTLS worker (authentication.type: tls)
bootstrapServers:
  tls: "<broker-bootstrap-host>:9093"   (1)

tls:
  trustedCertificates:
    - secretName: <broker-ca-secret>
      certificate: ca.crt
1 Confirm the exact port with your broker configuration. Axual clusters may differ from the standard :9093.
SASL/SCRAM-512 worker (authentication.type: scram-sha-512)
bootstrapServers:
  sasl: "<broker-bootstrap-host>:9094"  (1)

tls:
  trustedCertificates:
    - secretName: <broker-ca-secret>
      certificate: ca.crt
1 The broker always uses TLS on the wire. tls.trustedCertificates is required for both authentication types.

Configure worker authentication

Add the block that matches how the worker authenticates to the broker.

Authenticate with mTLS (type: tls)
authentication:
  type: tls
  certificateAndKey:
    secretName: <worker-cert-secret>
    certificate: tls.crt
    key: tls.key
Authenticate with SASL/SCRAM-512 (type: scram-sha-512)
authentication:
  type: scram-sha-512
  username: <worker-scram-username>
  passwordSecret:
    secretName: <worker-scram-username>   (1)
    password: password                    (2)
1 The Strimzi User Operator creates a Secret with the same name as the KafkaUser CR.
2 The key within the Secret. password is the Strimzi User Operator default.

On Strimzi-managed Kafka, the password Secret comes from a KafkaUser CR for the worker, so apply one if it does not exist yet:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: <worker-scram-username>
  namespace: <ns>
  labels:
    strimzi.io/cluster: <kafka-cluster-name>
spec:
  authentication:
    type: scram-sha-512

Wait for the password Secret to be ready before continuing:

kubectl wait kafkauser/<worker-scram-username> \
  --for=condition=Ready \
  --namespace <ns> \
  --timeout=120s

Configure plugins

List the connector and transform plugins the workers load at startup.

pluginRegistry: "registry.axual.io/internal/axual/connect-plugins"
plugins:
  - kafka-topic-name-transforms   (1)
  - name: aiven-jdbc              (2)
    image: registry.axual.io/internal/axual/connect-plugins/aiven-jdbc:6.12.0
  # See the reference for all three entry forms.
1 Required. Platform Manager validates that this plugin exists during cluster registration. String form: the image is resolved from pluginRegistry/<name>:<version> using the catalogue spec at plugins/<name>.yaml.
2 Explicit-image form for a plugin not in the kafka-connect-helm/plugins catalogue, or whose image lives in a different registry. Give name and the full image reference.
Loading a plugin onto the workers does not make it selectable in Self-Service on its own. Self-Service keeps its own copy of the cluster’s plugin list, which is refreshed on a schedule or on demand. Whenever you add a plugin to a cluster that is already registered, tell the tenant to refresh that list, or the plugin stays unavailable until the next scheduled scan. See Refreshing the cluster’s plugin list.

Configure replication factors

Set the replication factor for the three internal topics.

config:
  offset.storage.replication.factor: 3   (1)
  config.storage.replication.factor: 3
  status.storage.replication.factor: 3
1 Use 1 for a single-broker test cluster. Match your broker’s replication factor for production.

Secure the REST API

The REST API is plain HTTP with no password of its own. Two values put a check back: a route, and a NetworkPolicy making that route the only way in.

restApi:
  route:
    enabled: true
    kind: Ingress                        # or HTTPRoute for Gateway API implementations
    host: "<connect-host>"
    authMethod: basic                    # or none, matching Platform Manager's NO_AUTH
    className: "<ingress-class>"         # kind: Ingress only
    tlsSecret: "<wildcard-tls-secret>"   # kind: Ingress only
  basicAuth:
    htpasswd: "<htpasswd -nbB connect '<password>'>"   # SOPS-encrypt this file
    annotations:                         (1)
      nginx.ingress.kubernetes.io/auth-type: basic
      nginx.ingress.kubernetes.io/auth-secret: '{{ include "kafka-connect.restApi.authSecretName" . }}'
      nginx.ingress.kubernetes.io/auth-realm: "Kafka Connect REST API"
  networkPolicy:
    enabled: true
    router:
      namespace: "<router-namespace>"    # the ingress controller or Gateway data plane, not Platform Manager
      podLabels:
        <label-key>: "<label-value>"
    strimziOperator:
      namespace: "<strimzi-operator-namespace>"   (2)
1 Required with authMethod: basic. htpasswd only creates the credentials Secret; these annotations are what makes community ingress-nginx read it. Without them the chart refuses to render, because nothing would carry out the check. For F5 NGINX Ingress Controller and NGINX Gateway Fabric, use the matching block in The chart does not know your ingress implementation.
2 Only when the Strimzi operator runs outside the Connect cluster’s namespace, which is the usual case for a cluster-wide installation. Leave the key out when the two share a namespace.

Find the real namespaces and labels rather than guessing, a wrong selector blocks a caller instead of leaking anything:

  • The router: kubectl get pod -A -l app.kubernetes.io/name=ingress-nginx --show-labels for an ingress controller, kubectl get pod -n <gateway-namespace> --show-labels for a Gateway data plane.

  • The Strimzi operator: kubectl get pod -A -l strimzi.io/kind=cluster-operator --show-labels. It calls the REST API on every reconcile, and an empty strimziOperator.namespace allows it from the Connect cluster’s own namespace only. Miss this and the policy blocks the operator, leaving the KafkaConnect resource stuck short of Ready with nothing wrong in the worker itself.

Platform Manager is deliberately not a peer in networkPolicy.router or in allowedFrom: it reaches the API through this same authenticated route, not around it. See REST API access values for every restApi.* key, and The chart does not know your ingress implementation for a working block per ingress implementation (community ingress-nginx, F5 NGINX Ingress Controller, NGINX Gateway Fabric).

Step 2: Create Kubernetes Secrets for broker credentials

Create the Secrets the chart references for broker access.

mTLS worker certificate
kubectl create secret generic <worker-cert-secret> \
  --from-file=tls.crt=worker.crt \
  --from-file=tls.key=worker.key \
  --namespace <ns>
Harbor image pull Secret (development images only)
kubectl create secret docker-registry regcred-harbor \
  --docker-server=registry.axual.io \
  --docker-username=<harbor-username> \
  --docker-password=<harbor-password> \
  --namespace <ns>

A public Harbor project serves the production images, so production needs no pull Secret.

Step 3: Add the ACL bootstrap block (automated broker path)

You prepared the broker as a prerequisite (How to Prepare the Kafka Broker for Kafka Connect), which created the internal topics and granted the worker ACLs.

If you chose the aclBootstrap Helm hook (the automated path in that guide), add the aclBootstrap block to your connect-values.yaml now. The hook runs automatically during helm upgrade --install in step 5.

Step 4: Wire the Connector Vault into the chart values

You set up the Connector Vault as a prerequisite (How to Set Up the Connector Vault for Kafka Connect), which created the worker credentials Kubernetes Secret <cluster-name>-vault-creds.

Add the following Vault block to your connect-values.yaml:

vault:
  enabled:           true
  address:           "https://<vault-host>:8200"
  authMethod:        APPROLE
  approlePath:       approle
  sslVerify:         true
  credentialsSecret: "<cluster-name>-vault-creds"   (1)
  testPath:          "connector-auth/<connector-vault-path>/<tenant>/<instance>/<cluster-name>/test/probe"  (2)
1 The Secret created in step 9 of the Vault setup how-to.
2 The probe secret path from step 7 of the Vault setup how-to. The worker uses this to verify Vault connectivity at pod startup. For <connector-vault-path>, see Understanding the Vault KV v2 path structure.

On Vault Enterprise with namespaces, also set vault.namespace; leave it out for Vault OSS, which Axual Cloud runs.

HTTPS with a private CA: add the truststore reference
vault:
  truststoreSecret: "<cluster-name>-vault-truststore"   (1)
1 The Secret created in step 8 of the Vault setup how-to.
Plain HTTP Vault: no truststore needed

Omit vault.truststoreSecret and set vault.sslVerify: false.

The Vault enabled flag, the credentials Secret, and (if applicable) the truststore Secret must all exist before you install the chart. Installing with vault.enabled: true but without the credentials Secret causes the worker to fail at startup. The chart refuses to render any other vault.* value while vault.enabled is false.

Step 5: Install the chart

Install or upgrade the cluster with your values file.

helm upgrade --install <CLUSTER_NAME> \
  oci://registry.axual.io/axual-charts/kafka-connect \
  --version 0.7.0 \
  --namespace <NAMESPACE> \
  --create-namespace \
  -f connect-values.yaml
For development images, add --set 'imagePullSecrets[0].name=regcred-harbor' to the command above. Do not pass tenant, instance, or clusterName with --set under GitOps; keep them in the values file (see Set cluster naming and identity labels).

helm upgrade --install is idempotent and safe to re-run. To roll back to the previous revision:

helm rollback <CLUSTER_NAME> --namespace <NAMESPACE>

Step 6: Verify the worker is Ready

Confirm the worker reached the Ready state and loaded its plugins.

kubectl wait kafkaconnect/<cluster-name> \
  --for=condition=Ready \
  --namespace <ns> \
  --timeout=120s

Expected output:

kafkaconnect.kafka.strimzi.io/<cluster-name> condition met

Check that the worker pod is running:

kubectl get pods \
  -l strimzi.io/cluster=<cluster-name> \
  --namespace <ns>

Expected: one pod per replica in Running state with 1/1 ready.

Verify Vault connectivity (required when vault.testPath is set):

kubectl logs \
  -l strimzi.io/cluster=<cluster-name> \
  --namespace <ns> \
  | grep -iE "Retrieved test data from|invalid role or secret ID"

Expected: a line containing Retrieved test data from connector-auth/…​/test/probe.

Verify that plugins are loaded:

POD=$(kubectl get pod \
  -l strimzi.io/cluster=<cluster-name> \
  --namespace <ns> \
  -o name | head -1)

kubectl exec $POD --namespace <ns> -- \
  curl -s "localhost:8083/connector-plugins?connectorsOnly=false" | jq '.[].class'

Confirm kafka-topic-name-transforms appears in the output. connectorsOnly=false is required because kafka-topic-name-transforms is a Single Message Transform (SMT), not a connector, and the default plugin list returns connectors only.

Verify the REST API route asks for the password and the NetworkPolicy blocks every other way in. The chart only checks that authentication is wired, not that it works, so test the live route:

curl -o /dev/null -s -w '%{http_code}\n' https://<connect-host>/connectors                            # 401
curl -o /dev/null -s -w '%{http_code}\n' -u connect:'<password>' https://<connect-host>/connectors     # 200

kubectl run probe --rm -it --image=curlimages/curl --restart=Never --namespace <ns> -- \
  curl -m 5 http://<cluster-name>-connect-api.<ns>.svc:8083/connectors                              # times out

A 200 on the first call means nothing checks the password. A reply on the last call means the NetworkPolicy is not enforced, for example on a CNI without NetworkPolicy support. With authMethod: none, expect 200 on the first call and skip the second.

A Ready worker with its plugins listed completes the operator’s verification. This does not prove end-to-end data flow, and you cannot prove it with a throwaway connector at this stage. On a governed cluster the worker has no authority to create a topic or to write to an arbitrary one. Self-Service and Platform Manager provision each connector’s target topic and grant its Write access under governance (see How to Prepare the Kafka Broker for Kafka Connect). Data flow is proven by the first connector an App Owner deploys through Self-Service on this cluster.

Step 7: Register the cluster in Self-Service

Give the Tenant Admin the following items for the Self-Service cluster registration form. Hand over the Vault path exactly as you scoped the policies to it: a value that is registered differently resolves to paths no policy grants.

Item Note

KC REST URL

The route address set as restApi.route.host, not the internal Service (for example https://connect-<cluster>.<env>.axual.cloud).

Connect API authentication

basic or none, matching restApi.route.authMethod. Selecting Basic Auth in Self-Service asks for the username and password used to generate restApi.basicAuth.htpasswd.

Worker authentication method

tls or scram-sha-512. Must match authentication.type in values.

Connector Vault URL, key-value (KV) path prefix, AppRole mount path, Vault namespace

From the Vault setup how-to. KV path prefix: the <connector-vault-path> alone, for example connectors/<cluster-name> (Platform Manager appends <tenant>/<instance>/<cluster-name> itself, giving connectors/<cluster-name>/<tenant>/<instance>/<cluster-name>).

Platform Manager writer role_id and secret_id (strict on-premises only)

Only when the Connector Vault is a separate server from the Governance Vault. On Axual Cloud, Platform Manager uses its own AppRole, so you skip this item. See step 4 of the Vault setup how-to.

Log Provisioner URL

The URL of the Log Provisioner deployed for this Kubernetes cluster. It is entered in Self-Service as Connect log viewer URL. See How to Enable Kafka Connect Log Reading.
On strict on-premises deployments that pass writer credentials, do not send role_id or secret_id values by email or chat. Use a secrets manager or a secure channel.

Platform Manager validates reachability, Vault credentials, and the presence of kafka-topic-name-transforms. If validation fails, see Troubleshoot a Kafka Connect Deployment.