How to Enable Kafka Connect Log Reading

This guide shows you how to find the namespaces your Kafka Connect (KC) workers run in, switch log reading on in the Runtime Provisioner values, apply them to the running release, confirm the Provisioner can reach the worker pods, and note the URL Self-Service needs.

Type

How-to guide

Goal

Get connector logs from your KC workers into Self-Service through an already running Runtime Provisioner.

Audience

Platform Operator with Helm access to the Runtime Provisioner’s release and permission to create a Role and RoleBinding in every namespace where KC workers run.

When to use

Once per Kubernetes cluster, before registering any KC cluster in Self-Service. One Runtime Provisioner serves every KC cluster whose namespace it lists.

Contents

The sections below cover each task in this guide:

Where this guide fits

There is no separate log provisioner to deploy. From version 0.8.0 the Runtime Provisioner reads KC worker logs itself, on the same chart, the same Deployment and the same Service that provision KSML applications, and serves them at /kafkaconnect/logs. This guide switches that on for a Provisioner you already run.

Deploy the Provisioner first if you have not already, with How to Deploy the Runtime Provisioner. For the labels it matches worker pods on, see How the Log Provisioner finds connector logs. For every value this guide does not set, see Kafka Connect log reading.

Prerequisites

Confirm the following before you begin.

Access and permissions required

You need the following access and permissions:

  • Helm permission to upgrade the Runtime Provisioner release in its own namespace.

  • Permission to create a Role and RoleBinding in every namespace where KC workers run. The chart creates them for you; it cannot create them where you have no rights.

  • A Tenant Admin account in Self-Service, or a colleague holding one, to register the URL from Step 6 against each KC cluster.

Tools and versions required

You need the following tools:

  • helm >= 3.12.

  • kubectl >= 1.28.

Resources that must exist before starting

The following must already exist:

  • A running Runtime Provisioner at chart version 0.8.0 or higher. Earlier versions have no /kafkaconnect/logs endpoint at all.

  • KC workers deployed with their identity labels (axual.io/tenant, axual.io/instance, axual.io/connect-cluster), which How to Deploy a Kafka Connect Cluster sets.

  • The values file the Provisioner release runs with, as runtime-provisioner-values.yaml.

Replace every <VALUE> placeholder with your own value before running a command. Two namespaces matter here and they are rarely the same one: <PROVISIONER_NAMESPACE> holds the Runtime Provisioner, and <KC_WORKER_NAMESPACE> holds the KC worker pods.

Step 1: List the namespaces your KC workers run in

Find every namespace holding worker pods, because the Provisioner reads only the namespaces you name in Step 2.

kubectl get pods --all-namespaces -l axual.io/connect-cluster

The output lists one row per worker pod, with its namespace in the first column. Collect the distinct values from that column.

A pod missing the axual.io/connect-cluster label does not appear here, and the Provisioner cannot find it either. Fix the labels on the KC cluster first, as How to Deploy a Kafka Connect Cluster describes.

Step 2: Switch log reading on in the Provisioner values

Add the connect block to runtime-provisioner-values.yaml, listing every namespace from Step 1.

connect:
  enabled: true          (1)
  namespaces:            (2)
    - <KC_WORKER_NAMESPACE>
1 Defaults to false, which makes /kafkaconnect/logs answer 501. Setting it to true also creates the read-only Role and RoleBinding the next step applies.
2 Defaults to an empty list, which finds no pods, so /kafkaconnect/logs answers 404 once connect.enabled is true. Readiness checks these same namespaces, so name only ones the Provisioner’s ServiceAccount may read.

connect.namespaces is the tenancy boundary as well as the search list: the chart grants read permission for these namespaces and no others, so tenants who must not see each other’s logs need separate namespaces, or a Provisioner of their own.

For the read-window, container-name and download-cap settings, which all have working defaults, see Kafka Connect log reading.

Step 3: Apply the values to the running release

Upgrade the Provisioner release with the edited values file. The command is safe to re-run, because helm upgrade --install converges on the values you pass it.

helm registry login registry.axual.io --username <YOUR_USERNAME>

helm upgrade --install <RELEASE> oci://registry.axual.io/axual-charts/runtime-provisioner \
  --version <CHART_VERSION> \
  --namespace <PROVISIONER_NAMESPACE> \
  -f runtime-provisioner-values.yaml

Kafka Connect log reading arrives in chart version 0.8.0, so a release on an earlier version has to move to at least that version here. If the release already runs a later version, pass that version instead of downgrading it.

Helm replaces the pod and creates the Role and RoleBinding in each namespace you listed. To undo the change, run helm rollback <RELEASE> --namespace <PROVISIONER_NAMESPACE>.

Step 4: Confirm the Provisioner can read the worker namespaces

Find the Service, forward a local port to it, and call /readyz. This is the check worth running first, because readiness lists one pod in every namespace in connect.namespaces, so an ok proves the permissions landed.

kubectl get service --namespace <PROVISIONER_NAMESPACE> \
  -l app.kubernetes.io/name=runtime-provisioner

kubectl port-forward --namespace <PROVISIONER_NAMESPACE> \
  svc/<PROVISIONER_SERVICE> 8000:80

Use the Service name the first command printed as <PROVISIONER_SERVICE>. Port 80 is the chart default for service.port; use your own value if you changed it.

curl -s localhost:8000/readyz

A ready Provisioner answers {"status":"ok"}, and needs no credentials to do so.

A 503 means the ServiceAccount cannot list pods in at least one namespace in connect.namespaces. Either the Role did not reach that namespace, or the namespace name is wrong. Left unfixed, it surfaces later as an empty log window in Self-Service with no obvious cause.

Step 5: Confirm the log endpoint answers

Ask for a few lines from one KC cluster, over the port you forwarded in Step 4.

curl -s "http://localhost:8000/kafkaconnect/logs\
?tenant=<TENANT>\
&instance=<INSTANCE>\
&connectCluster=<CLUSTER_NAME>\
&tailLines=10"

Use the lowercase names the KC cluster’s chart values set as tenant, instance and clusterName, because those become the pod labels the Provisioner matches.

A working call answers with a JSON object of six fields, trimmed here to one entry:

{
  "tailLines": 10,
  "entries": [
    {
      "line": "{\"@timestamp\":\"2026-09-16T08:54:10.092Z\",\"ecs.version\":\"1.2.0\",\"log.level\":\"INFO\",\"message\":\"Sink task finished initialization\",\"log.logger\":\"org.apache.kafka.connect.runtime.WorkerSinkTask\"}",
      "origin": "<WORKER_POD>",
      "timestamp": "2026-09-16T08:54:10.092028739Z"
    }
  ],
  "linesRead": 40,
  "linesReturned": 10,
  "truncated": false,
  "degraded": []
}

entries is the answer: one time-ordered list merged across every worker of the cluster, each line naming the worker that wrote it. Each line holds the worker’s raw log entry, which the Kafka Connect chart writes as one ECS JSON object, trimmed here to a few fields. An empty entries list means the workers were found but have written nothing recently. A non-empty degraded list names workers the Provisioner could not read at all, which points back at the namespace list from Step 2.

Two status codes point at the two settings from Step 2. A 501 means connect.enabled is still false. A 404 means no worker pod matched, so check the namespace list against Step 1 and the cluster’s identity labels against what Self-Service knows.

Step 6: Note the Log Provisioner URL for Self-Service registration

Platform Manager reaches the Provisioner over the cluster network, so the address it stores is the in-cluster Service URL. Build it from the Service name and namespace you used in Step 4.

http://<PROVISIONER_SERVICE>.<PROVISIONER_NAMESPACE>.svc.cluster.local:80

Give this URL to the Tenant Admin as the Log Provisioner URL when registering each KC cluster, at step 7 of How to Deploy a Kafka Connect Cluster. The address is stored per KC cluster, so several clusters can point at one Provisioner, and a second Provisioner needs no extra work beyond its own URL.

Use your own ServiceAccount

Skip this section unless you supply the ServiceAccount yourself instead of letting the chart create it. Switch both toggles off and name yours:

serviceAccount:
  create: false
  name: <MY_SERVICE_ACCOUNT>
rbac:
  create: false

Creating the permissions then becomes your job. Reading logs needs get, list and watch on pods and pods/log in every namespace you listed in connect.namespaces, which is a subset of Required Kubernetes permissions. The remaining permissions in that table exist to deploy KSML applications through Helm and are not needed for log reading alone.