Enable Kafka Connect for an Instance

This guide shows you how to turn on Kafka Connect support for an Instance, and how to register, edit and delete the Kafka Connect clusters that connector applications deploy to.

Type

How-to guide

Goal

Give an Instance one or more registered Kafka Connect clusters, so application owners can deploy connectors to them.

Audience

Tenant Admin, a user who can configure an Instance settings. Editing a registered cluster is also open to members of the cluster’s Owner Group.

When to use

Use this guide when an Instance gains its first Kafka Connect cluster, or when a cluster’s address, credentials or log viewer change.

Contents

The sections below cover each task in this guide:

Prerequisites

To enable Kafka Connect for your Instance, you will need the following:

  1. a Kafka Connect runtime with the expected Axual Provided Single Message Transformers running and reachable from Platform Manager

    • The URL of the Kafka Connect REST interface

    • (Conditionally) Basic Authentication credentials

    • Truststore to be used for REST Client configuration

  2. a Log Provisioner running and reachable from Platform Manager, so connector logs reach Self-Service

  3. a Vault to manage Application Deployment credentials running and reachable from Platform Manager

    • The URL of the service

    • namespace

    • Approle Path/ID

    • Approle Secret ID

    • Vault path in which to store secrets

Configuring the Instance

Once the above is ready, the Tenant Admin is able to configure Kafka Connect for any Instance that is part of their Tenant.

  1. Go to the Instances page

  2. Select the Instance for which you are enabling the Kafka Connect Support and open the edit Instance form

  3. Under Kafka Connect Support, enable the Connect Support toggle

  4. Click Update Instance button

Now the Tenant Admin is able to configure the Instance Kafka Connect Clusters.

Registering a Kafka Connect cluster to the Instance Cluster

  1. Open the Instance overview page for the Instance configured above.

  2. From the Connect section of the page select the Instance Cluster for which you want to register a Kafka Connect cluster. Click View clusters

  3. Click + Register Connect Cluster

  4. Fill in the metadata:

    1. Name: use a name to refer to your cluster

    2. Description: provide a description for this cluster

    3. Owner Group: the Group that is able to manage the cluster

    4. Authorized Groups: (optional) select zero or more additional Groups that can deploy connectors to the cluster and view its connector logs

      1. To add a group, type its name in the Add Authorized Groups field and select it from the list. The group moves to the Authorized Groups list.

      2. To remove a group, click the close icon on its card.

        You can select only one Owner Group, and you cannot add it as an Authorized Group. The Add Authorized Groups field hides the Owner Group and every group already on the Authorized Groups list.
        For more on how the Owner Group and Authorized Groups control deployment and log access, see Group authorization for Kafka Connect clusters.

  5. Fill in the Kafka Connect REST API info:

    1. Connect REST url: the HTTPS URL of the Kafka Connect REST API route, which is the restApi.route.host the operator set on the chart (e.g. https://connect-analytics.internal). Use the route address, not the internal Service on port 8083: the chart’s NetworkPolicy blocks direct access to the Service. Self-Service uses this endpoint to validate connector configurations and manage connectors on the cluster.

    2. Connect API authentication: the authentication method used to connect to the REST API. Select No Auth if the endpoint is open, or Basic Auth to supply a username and password. It must match the chart’s restApi.route.authMethod: none for No Auth, basic for Basic Auth.

    3. Trust store certificate: (optional) upload a PEM certificate for TLS verification when connecting to the REST API.

  6. Click Validate to check connectivity to the Connect REST API.

  7. Fill in the Connect log viewer info:

    1. Connect log viewer URL: (optional) the address of the Log Provisioner that serves this cluster’s connector logs, taken from How to Enable Kafka Connect Log Reading. Self-Service reads connector logs through it. Leave it empty and the Logs button on a connector application of this cluster stays disabled.

      The Log Provisioner is normally reached on an in-cluster address such as http://axual-runtime-provisioner.axual.svc.cluster.local, so this field is commonly filled in with http://.

  8. Click Validate to check that the Log Provisioner answers for this cluster. The check asks it for a single log line, the same call the Logs button makes later, and reports Log viewer validated successfully when it works. The button stays disabled while the field is empty or malformed.

  9. Fill in the Kafka Connect to Kafka access info:

    1. Kafka authentication method: how connectors in this cluster authenticate to Kafka. Each cluster supports one mechanism, either mTLS or SASL SCRAM.

  10. Fill in the Vault info:

  1. Connector Vault URL: (required) the HTTPS URL of the Vault instance used to store connector credentials (e.g. https://vault.internal:8200).

  2. Vault path: (required) the path within Vault where connector secrets are written (e.g. connectors/production). This is a path prefix nested under the Vault engine mount, not the mount name itself. See Understanding the Vault KV v2 path structure.

  3. Vault AppRole path: (optional) the mount path of the AppRole auth method in Vault. Defaults to approle if not set.

  4. Vault namespace: (optional) the Vault namespace to use, if your Vault deployment uses namespaces.

  5. AppRole ID: (required) the AppRole role ID that Self-Service uses to write connector credentials into Vault.

  6. AppRole Secret ID: (required) the AppRole secret ID corresponding to the role above. This value is write-only and cannot be retrieved after saving.

    1. Click Validate to check connectivity to the Vault using the values provided above. If validation fails with a Vault write-capability error, see Self-Service cluster registration fails.

    2. Click Register Kafka Connect button

Kafka Connect support is now available for the Instance, you can start developing Kafka Connect applications.

Editing a Kafka Connect cluster

Once a Kafka Connect cluster is registered, the Tenant Admin and the members of the cluster’s Owner Group can update its configuration.

  1. Open the Instance overview page for the Instance that owns the cluster.

  2. From the Connect section of the page select the Instance Cluster. Click View clusters

  3. Click the name of the Kafka Connect cluster you want to edit, then click Edit Connect Cluster

  4. Change the fields you need to update:

    1. Name: cannot be changed after registration.

    2. Connect log viewer URL: can be added or changed at any time, so a cluster registered before the Log Provisioner existed can gain the log console without being re-registered. Clearing the field removes the stored address, which takes the log console away from every connector on the cluster. A cluster with no address set shows No log viewer configured. in the Connect log viewer section of its overview.

    3. Connect API password, AppRole ID, and AppRole Secret ID are write-only, so they load as empty fields. Leave a field empty to keep its current value in Governance Vault, or enter a new value to replace it.

  5. Saving does not require a successful validation: the platform automatically re-validates a section on save, but only if you changed a field in that section.

  6. Click Save Changes button

Only a Tenant Admin can transfer ownership of a cluster by changing its Owner Group. If a member of the Owner Group submits a change to the Owner Group, the platform rejects the update.
Every update to a Kafka Connect cluster is recorded in the Audit History, with the old and new value of each changed field.

Refreshing the cluster’s plugin list

Self-Service keeps its own list of the plugins a Kafka Connect cluster offers, and a connector application can only use a plugin that is on that list. Installing a plugin on the cluster does not update that list by itself. A plugin an operator has installed is therefore not selectable yet, and the cluster shows as unavailable with the reason Required plugin is not installed until the list is refreshed.

The list is refreshed in one of two ways:

  • Automatically, when the plugin scan runs. Platform Manager runs it on a schedule, set by the axual.systemmanagement.kafka-connect.plugins-scan.cron property (default 0 0 4 * * ?). Wait for that run if you are not in a hurry.

  • On demand, from the Plugins tab of the Kafka Connect cluster. Use this when a plugin has been installed and you want it available now.

To refresh the list yourself:

  1. Open the Instance overview page for the Instance that owns the cluster.

  2. From the Connect section of the page select the Instance Cluster. Click View clusters

  3. Click the name of the Kafka Connect cluster to open its detail page.

  4. Open the Plugins tab.

  5. Refresh the plugin list from that tab.

The tab then lists the plugins the cluster currently reports, and the new plugin can be selected the next time you create or edit a connector application.

Refreshing the plugin list is not restricted to Tenant Admins. Any user who can open the cluster can refresh it, because the refresh only re-reads what the cluster already reports and changes no cluster configuration.
A refresh is recorded in the Audit History as Connect Cluster Plugins Refreshed.

Deleting a Kafka Connect cluster

A Tenant Admin can remove a Kafka Connect cluster that is no longer needed, once no Connector application deployment still targets it.

  1. Open the Instance overview page for the Instance that owns the cluster.

  2. From the Connect section of the page select the Instance Cluster. Click View clusters

  3. Click the name of the Kafka Connect cluster you want to delete to open its detail page.

  4. Click Delete.

    Only a Tenant Admin sees the Delete button. Unlike editing, membership of the cluster’s Owner Group does not grant delete rights.

  5. Review the confirmation dialog:

    1. If no Connector deployment targets the cluster, click Delete to confirm.

    2. If one or more Connector deployments still target the cluster, in any deployment state, the dialog lists each one under Connector deployments by application name and environment. The Delete button in the dialog stays disabled until you remove every listed deployment. The platform does not stop or remove deployments on your behalf.

Deleting a Kafka Connect cluster cannot be undone. It also removes the cluster’s AppRole credentials from the Governance Vault.
Cluster deletion is recorded in the Audit History.

Optional configurations

The platform provides default settings that allow you to start using Kafka Connect immediately without any additional configuration. However, depending on your operational requirements, you may want to customize certain aspects of how Kafka Connect Applications are deployed and managed.

The following configurations are optional but give greater control over Kafka Connect Application Deployments: