How to Set Up the Connector Vault for Kafka Connect

This guide shows you how to prepare the Connector Vault for a Kafka Connect cluster: the KV secret engine, the AppRole the worker authenticates with, the policies that bound what it can read, and the probe secret it checks at startup.

Type

How-to guide

Goal

Give a Kafka Connect cluster the Vault credentials its connectors read at runtime.

Audience

Platform Operator with a Vault admin token for the Connector Vault, which is the data-plane Vault.

When to use

Use this guide before deploying a new Kafka Connect cluster.

Contents

The sections below cover each task in this guide:

Where this guide fits

This guide covers the Connector Vault only, and the Vault objects one Kafka Connect cluster needs in it.

On Axual Cloud, Platform Manager already has its own Vault AppRole from the platform installation, so you create only the worker AppRole here, and extend the Platform Manager policy only if it does not already cover the new cluster’s paths. This guide operates on the Connector Vault alone.

For background, see Understanding the two-Vault model and Understanding the Vault KV v2 path structure.

Prerequisites

Confirm the following before you begin.

Access and permissions required

You need the following access and permissions:

  • Vault admin token for the Connector Vault (data plane Vault). On Axual Cloud this is the Vault root token from the governance umbrella chart.

  • Network access to the Connector Vault from your workstation, or use kubectl port-forward.

Tools required

You need the following tools:

  • vault CLI >= 1.14, or access to the Vault UI.

Resources that must exist before starting

The following resources must already exist before you start:

  • The Connector Vault is running and reachable from the data plane. On Axual Cloud this is the bundled Vault instance in the data plane cluster.

Step 1: Log in to the Connector Vault

Every command in this guide runs against the Connector Vault, so authenticate to it once and confirm which Vault you are pointed at before creating anything.

Replace every <…​> placeholder with your actual value before running a command. <tenant>, <instance>, and <cluster-name> are placeholders for the cluster’s real short names (for example axual, dev, my-cluster); substitute them in every command, policy, AppRole name, and path. <connector-vault-path> is the Vault path holding this cluster’s connector credentials. You choose it in step 2. The rest (<vault-host>, <admin-token>, <ns>, and so on) are your infrastructure values.
Vault UI

  1. Open https://<vault-host>/ui in a browser.

  2. Sign in with the admin token.

Vault CLI
export VAULT_ADDR=https://<vault-host>:8200
vault login <admin-token>

If Vault runs inside the cluster, port-forward first:

kubectl port-forward svc/<vault-service> 8200:8200 --namespace <vault-ns>
export VAULT_ADDR=http://127.0.0.1:8200
vault login <admin-token>

When port-forward is not available (restricted networks, continuous integration (CI) runners), run the vault commands directly inside the Vault pod instead:

kubectl exec -n <vault-ns> -it <vault-pod> -- vault login <admin-token>

Step 2: Enable the KV v2 secret engine (once per Vault)

The connector-auth Key-Value version 2 (KV v2) engine is shared by all Kafka Connect clusters on the same Vault. connector-auth is Platform Manager’s default mount name; if the operator set axual.systemmanagement.kafka-connect.vault.engine-path to a different value, substitute that value for connector-auth everywhere in this guide (see Understanding the Vault KV v2 path structure).

Skip this step if a previous cluster already created the connector-auth engine.
Choose the cluster’s connector-vault-path now

Under that shared mount, each cluster keeps its connector credentials at its own path. Choose it now: every path in this guide sits under it, and the Tenant Admin needs it to register the cluster in step 7 of How to Deploy a Kafka Connect Cluster. Any path works; connectors/<cluster-name> is a reasonable default.

Use the path on its own. Platform Manager adds the engine mount in front of it and <tenant>/<instance>/<cluster-name> after it, so including either yourself produces a doubled path that none of the policies below match.

Clusters may share a path, but the policies below grant access to all of it, so every cluster sharing it can read and write the others' connector credentials. Give each cluster its own path to keep them separate.

This guide writes it as <connector-vault-path>; substitute the value you chose everywhere it appears.
Vault UI

Secrets engines > Enable new engine + > KV > Version 2 > Path connector-auth > Enable Engine.

Vault CLI
vault secrets list -format=json | grep -q '"connector-auth/"' \
  || vault secrets enable -path=connector-auth kv-v2

This command is idempotent: it enables the engine only if it does not already exist.

Step 3: Enable AppRole authentication (once per Vault)

AppRole is the Vault authentication method that allows applications to authenticate with a role ID and secret ID pair. This engine is shared across all clusters on the same Vault.

Skip this step if AppRole is already enabled.
Vault UI

Access > Enable new method + > AppRole > path approle > Enable Method.

Vault CLI
vault auth list -format=json | grep -q '"approle/"' \
  || vault auth enable approle

This command is idempotent: it enables AppRole only if it does not already exist.

Step 4: Confirm or extend the Platform Manager writer policy

On Axual Cloud, Platform Manager authenticates to the Connector Vault with an AppRole provisioned when the platform was installed. Those credentials live in Platform Manager’s own configuration, set through the vault block described in Platform Manager Vault configuration, not in this guide. The AppRole carries an Access Control List (ACL) policy that grants write access to the connector-auth paths so Platform Manager can write, soft-delete, and destroy connector credentials on behalf of App Owners. Soft-delete and destroy are separate KV v2 sub-paths from data and metadata, so the policy needs explicit delete and destroy path grants alongside them; see Understanding the Vault KV v2 path structure.

Inspect the existing policy (its name is set at platform installation, for example platform-manager):

vault policy read <pm-policy-name>

If the policy is a wildcard over the whole engine, it already covers the new cluster and you do nothing:

path "connector-auth/data/*"     { capabilities = ["create", "read", "update"] }
path "connector-auth/metadata/*" { capabilities = ["list", "delete"] }
path "connector-auth/delete/*"   { capabilities = ["update"] }
path "connector-auth/destroy/*"  { capabilities = ["update"] }

If the policy is scoped to explicit paths instead, add the new cluster’s writer paths.

vault policy write <kc-policy-name> - <<'EOF'
# keep the policy's existing paths here
path "connector-auth/data/<connector-vault-path>/*"     { capabilities = ["create", "read", "update"] }
path "connector-auth/metadata/<connector-vault-path>/*" { capabilities = ["list", "delete"] }
path "connector-auth/delete/<connector-vault-path>/*"   { capabilities = ["update"] }
path "connector-auth/destroy/<connector-vault-path>/*"  { capabilities = ["update"] }
EOF

Grant all four paths with every capability shown. The registration pre-flight check only exercises write access, so a policy that grants create and update but omits read passes registration and then fails later, the first time a connector credential has to be read back.

vault policy write replaces the entire policy body. Include the existing paths, or Platform Manager loses write access to every other cluster.
Grant on <connector-vault-path>/, exactly as the policy bodies above do, in this guide and in any policy you write later. Two near-misses both break the cluster: a grant on the bare path with no trailing / covers nothing, because Vault matches these patterns literally; and a grant that reaches further down, naming the tenant, instance or cluster, fails the check Platform Manager runs when the Tenant Admin registers the cluster, because that check looks just below <connector-vault-path>. See Self-Service cluster registration fails for the error either one produces.

Strict on-premises with two separate Vaults. When the Connector Vault is a different server from the Governance Vault, Platform Manager has no AppRole in the Connector Vault. Create a dedicated writer policy and AppRole here, then hand its role_id and secret_id to the Tenant Admin for the Self-Service registration form. Platform Manager stores them in the Governance Vault. The writer policy below already uses the required nested-glob form; see the IMPORTANT admonition above if you scope it to different paths.

vault policy write <tenant>-<instance>-<cluster-name>-connector-writer - <<'EOF'
path "connector-auth/data/<connector-vault-path>/*"     { capabilities = ["create", "read", "update"] }
path "connector-auth/metadata/<connector-vault-path>/*" { capabilities = ["list", "delete"] }
path "connector-auth/delete/<connector-vault-path>/*"   { capabilities = ["update"] }
path "connector-auth/destroy/<connector-vault-path>/*"  { capabilities = ["update"] }
EOF
vault write auth/approle/role/<tenant>-<instance>-<cluster-name>-connector-writer \
  token_policies=<tenant>-<instance>-<cluster-name>-connector-writer \
  token_ttl=1h token_max_ttl=24h
vault read  auth/approle/role/<tenant>-<instance>-<cluster-name>-connector-writer/role-id
vault write -f auth/approle/role/<tenant>-<instance>-<cluster-name>-connector-writer/secret-id

Step 5: Create the worker reader policy

This ACL policy allows the Kafka Connect worker to read connector credentials at connector startup.

Vault UI

Policies > Create ACL policy + > name <tenant>-<instance>-<cluster-name>-connect > paste the policy body > Create policy.

Vault CLI
vault policy write <tenant>-<instance>-<cluster-name>-connect - <<'EOF'
path "connector-auth/data/<connector-vault-path>/*"     { capabilities = ["read"] }
path "connector-auth/metadata/<connector-vault-path>/*" { capabilities = ["read", "list"] }
EOF

The worker only ever reads, so this policy stays read-only.

Step 6: Create the worker AppRole and collect its credentials

Run the following commands in the Vault CLI terminal or in the Vault UI console (>_ icon, top right).

# Worker AppRole: non-expiring secret ID because the worker is a long-running process
vault write auth/approle/role/<tenant>-<instance>-<cluster-name>-connect \
  token_policies=<tenant>-<instance>-<cluster-name>-connect \
  token_ttl=1h token_max_ttl=24h \
  secret_id_ttl=0 secret_id_num_uses=0
vault read  auth/approle/role/<tenant>-<instance>-<cluster-name>-connect/role-id
vault write -f auth/approle/role/<tenant>-<instance>-<cluster-name>-connect/secret-id

The vault read call outputs a role_id field. The vault write -f call outputs a secret_id field.

Record both values immediately:

  • Worker role_id

  • Worker secret_id

Never paste a secret_id into a log, chat message, or shell history. Copy it directly from the terminal into the Kubernetes Secret (step 9).
Output Destination

Worker role_id + secret_id

Used in step 9 to create the Kubernetes Secret for the chart. Platform Manager never sees these values.

The <connector-vault-path> chosen in step 2

Handed to the Tenant Admin as the connectorVaultPath registration field, in step 7 of How to Deploy a Kafka Connect Cluster.

Probe secret path (step 7)

Used as vault.testPath in the chart values.

Truststore Secret name (step 8, if applicable)

Used as vault.truststoreSecret in the chart values.

Step 7: Write the Vault probe secret

The worker reads this secret at startup when vault.testPath is set in the chart values. A broken Vault setup fails at pod startup rather than at first connector deployment.

vault kv put connector-auth/<connector-vault-path>/<tenant>/<instance>/<cluster-name>/test/probe value=ok

Step 8: Build the Vault CA truststore (HTTPS with a private CA only)

Skip this step if your Vault uses a public CA or plain HTTP.

The Vault server certificate must have a Subject Alternative Name (SAN) matching the vault.address host you set in the chart. A mismatched SAN causes the TLS handshake to fail regardless of the truststore. 
TSPW=$(openssl rand -hex 16)   (1)

keytool -importcert -noprompt -alias root-ca \
  -file root-ca.crt \
  -keystore vault-truststore.p12 -storetype PKCS12 -storepass "$TSPW"

keytool -importcert -noprompt -alias intermediate-ca \
  -file intermediate-ca.crt \
  -keystore vault-truststore.p12 -storetype PKCS12 -storepass "$TSPW"

kubectl create secret generic <cluster-name>-vault-truststore \
  --from-file=truststore.p12=vault-truststore.p12 \
  --from-literal=truststore.password="$TSPW" \
  --namespace <ns>
1 Generate a random truststore integrity password. Store it only in the Kubernetes Secret. Do not commit it.
This command is not idempotent. If the Secret already exists, delete it first with kubectl delete secret <cluster-name>-vault-truststore --namespace <ns> --ignore-not-found.

Step 9: Create the Kubernetes Secret for the worker AppRole

Store the worker role_id and secret_id in the Secret the chart reads at deploy time.

kubectl create secret generic <cluster-name>-vault-creds \
  --from-literal=VAULT_APPROLE_ROLE_ID=<worker-role-id> \
  --from-literal=VAULT_APPROLE_SECRET_ID=<worker-secret-id> \
  --namespace <ns>
This command is not idempotent. If the Secret already exists, delete it first: 
kubectl delete secret <cluster-name>-vault-creds --namespace <ns> --ignore-not-found

Verify the setup

Confirm the worker AppRole is readable before handing credentials to the deploy how-to:

vault read auth/approle/role/<tenant>-<instance>-<cluster-name>-connect/role-id

Expected output:

Key        Value
---        -----
role_id    <uuid>

If the role_id field appears, the AppRole is correctly registered.

Next steps

Return to How to Deploy a Kafka Connect Cluster. In that guide, the "Wire Vault into the chart values" step uses the Secret name from step 9, and vault.testPath uses the probe secret path from step 7.

When the cluster is being removed instead, the AppRole, the policy and the connector credentials created here are deleted in Stage 2: Remove the cluster’s Vault objects. Platform Manager’s own AppRole is shared by every cluster and stays.