How to Load Remote Cluster TLS Material from Kubernetes Secrets

This guide shows you how to mount a Kubernetes Secret holding a remote cluster’s certificate and key into the Axual Distributor pod, and how to point the cluster’s client configuration at the mounted files instead of inlining PEM data in values.yaml.

Type

How-to guide

Goal

Keep remote cluster certificates and private keys in Kubernetes Secrets rather than in the Axual Distributor’s values.yaml.

Audience

Platform Operator who can create Secrets in the Axual Distributor’s namespace and edit its values.yaml.

When to use

Use this guide when the Axual Distributor’s values.yaml is stored in git, where inlined private keys do not belong.

By default a remote cluster’s TLS material goes into distribution.clusters.<name>.tls as PEM data. That puts a private key in the values file. Mounting a Secret instead keeps the key out of it, at the cost of two extra pieces of configuration.

Contents

The sections below cover each task in this guide:

Prerequisites

Confirm the following before you begin.

Access and permissions required

You need the following access and permissions:

  • Permission to create Secrets in the namespace the Axual Distributor runs in.

  • Permission to edit the Axual Distributor’s values.yaml and apply it.

Tools and versions required

You need the following tools:

  • helm >= 3.12, or access to the pipeline that applies the values.

  • kubectl >= 1.28.

Resources that must exist before starting

The following must already exist:

  • One Secret per remote cluster, holding the client certificate chain, the private key and the trusted certificate authorities, all in PEM format.

  • An Axual Distributor deployment with its distribution.clusters section already written. See How to Deploy the Axual Distributor.

The Secret has to exist before the Axual Distributor starts. Kafka Connect resolves these paths at startup, so a missing Secret is a startup failure rather than a warning.

Mount the Secret into the pod

Add each Secret to connect.externalConfiguration. That is the Strimzi mechanism for getting a ConfigMap or Secret into a Kafka Connect pod as a volume or an environment variable. Strimzi mounts each volume under /opt/kafka/external-configuration/<volume-name>.

connect:
  # You can mount K8S ConfigMaps or Secrets into a distributor pod as environment variables or volumes. Volumes and environment variables are configured in the externalConfiguration property
  # for full documentation visit: https://strimzi.io/docs/operators/latest/configuring.html#type-ExternalConfiguration-reference
  externalConfiguration:
    volumes:
      - name: distribution-secret-cluster02
        secret:
          secretName: <distribution-secret-name>
      - name: distribution-secret-cluster03
        secret:
          secretName: <distribution-secret-name>

The Strimzi ExternalConfiguration reference covers the rest of its fields.

Read the mounted files from the client config

Register Apache Kafka’s DirectoryConfigProvider on the cluster’s additionalClientConfigs, then reference the mounted files through it. The placeholder syntax is ${directory:PATH:FILE-NAME}.

Empty the cluster’s tls block in the same edit. Left populated, it competes with the values resolved from the mounted Secret.

distribution:
  clusters:
    cluster02:
      # Skipping all other settings to illustrate needed settings
      additionalClientConfigs:
        # define the DirectoryConfigProvider
        config.providers: directory
        config.providers.directory.class: org.apache.kafka.common.config.provider.DirectoryConfigProvider
        # The placeholder structure is directory:PATH:FILE-NAME. DirectoryConfigProvider reads and extracts the credentials from the mounted Secret in schema distributor configurations.
        ssl.keystore.certificate.chain: "${directory:/opt/kafka/external-configuration/distribution-secret-cluster02:cert-chain.pem}"
        ssl.keystore.key: "${directory:/opt/kafka/external-configuration/distribution-secret-cluster02:private-key.pem}"
        ssl.truststore.certificates: "${directory:/opt/kafka/external-configuration/distribution-secret-cluster02:trusted-ca.pem}"
      # Disable the tls settings for the cluster
      tls: {}

    cluster03:
      # Skipping all other settings to illustrate needed settings
      additionalClientConfigs:
        # define the DirectoryConfigProvider
        config.providers: directory
        config.providers.directory.class: org.apache.kafka.common.config.provider.DirectoryConfigProvider
        # The placeholder structure is directory:PATH:FILE-NAME. DirectoryConfigProvider reads and extracts the credentials from the mounted Secret in schema distributor configurations.
        ssl.keystore.certificate.chain: "${directory:/opt/kafka/external-configuration/distribution-secret-cluster03:cert-chain.pem}"
        ssl.keystore.key: "${directory:/opt/kafka/external-configuration/distribution-secret-cluster03:private-key.pem}"
        ssl.truststore.certificates: "${directory:/opt/kafka/external-configuration/distribution-secret-cluster03:trusted-ca.pem}"
      # Disable the tls settings for the cluster
      tls: {}

Apply the values and confirm the Axual Distributor connects to each remote cluster. A path that resolves to nothing produces an authentication failure against that cluster, not a startup error, so check the connector status rather than only the pod.