How to Replace the Root Certificate Authority

This guide shows you how to add a new root Certificate Authority (CA) alongside the one the platform already trusts, move every component and client application onto it, and remove the old authority once nothing depends on it.

Type

How-to guide

Goal

Move the whole platform onto a new root Certificate Authority without losing connectivity part way through.

Audience

Platform Operator who can edit Secrets and restart pods in every namespace an Axual component runs in, and who can edit cluster and instance settings in Self-Service.

When to use

Use this guide when the root Certificate Authority is expiring, when a different authority replaces it, or when a second authority has to be trusted alongside the current one.

Run this as a two-pass migration. Both authorities stay trusted in between, so nothing loses connectivity while the certificates move over. To replace a single component’s certificate under an authority the platform already trusts, use How to Rotate a Certificate instead.

Contents

The sections below cover each task in this guide:

Prerequisites

Confirm the following before you begin. This migration reaches every component and every client application, so it needs more in place than a single certificate rotation does.

Access and permissions required

You need the following access and permissions:

  • Permission to get, edit and patch Secrets in every namespace an Axual component runs in, not only the namespace of the component you start with.

  • Permission to delete pods, or to trigger a rollout, for every component that mounts a truststore Secret.

  • Edit access to the cluster and instance settings in Self-Service, which hold certificate authorities of their own.

Tools and versions required

You need the following tools:

  • kubectl >= 1.28, with access to every namespace the platform runs in.

  • base64, to encode the new authority certificate before writing it into a Secret. The command below uses the GNU -w 0 flag to keep the output on one line, which a Secret value requires; on macOS use -b 0 instead.

  • openssl, to read back the certificate a component serves once it restarts.

Resources that must exist before starting

The following must already exist:

  • The new root Certificate Authority certificate, in PEM form, on the machine you run the commands from.

  • A list of every truststore Secret the installation uses, in the shape described in Truststore (CA) Secrets. A Secret missed here keeps trusting the old authority alone, and its component starts rejecting connections at the final step.

  • A named owner for every client application that connects to the platform. Each application has to obtain a certificate signed by the new authority before the old one comes out, and the platform cannot report which applications still hold old certificates.

  • The Self-Service cluster whose CA (PEM) value holds the authority that signs the broker server certificate, described in CA (PEM).

  • The Self-Service instance whose signing CA signs application certificates, uploaded as described in Create the Instance.

Add the new Certificate Authority alongside the old one

Add the new authority everywhere the old one is trusted, before anything starts presenting certificates issued by it. Both authorities stay trusted until the final section, which is what keeps every step up to that point reversible.

Replace every <VALUE> placeholder with your own value before running a command.

  1. Add the new authority to every truststore Secret as an extra entry. A truststore Secret carries any number of key and value pairs and each value is appended to the truststore, so the old and new authority coexist. The entry’s key must end in .crt, or it is silently skipped. See Truststore (CA) Secrets.

    kubectl -n <NAMESPACE> patch secret <TRUSTSTORE_SECRET> \
  --type merge \
  -p "{\"data\":{\"new_root_ca.crt\":\"$(base64 -w 0 <NEW_ROOT_CA_FILE>)\"}}"

  2. Restart every component that mounts the Secret. The Axual Keystore Provider init container rebuilds the truststore at pod startup, so a running pod keeps the old truststore until it restarts. How to Restart a Service covers the restart.

  3. Register the new authority where Self-Service holds a certificate authority of its own: the cluster’s CA (PEM), and the instance’s signing CA for application certificates.

Move every certificate onto the new authority

Start this section only once every component trusts both authorities. Before that, a component presenting a certificate from the new authority is rejected by the peers that have not restarted yet.

  1. Reissue each component’s certificate from the new authority, following either Rotate a certificate cert-manager issued or Rotate a manually managed certificate. Under cert-manager this is a change of issuerRef on each Certificate.

  2. Have every client application obtain a certificate signed by the new authority. Until an application does, it can still connect, because the old authority is still trusted.

Remove the old Certificate Authority

Removing the old authority ends the migration, and it is the step that cannot be undone without repeating the whole procedure. Leave it until every component and every client application presents a certificate from the new authority.

Removing the old authority before every client application has a certificate from the new one rejects those applications at the TLS handshake. Confirm the client certificates first, because the platform cannot tell you which applications still hold old ones.

  1. Remove the old authority’s entry from every truststore Secret, naming the key it was stored under.

    kubectl -n <NAMESPACE> patch secret <TRUSTSTORE_SECRET> \
  --type json \
  -p '[{"op": "remove", "path": "/data/<OLD_ROOT_CA_KEY>"}]'

  2. Restart every component that mounts the Secret, so the Axual Keystore Provider init container rebuilds each truststore without the old authority.

Confirm the migration is complete

Read the certificate back off each live endpoint, because that is the only check proving a component reloaded rather than kept the certificate it started with.

openssl s_client -connect <HOST>:<PORT> -showcerts

The issuer in the output names the new authority once the component has moved, and the old one while its pod has not restarted. How to Inspect a Certificate covers the other ways to read the same values.

Then check the client side. Every application still connects, and no component logs a TLS handshake failure. A handshake failure after the old authority comes out names an application whose certificate was never reissued.