Axual Platform Certificates

Introduction

This document covers all aspects of certificates, private keys and keystores that will be used in Axual Cloud. It summarizes the discussion about standards, lifecycle and security with TLS.

Installing Certificates (Helm Install)

In the Helm installation the certificates are installed into the values.yaml configuration for each service in the tls settings.

It is neccessary to have a ca truststore. This store contains a list of the trusted certificate roots. The server certificates are used to secure the services of each module and the client truststore is used for connections to other services.

Given the Helm setup is is only neccessary to enter the pem for the truststore, client and server keystores.

Automated Keystore Management

Using the Axual Keystore Provider, the keystores and truststores used by many Axual components can be automatically generated and passed to the relevant components. To enable this feature, turn on the feature flag in the component where needed.

tls:
automatedKeystores: true

When turned on, the Keystore provider is started as an Init container. For proper setup, configure the chart as described below.

Let Helm Chart create the secrets In this setup, the values.yaml is configured with the server and client certificate and private key. Secrets will be automatically created and mounted in the init container and app container for keystore generation. Set the configuration as below: For server keystore:

tls:
createServerKeypairSecret: true
serverCertificatePem: |
-----BEGIN CERTIFICATE-----
...
serverKeyPem: |
-----BEGIN PRIVATE KEY-----
...

For client keystore:

tls:
createClientKeypairSecret: true
clientCertificatePem: |
-----BEGIN CERTIFICATE-----
...
clientKeyPem: |
-----BEGIN PRIVATE KEY-----
...

For truststore:

tls:
createTruststoreCaSecret: true
caCerts:
some_ca.crt: |
-----BEGIN CERTIFICATE-----
...
another_ca.crt: |
-----BEGIN CERTIFICATE-----
...

All CA certificate files must have a .crt extension.

Components Interaction Overview

Below table gives an overview of various components in Axual Platform and the communication channels between them. Most customers will install Axual for their own use. As such there is only one tenant. In the documentation below tenant and client are interchangable.

ZooKeeper

Axual

High

Broker

N/A

Non SSL. Secured inside private network.

Broker

Axual

High

Other brokers (SSL), Cluster API (SSL), Distributor (SSL), Producers/Consumers (SSL)

ZooKeeper (Non-SSL)

Inter broker is SSL. Cluster API needs super user access to brokers. Distributor Connect cluster needs access to certain topics on brokers with mutual SSL + ACL check.

Broker is publicly accessible service exposed over SSL. Client applications directly talk to Broker for producing/consuming messages.

Cluster API

Axual

Very High

Instance API (SSL), DevOps Browser (SSL), Axual Deploy scripts (SSL), CDA (SSL)

Brokers (SSL), ZooKeeper (Non-SSL)

Instance API is tenant-instance specific service which will use tenant CA signed certificates. DevOps users will have their own certificates to authenticate via Web Browser. Axual deploy scripts will also get their own certificate to authenticate with.

Cluster API is a very critical component with super user access to brokers. Access to Cluster API is provided to carefully selected certificates (DevOps users, Axual-Deploy, Instance APIs that are hosted on this Cluster).

Cluster API - ZooKeeper communication is on a private locked down subnet.

Distributor

Tenant

Very High

N/A

Brokers (SSL), CDA (SSL)

Distributor has read-write access to system topics (_distributor-config, _distributor-offsets, _distributor-status).

Distributor should have read permissions on all topics belonging to same instance of the source cluster. It should have write permissions on all topics belonging to same instance of other clusters in the instance.

Distributor connects via Mutual SSL to CDA to be notified of any cluster marked as inactive.

Schema Registry Slave

Tenant

Medium

Producers/Consumers (SSL)

ZooKeeper (Non-SSL), Brokers (SSL)

SR Slave needs mutual SSL connection with broker. It must have read/write permissions on $INSTANCE-schemas topic.

SR Slave is a publicly accessible service exposed over SSL. Client applications directly talk to SR slaves to fetch schemas.

SR Slave also talks to ZooKeeper over private network.

Schema Registry Master

Tenant Medium

Instance API (SSL)

ZooKeeper (Non-SSL), Brokers (SSL)

Instance API also talks to SR Master over mutual SSL. This communication must be restricted to same instance.

SR Master is only accessible from localhost. This restricts us to run Instance API and SR Master on the same host.

Instance API

Tenant High

Management API (SSL), DevOps Browser (SSL)

Cluster API (SSL), SR Master (SSL)

Instance API issues topic apply commands to Cluster API over SSL. Instance API should only be able to communicate with Cluster APIs that belong to it.

Instance API talks to SR Master to retrieve schema IDs. This should be restricted to SR Master of same instance.

Management API talks to Instance API over SSL. Only Management API’s certificate should be allowed to talk to Instance API. DevOps users can also identify via mutual SSL and view contents of AVRO topics.

Cluster Discovery API (formerly CCP)

Tenant High

Producers/Consumers (SSL), Rest Router (SSL)

Cluster API (SSL)

CDA connects to Cluster API via mutual SSL to check Cluster status.

CDA is publicly accessible service exposed over SSL. Client applications directly talk to CDA to be redirected to brokers.

Management API

Axual

Very High

Management UI (SSL)

Instance API (SSL), Management DB (SSL) Management API talks to Instance API via mutual SSL. Management API must have unique certificate identity that is authorized to connect to every Instance API.

Management API connects to Management DB over SSL + password.

Management UI connect to Management API over SSL and specific authentication mechanism per tenant.

Management UI

Axual

High

End Users (SSL)

Management API (SSL)

End users connect to Management UI based on their tenant’s defined authentication mechanism + SSL.

Management UI connects to Management API over SSL.

Axual Rest Proxy

Tenant High

Rest Router (SSL)

Brokers (SSL), SR Slaves (SSL), ZooKeeper (Non-SSL)

Rest Router

Tenant

High

End Users (SSL)

Axual Rest Proxy (SSL), CDA (SSL)

Component Keystore Requirement

This section deep-dives into every component and identifies the keystores and truststores needed for the component to function properly. It also explains what certificates / key-pairs must be present inside those keystores.

Server Keystore

This is used by the component to host a TLS endpoint (TCP or HTTP). If it is an HTTPS endpoint, then the key pair inside this keystore must have CN value matching the FQDN which will be used to access this endpoint.

Server Truststore

This is used by the component to validate the client certificate presented by the clients connecting to this component. It should not have key pairs but only public certificates (CA or otherwise).

Client Keystore

Key pair inside this keystore is presented by the component when trying to access other components. Only required when other components require a client certificate to be presented.

Client Truststore

If other components endpoint are also SSL, then this truststore is used to validate the trust chain of remote endpoint. Only required when other components endpoint are TLS encrypted.

With above nomenclatures clarified, let’s look into each components needs.

Keystore Diagram

SSL Component Interaction (Axual Cloud Installation)

Should have the Axual CA certificate that can validate broker’s certificate.

Management API is accessed by Management UI (1-way SSL + authentication). It connects to Instance API (mutual SSL) and Management DB (to be 1-way SSL).

Server Keystore

mgmt-api.server.keystore.jks

Should have key pair (Axual) with CN matching the FQDN.

Server Truststore

N/A

Client Keystore

mgmt-api.client.keystore.jks

Should have the key pair (Axual) authorized to talk to Instance API (via mutual SSL).

Client Truststore

mgmt-api.client.truststore.jks

Should have Axual CA certificate to validate the key pair presented Instance API.

Management UI

Management UI is accessed by end users (1-way SSL + backend authentication). It connects to Management API (1-way SSL).

Server Keystore

N/A

Server Truststore

N/A

Client Keystore

N/A

Client Truststore

N/A

Axual Rest Proxy

Axual Rest Proxy is accessed by producers/consumers via Rest Router (mutual SSL). It connects to Broker (mutual SSL), SR slaves (mutual SSL) and ZooKeeper (non-SSL).

Server Keystore

instance-rest-proxy.server.keystore.jks

Should have key pair (Axual) with CN matching the FQDN.

Server Truststore

instance-rest-proxy.server.truststore.jks

Should have CA certificate of the instance it belongs to.

Client Keystore

instance-rest-proxy.client.keystore.jks

Should have the key pair (Tenant) authorized to talk to Brokers (via mutual SSL).

Client Truststore

instance-rest-proxy.client.truststore.jks

Should have Axual CA certificate to validate the Broker certificate.

Axual Rest Router

Axual Rest Router is a TCP relay that sits in front of Axual Rest Proxy using the same server certificate as Rest Proxy. It receives calls from all producers/consumers, checks the CDA to determine the location of Rest Proxy and proxies the client to one of the backend rest proxy.

Rest Router does not get its own dedicated keystore. It mirrors the Rest Proxy SSL configuration.

Server Keystore

N/A

Server Truststore

N/A

Client Keystore

instance-rest-router.client.keystore.jks

Should have key pair (Tenant) authorized to talk to CDA (mutual SSL)

Client Truststore

instance-rest-router.client.truststore.jks

Should have Axual CA certificate to validate the CDA certificate.

Certificate / Keystore List

Below are the Axual certificates/key pairs needed for proper functionality:

*.cloud.axual.io Signed by COMODO. Will be used wherever Axual FQDN is shown in the diagram above. Axual Client certificate for Cluster API Self signed certificate. This certificate must also be added in Broker server truststore. Alternative is to use the *.cloud.axual.io certificate. Axual Client certificate for Management API Can be self signed certificate. It must also be added in Instance API server truststore. Alternative is to use the *.cloud.axual.io certificate.

client certificate for Instance API Signed by <Client Root>. Will be used to communicate with Cluster API.
client certificate for Distributor Signed by <Client Root>. Will be used to communicate with Brokers and CDA.
client certificate for CDA Signed by <Client Root>.. Will be used by to communicate with Cluster API.
client certificate for SR Master/Slaves Signed by <Client Root>.. Will be used to communicate with Brokers.
client certificate for Axual Rest Proxy Signed by <Client Root>. Will be used to communicate with Brokers and SR Slaves.

Important Considerations

Broker truststore must be changed whenever new instance is added. This would require a broker restart (until dynamic reloading of truststores is added). Broker Super User ACLs must be updated whenever new instance is added. This would require a broker restart. Axual Rest Proxy has an in-built producer and consumer that must have access to all topics for proper functionality. Currently, we use super user certificate to meet this requirement. The super user certificate must be an Axual certificate. If we want to use Tenant provided certificates, then they should be provided access to Tenant’s topics only (possible in Kafka 2.0 with Topic ACLs with pattern matching.

Verification Cheatsheet

[cluster.sh] SUPERUSER_PRINCIPAL should be same as broker.server.keystore.jks, cluster-api.client.keystore.jks
[cluster.sh] DISTRIBUTOR_PRINCIPAL should be same as distributor.keystore.jks [used for producing/consuming from/to distributor topics]
[cluster.sh] CLUSTERAPI_AUTHORIZED_DNS should have values that match CN of {tenant}-{instance}-instance-api.client.keystore.jks, {tenant}-{instance}-discovery-api.client.keystore.jks, axual_deploy_client.pem
[instance.sh] DISTRIBUTION_PRINCIPAL should be same as {tenant}-{instance}-distributor.keystore.jks
[instance.sh] SCHEMAREGISTRY_PRINCIPAL should be same as {tenant}-{instance}-schemaregistry-client.jks

Runtime security of Private Keys

At runtime, private keys (inside keystores) would be available on the VMs. These locations must be secured with proper permissions (600). Only applications that are meant to use the private keys should have access.

Security at Rest

At rest, private keys (inside key pairs) would be stored in Azure KeyVault. Access to KeyVault will be allowed only to Axual Admins. More details are available in below section.

Azure KeyVault

Azure Key Vault is a tool for securely storing and accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, or certificates. Secrets and keys are safeguarded by Azure, using industry-standard algorithms, key lengths, and hardware security modules (HSMs). The HSMs used are Federal Information Processing Standards (FIPS) 140-2 Level 2 validated.

For situations where added assurance is required, keys can be imported or generated in hardware security modules (HSMs) that never leave the HSM boundary. Microsoft uses Thales hardware security modules.

More details are available at https://blogs.technet.microsoft.com/kv/2016/09/26/manage-certificates-via-azure-key-vault/

Azure KeyVault documentation is available at https://docs.microsoft.com/en-us/azure/key-vault/key-vault-overview