Installing Axual-Connect

Prerequisites

Axual Connect needs a functioning Axual Platform installation, deployed with helm charts. For more information on deployment of the platform refer to the docs

Install Axual Connect component

To install Axual-Connect component the following steps should be applied:

  1. Vault should be configured properly to support Axual-Connect. Before proceeding to the next steps, have a look here

  2. Perform the following changes in governance values:

    platform-manager:
       config:
          axual:
             # Connect Configuration
             connect:
             available: true
             instance-connect-credentials:
                # This should be the same with the basicAuth of axual-connect values
                <{tenant}-{instance}>:
                   authorizer: "basic"
                   username: "axual"
                   password: "password!"
          connector-vault:
             enabled: true
             instances:
                <{tenant}-{instance}>:
                   uri: "http://axual-governance-platform-manager-vault:8200"
                   roleId: "<connect-role-id>"
                   secretId: "<connect-secret-id>"
                   certChainKeyname: "certificate.chain"
                   privateKeyname: "private.key"
                   connectorsPath: "connectors"
    platform-ui:
      config:
        connectEnabled: true
  3. Prepare the values for axual-connect deployment

    Click to open axual-connect-values.yaml
    axual:
      tenant: &tenant "<tenant-short-name>"
      instance: &instance "<instance-short-name>"
      environment: "<environment-short-name>"
      applicationId: "_<application-short-name>"
      applicationVersion: "1.0.0"
      # Do not use Discovery API, but static configs
      staticConfig:
        tenant: *tenant
        instance: *instance
        cluster: "<cluster-name>"
        # If connect is running in the same Kubernetes cluster we can just use the internal service url as well.
        bootstrapServers: "bootstrap-kafka.<domain>:443"
        schema.registry.url: "https://apicurio.<domain>"
        enable.value.headers: "false"
        group.id.resolver: "io.axual.common.resolver.GroupPatternResolver"
        group.id.pattern: "{tenant}-{instance}-{environment}-{group}"
        topic.resolver: "io.axual.common.resolver.TopicPatternResolver"
        topic.pattern: "{tenant}-{instance}-{environment}-{topic}"
        transactional.id.resolver: "io.axual.common.resolver.TransactionalIdPatternResolver"
        transactional.id.pattern: "{tenant}-{instance}-{environment}-{transactional.id}"
      tls:
        clientEnabled: true
        serverEnabled: false
        automatedKeystores: true
        createClientKeypairSecret: false
        clientKeypairSecretName: "axual-connect-cert-secret"
        clientKeypairSecretCertName: "tls.crt"
        clientKeypairSecretKeyName: "tls.key"
        createTruststoreCaSecret: false
        truststoreCaSecretName: "my-external-ca"
      
      basicAuth:
        enabled: true
        username: "axual"
        password: "password!"
    
    persistPlugins:
      enabled: false
      createPersistentVolume: false
    
    downloadPlugins:
      enabled: true
      artificateBaseUrl: "https://stpaxualconnect.blob.core.windows.net"
      connectPluginsFile: "dizzl-0eafdac8/dizzl-axual-connect-plugins-2.0.0.tgz"
      commonResourcesFile: "dizzl-0eafdac8/dizzl-axual-connect-common-2.0.0.tgz"
      resourcePath: "/usr/share"
      resources:
        requests:
          cpu: 100m
          memory: 128Mi
        limits:   
          memory: 2Gi
    
    configurationOverrides:
      "plugin.path": "/usr/share/plugins"
      "plugin.discovery": "SERVICE_LOAD"
    
    
    # Vault Configuration
    vault:
      address: "http://axual-governance-platform-manager-vault.axual.svc.cluster.local:8200"
      authMethod: APPROLE
      approleRoleId: "987d4b9f-a0b7-ac36-afab-6ab1f597e149"
      approleSecretId: "20b5bfaa-7d97-a0c2-fcfb-6f8f43718533"
      ssl:
        verify: true
      secretProvider:
        enabled: true
        class: io.axual.utilities.config.providers.VaultConfigProvider
      keystoreProvider:
        enabled: true
        class: io.axual.utilities.config.providers.VaultKeyStoreProvider
        truststorePassword: notsecret
        certificateChainKeyname: certificate.chain
        privateKeyKeyname: private.key
    
    # Init Resouce Configuration
    keystoreProvider:
      resources:
        requests:
          cpu: 100m
          memory: 128Mi
        limits:
          memory: 256Mi
    
    kafkaInitContainer:
      # -- Enable topic and ACL creation
      enabled: true
      # If connect is running in the same Kubernetes cluster we can just use the internal service url as well.
      bootstrapServers: "SSL://bootstrap-kafka.<domain>:443"
      # -- min.isr of topics used to store connect state/offset/config
      minIsr: "1"
      # -- Replication factor of topics used to store connect state/offset/config
      replicationFactor: "1"
      # -- Principal common name used to produce and consume from connect state/offset/config topics (should match the one on axual.tls.clientKeystore)
      principal: "User:CN=Connect client"
      tls:
        # -- Existing Keypair secret name
        keypairSecretName: "server-cert-secret"
        # -- Existing Keypair key name
        keypairSecretKeyName: "tls.key"
        # -- Existing Keypair certificate name
        keypairSecretCertName: "tls.crt"
        # -- Existing Truststore secret name
        truststoreCaSecretName: "my-external-ca"
        # -- Existing Truststore certificate name
        truststoreCaSecretCertName: "ca.crt"
      # -- The [resource requirements](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) for this container.
      resources:
        requests:
          cpu: 100m
          memory: 128Mi
        limits:
          memory: 256Mi
    
    # configuration values for RoutingKafkaAppender
    routedLogging:
      # indicates if we add the routing Kafka appender to the Logback config
      enabled: false
      # indicates if we take Axual environment in consideration when routing connector logging
      suppressEnvironment: false
      pattern: '%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} %msg'
      enableHostnameVerification: true
      debugMode: false
    
    securityContext:
      { }
    
    additionalWorkerConfig:
      { }
    
    podLabels: {}
    
    connect-vault:
      enabled: true
      ui:
        enabled: true
        serviceType: "LoadBalancer"
        externalPort: 8200
      server:
        dataStorage:
          storageClass: "hostpath"
      injector:
        enabled: false
  4. From the project root directory execute:

    helm dependency update
    helm install axual-connect oci://registry.axual.io/axual-charts/axual-connect -f axual-connect-values.yaml -n kafka

Connectors maintenance on a local Docker-Desktop environment

Currently, Axual Connect download connector plugins JARs are hosted on the AxualCloud.

In case we want to change the available connector plugins JARs for an Axual Connect installation we would need to run a local FileServer that will serve a set of JARs.

  1. Gather the connector plugins JARs you want to include as part of your Axual Connect

      # download a large connector collection
       wget -P "${DOWNLOAD_LOCATION}" "https://stpaxualconnect.blob.core.windows.net/default/axual-connect-plugins-2.0.0.tgz"
       # download additional connect plugins
       # extract all connect plugins from their archives, leaving only directories and JARs
  2. Bundle all connector plugins JARs into a single Tarball Archive

    1. Move to the location where all plugins are in JAR format

    2.    tar --disable-copyfile \
         -czf "${FILESERVER_ROOT}/my-axual-connect-plugins.tgz" \
          *
  3. Download the connector common resources as Tarball Archive

       wget -O "${FILESERVER_ROOT}/my-axual-connect-commons.tgz" \
             "https://stpaxualconnect.blob.core.windows.net/default/axual-connect-common-2.0.0.tgz"
  4. Start a local FileServer serving the connector plugins and connector common resources

       docker run -it --rm --name mylocalfileserver \
       -p 8000:8000                             \
       -v "${FILESERVER_ROOT}:/public-files"    \
       'python:3.9.13-slim'                     \
       sh -c 'cd /public-files; python -m http.server 8000'
  5. Change the downloadPlugins configuration to load plugins and common resources from the local FileServer

       helm upgrade --install -n kafka axual-connect \
         --set downloadPlugins.artifactsBaseUrl='http://platform.<domain>:8000'      \
         --set downloadPlugins.connectPluginsFile='my-axual-connect-plugins.tgz'  \
         --set downloadPlugins.commonResourcesFile='my-axual-connect-commons.tgz' \
         -f ./values.docker-desktop.yaml                                          \
         .

This will upgrade Axual Connect with the Plugins JARs you have bundled in the my-axual-connect-plugins.tgz

Installed Components

You can use helm status <release name> -n <namespace> to view the status.

For example:

$ helm status axual-connect -n kafka

NAME: axual-connect
LAST DEPLOYED: Fri Nov 20 16:36:51 2020
NAMESPACE: kafka
STATUS: deployed
REVISION: 5
TEST SUITE: None
NOTES:
This chart installs Axual Connect

This will install following components:

  1. Deployment <tenant>-<instance>-axual-connect which contains Axual Connect Pods.

  2. Service axual-connect for clients to connect to Axual Connect REST endpoint.

  3. Secrets sensitive data used by Pods.

  4. ConfigMap configuration used by Pods.

  5. Persistent Volume persists connect plugins & common-resources.

  6. Persistent Volume Claim volume mount inside Pods to persist and use connect plugins & common-resources.

Self Service UI: Enable Axual-Connect

  1. Login to Self Service UI using the following URL: https://platform.<domain>/login/{tenant-shortname}

  2. Go to the Instances page

    Axual Self-Service UI, showing the Instances Page
  3. Select {tenant-shortname}-{instance-shortname} and open the edit form

  4. Enable Axual Connect:

    Edit Instance
  5. Provide the following value for Connect URL: https://platform.<domain>:11000/

  6. Click Update Instance in the bottom-right of the screen.

Congratulations! Axual-Connect is now available! You can now create a Connect application in Selfservice.