Installing Connect-Plugins

Installing plugins

The way connect-plugins are currently provided for the helm installation of the platform is:

  • Host connect-plugins on a webserver

  • An init-job downloads the plugins and common jars

  • The plugins are stored in a volume

  • Axual-Connect pods mount the volume

The plugins and common jars are downloaded on every restart of the pod. A more efficient method is being developed, but for now this is the supported method.

In order to install a new connect-plugin, you have to:

  1. Add the new Connect-Plugin JARs to the FileServer
    There is one archive file for connect-plugins, and one for the common-jars

  2. Confirm that Axual Connect has the correct configuration to download Connect-Plugins from your FileServer

    downloadPlugins:
  artifactsBaseUrl: "[URL_OF_YOUR_FILE_SERVER]"
  connectPluginsFile: "[PATH_TO_YOUR_PLUGINS_TARBALL]"
  commonResourcesFile: "[PATH_TO_YOUR_COMMON_RESOURCES_TARBALL]"

  3. Issue a helm upgrade command for Axual-Connect:

    helm upgrade --install -n kafka axual-connect -f [YOUR_CUSTOM_VALUES]

Make the new Connect-Plugins available in the Self-Service portal

After Axual Connect has been restarted, confirm that the plugins are available by checking the /connector-plugins endpoint of any Connect-Node’s API.

Although the plugins are available in the Connect-Cluster, they may not yet available on the Self-Service portal.
You have two options to make the new Connect-Plugins available:

  1. Wait until the reconciliation.connect.plugin job has executed

    1. You can check how often the reconciliation connect plugin job runs by checking selfservice-api’s SCHEDULER_RECONCILIATION_CONNECT_PLUGINS_CRON config value.

    2. You can change the frequency by configuring the SCHEDULER_RECONCILIATION_CONNECT_PLUGINS_CRON value. see Configuring Connect Reconciliation Jobs

  2. Manually trigger a refresh of Connector Plugin by editing the Instance associated the Axual Connect in the Self-Service Portal

    1. Log in the Self-Service Portal as a TENANT_ADMIN

    2. Go to Instances page

    3. Select the Instance associated with the Axual Connect instance you updated the plugins for

    4. Press the edit button

    5. Save the Instance as-is, without changing anything. This will trigger the reconciliation plugin job to run immediately

When upgrading connect-plugin versions, existing connectors may start failing due to missing mandatory configurations.
We advise you to collect a list of Connect-Applications using a certain Connect-Plugin before performing any maintenance on it.
Make sure that the Connect-Application owners are aware of the implications of your maintenance operation.

Manual JAR installation

If you are not using the Helm-based plugin distribution, you can install a connector JAR manually on each Kafka Connect node.

  1. Download the connector JAR (see the connector’s page in the Connect Plugins Catalog for the download link).

  2. For each node in the Kafka Connect cluster:

    1. Locate the plugins directory used by Kafka Connect.

    2. Create a subdirectory named after the connector inside plugins (e.g. my-connector).

    3. Copy the downloaded JAR to that subdirectory.

    4. Restart Kafka Connect.

    5. Verify the connector is listed by querying the connector-plugins endpoint:

      curl -s -X GET http://localhost:8083/connector-plugins | jq -S '.[]'
      The URL depends on your Kafka Connect configuration. Additional curl options may be required if basic authentication or TLS is used.