How to Host a Plugin Download Location

This guide shows you how to host the plugin archives Axual Connect downloads at startup, either as an nginx deployment inside the cluster or as a throwaway file server on a local machine.

Type

How-to guide

Goal

Serve the plugin and common-resources archives from somewhere the Connect pods can reach.

Audience

Platform Operator who can create Deployments, Services and PersistentVolumeClaims in the Connect namespace.

When to use

Use this guide when setting up plugin hosting for the first time, or when testing a plugin set locally.

Axual Connect is deprecated. See Why Kafka Connect replaces Axual Connect for the deprecation timeline. On a new installation use Kafka Connect instead, where plugins are added per cluster rather than shared across every tenant. See How to Deploy a Kafka Connect Cluster.

Axual Connect fetches its plugin archives over HTTP on every pod restart, so the download location has to be permanently reachable rather than available only at install time. Hosting it inside the Connect namespace is the simplest way to guarantee that.

The sections below cover each task in this guide:

Prerequisites
Host the archives inside the cluster
Serve a plugin set from a local machine

Prerequisites

Confirm the following before you begin.

Access and permissions required

You need the following access and permissions:

Permission to create Deployments, Services and PersistentVolumeClaims in the namespace Axual Connect runs in.
Helm access to that namespace, for pointing Axual Connect at the result.

Tools and versions required

You need the following tools:

kubectl >= 1.28.
helm >= 3.12.
docker and tar, for the local option only.

Resources that must exist before starting

The following must already exist:

A standard storage class, or another one to substitute in the manifest below.
The plugin archives themselves. See How to Install Connect Plugins for how they are packaged.

Host the archives inside the cluster

Run nginx over a persistent volume in the Connect namespace. The volume keeps the archives across restarts, and a ClusterIP Service is enough because only the Connect pods need to reach it.

Apply the three resources below with kubectl apply -f connectorstore.yaml:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: connectorstore-volume
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: standard
  volumeMode: Filesystem

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: connectorstore
spec:
  replicas: 1
  selector:
    matchLabels:
      app: connectorstore
  template:
    metadata:
      labels:
        app: connectorstore
    spec:
      containers:
        - name: nginx
          image: nginx:1.27   # pinned, so a restarted pod serves the archives from the same nginx build
          ports:
            - containerPort: 80
          volumeMounts:
            - name: connectors
              mountPath: /usr/share/nginx/html
      volumes:
        - name: connectors
          persistentVolumeClaim:
            claimName: connectorstore-volume

---
apiVersion: v1
kind: Service
metadata:
  name: connectorstore
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: 80
  selector:
    app: connectorstore

Then load the archives into the running pod and confirm they are served:

kubectl cp connector-package.tgz <POD_NAME>:/usr/share/nginx/html
kubectl exec <POD_NAME> -- ls -larth /usr/share/nginx/html

Replace every <VALUE> placeholder with your own value before running a command.

Confirm the axual-connect-common-resources archive is among the files. Axual Connect fails to start without it, and the failure looks like a plugin problem rather than a missing archive.

To check over HTTP rather than by listing the directory, port-forward the pod and fetch a file:

kubectl port-forward pods/<POD_NAME> 8000:80
curl localhost:8000/connector-package.tgz > check.tgz

Finally point Axual Connect’s downloadPlugins.artifactsBaseUrl at the Service. See Package and install the plugin.

Serve a plugin set from a local machine

Use this for a local Docker Desktop environment, where the point is to try a different plugin set rather than to host anything permanently. The archives Axual publishes are hosted in the Axual cloud, so a local run needs its own file server.

The steps below use two directories on the local machine: <DOWNLOAD_LOCATION>, where the plugins are gathered and unpacked, and <FILESERVER_ROOT>, which the file server serves.

Download the published collection into the directory you gather the plugins in, then unpack every archive in it so the directory holds only plugin directories and JARs.
```
wget -P "<DOWNLOAD_LOCATION>" "https://stpaxualconnect.blob.core.windows.net/default/axual-connect-plugins-2.0.0.tgz"
cd "<DOWNLOAD_LOCATION>"
for archive in *.tgz; do tar -xzf "$archive"; done
rm *.tgz
```
Add any further plugins to the same directory before unpacking, so they end up in the archive built in the next step.

Bundle them into one archive, from the directory holding the plugins.

tar --disable-copyfile \
   -czf "<FILESERVER_ROOT>/my-axual-connect-plugins.tgz" \
    *

Download the connector common resources archive next to it.

wget -O "<FILESERVER_ROOT>/my-axual-connect-commons.tgz" \
         "https://stpaxualconnect.blob.core.windows.net/default/axual-connect-common-2.0.0.tgz"

Serve both from a container on port 8000. The python tag is pinned, so every run serves the archives from the same image.

docker run -it --rm --name mylocalfileserver \
   -p 8000:8000                             \
   -v "<FILESERVER_ROOT>:/public-files"     \
   'python:3.9.13-slim'                     \
   sh -c 'cd /public-files; python -m http.server 8000'

Point Axual Connect at the local server and upgrade.

helm upgrade --install -n kafka axual-connect \
  --set downloadPlugins.artifactsBaseUrl='http://platform.<DOMAIN>:8000' \
  --set downloadPlugins.connectPluginsFile='my-axual-connect-plugins.tgz' \
  --set downloadPlugins.commonResourcesFile='my-axual-connect-commons.tgz' \
  -f ./values.docker-desktop.yaml \
  .

Axual Connect now runs with the plugins bundled in my-axual-connect-plugins.tgz. Confirm they loaded by calling the /connector-plugins endpoint on a Connect node.