Connectors

Supported Connectors

Connector Name Supplier Version Connector Type (Source/Sink) Documentation Source Code Description

Connector Name	Supplier	Version	Connector Type (Source/Sink)	Documentation	Source Code	Description
JDBC	Confluent	`5.0.4`	`Source` & `Sink`	Link	Link	`JDBC source` connector to import data from any relational database with a JDBC driver into Kafka topics. `JDBC sink` connector to export data from Kafka topics to any relational database with a JDBC driver. Note : supported only for `mariadb`
Snowflake	Snowflake	`1.4.0`	`Sink`	Link	Link	This connector exports data from one or more Kafka topics and loads the data into a Snowflake table.
Amazon S3	Confluent	`5.0.4`	`Sink`	Link	Link	This connector allows you to export data from Kafka topics to Amazon S3. Important: Your Connect cluster should be configured to use S3. Contact your operator before using this connector.
HTTP-Sink	Axual	`1.0.0`	`Sink`	Link	Link	The connector consumes records from Kafka topic(s) and converts each record to a JSON with the following format `{"topic": "<topicName>", "key": "<keyData>", "value": "<valueData>"}` and builds an HTTP request based on the configured method. All HTTP methods are supported, default is `POST`. The connector supports custom SSL/TLS settings. Basic Authentication as well as authentication using client certificates when using mutual TLS are supported. HTTP settings such as redirect following and retries can be set. Developers can implement their own formatting and authentication extensions if the provided implementations are not sufficient.

JDBC

Confluent

5.0.4

Source & Sink

JDBC source connector to import data from any relational database with a JDBC driver into Kafka topics.
JDBC sink connector to export data from Kafka topics to any relational database with a JDBC driver.
Note : supported only for mariadb

Snowflake

1.4.0

Sink

Link

This connector exports data from one or more Kafka topics and loads the data into a Snowflake table.

Amazon S3

Confluent

5.0.4

Sink

Link

This connector allows you to export data from Kafka topics to Amazon S3.
Important: Your Connect cluster should be configured to use S3. Contact your operator before using this connector.

HTTP-Sink

Axual

1.0.0

Sink

Link

The connector consumes records from Kafka topic(s) and converts each record to a JSON with the following format {"topic": "<topicName>", "key": "<keyData>", "value": "<valueData>"} and builds an HTTP request based on the configured method. All HTTP methods are supported, default is POST. The connector supports custom SSL/TLS settings.
Basic Authentication as well as authentication using client certificates when using mutual TLS are supported.
HTTP settings such as redirect following and retries can be set.
Developers can implement their own formatting and authentication extensions if the provided implementations are not sufficient.

Connectors Maintenance

Adding New Connector Plugins via BASH

Repeat the following steps for each machine where Connect is running.

Copy the Connector JARs files to the path define in CONNECT_PLUGINS_DIR_PATH and create a new directory.
If multiple jars are required for a plugin (like JDBC Connector and JDBC Drivers), put them together in one directory
For example:

[CONNECT_PLUGINS_DIR_PATH]

├── http-sink
│   └── http-sink-connector-1.0.0-jar-with-dependencies.jar
├── jdbc
│   ├── jdbc-connector.jar
│   └── mariadb-jdbc.jar
├── s3-sink
│   ├── ST4-4.0.4.jar
│   ├── activation-1.1.jar
│   ├── ant-1.9.1.jar
.   .
.   .
.   .
│   ├── zkclient-0.10.jar
│   └── zookeeper-3.4.13.jar
├── snowflake
│   ├── snowflake-connector.jar
│   └── bouncy-castle.jar
└─ [NEW PLUGIN NAME]
    ├── new-plugin.jar
    └── helper.jar

Restart Connect
```
axual.sh [start|restart|stop] client <instance-name> axual-connect
```
Restart Connect services in a rolling fashion, otherwise there will be down time

Continue with Creating application Connector type Creating A Connector Application

To share Java resources / configuration / certificate, see CONNECT_COMMON_CLASSES_PATH,CONNECT_CONFIG_DIR_PATH, CONNECT_CLIENT_CERTS_PATH configurations
You will require restarting Connect to make those resources available.

Adding New Connector Plugins via HELM

Updating the FileServer hosting the Connector JARs

Currently, we require to host Connector JARs in a FileServer (like a S3 bucket) that will be used by Axual Connect to download them when starting up.

This approach is under refactoring, and it will change in the near future

Add the new Connector JARs to your FileServer

Confirm that Axual Connect has the correct configuration to download Connector Plugins from your FileServer

downloadPlugins:
  artifactsBaseUrl: "[URL_OF_YOUR_FILE_SERVER]"
  connectPluginsFile: "[PATH_TO_YOUR_PLUGINS_TARBALL]"
  commonResourcesFile: "[PATH_TO_YOUR_COMMON_RESOURCES_TARBALL]"

Issue an upgrade command for your Axual-Connect installation

helm upgrade --install -n kafka axual-connect -f [YOUR_CUSTOM_VALUES]

Make the new Connector Plugins available in the Self-Service portal

After the Axual Connect has been restarted, the new Connector Plugins are not yet available on the Self-Service portal.

There are two options to make the new Connector Plugins available:

Wait till the reconciliation.connect.plugin job has executed
1. You can check how often the reconciliation connect plugin job runs by checking the selfservice-api’s SCHEDULER_RECONCILIATION_CONNECT_PLUGINS_CRON value
2. You can change the frequency by configuring the SCHEDULER_RECONCILIATION_CONNECT_PLUGINS_CRON values, see Configuring Connect Reconciliation Jobs
Manually trigger a refresh of Connector Plugin by editing the Instance associated with the Axual Connect
1. Log in the Self-Service as a TENANT_ADMIN
2. Move to Instances page
3. Select the Instance associated with the Axual Connect you have upgrade
4. Press the edit button
5. Save the Instance as it is, this will trigger a reconciliation plugin job

Based on the type of maintenance performed on the running Axual Connect, some existing connectors may start failing due to missing mandatory configurations.

We advise to collect a list of Application using a certain Connector Plugin before performing any maintenance.

You can use the /api/applications/search/findByAttributes endpoint available since Platform Manager 6.16.0

e.g.

/api/applications/search/findByAttributes?applicationClass=io.axual.connect.plugins.kafka.sink.KafkaSinkConnector

This will make sure that Application’s Owners are aware of the required change.