Installing the ADLS Gen2 Sink Connector
Downloading the connector library
The library required to run the connector can be found in the Maven Central
-
Search for the artifact on Maven Central
-
Select the version of the connector you wish to install.
-
Download the JAR type for your Kafka Connect installation.
Using the wrong JAR type can result in failing connectors caused by class not found exceptions These JAR types are available:
-
jar-with-dependencies
This version contains all the dependencies in the JAR. This includes the Azure SDK and the Avro and Confluent Schema Registry Converter libraries.
Use this if the common classpath of the Kafka Connect installation does not contain the Avro or Confluent libraries -
jar-without-avro-dependencies
This version contains all the dependencies in the JAR. This includes the Azure SDK and the Avro and Confluent Schema Registry Converter libraries.
Use this if the common classpath of the Kafka Connect installation already contain the Avro or Confluent libraries. -
jar
Not recommended as this version only contains the compiled code, but none of the dependencies.
-
-
Continue with Installing the connector
Installing the connector
-
Determine all machines in the Kafka Connect cluster
-
Follow these steps for each machine found:
-
Determine the plugins directory used by Kafka Connect
-
Create a subdirectory *adls* in the *plugins* directory
-
Copy the downloaded connector library to the newly created *adls* directory
-
Restart Kafka Connect
-
Verify that the ADLSGen2SinkConnector is shown by querying the connector-plugins endpoint of the REST API of Kafka Connect on the machine.
# curl command piped through the JQ tool to format the JSON output curl -s -X GET http://localhost:8083/connector-plugins | jq -SThe URl used depends on the configuration of your Kafka Connect installation. Additional curl options might be required if basic authentication or TLS is used.
-