Connect: Integrating Axual with external systems

Overview

Kafka Connect is a tool for scalable and reliable streaming of data between Apache Kafka and other systems.
It makes it simple to quickly define Connectors that move large collections of data into and out of Kafka. Kafka Connect can ingest entire databases or can collect metrics from all your application servers, into Kafka topics, making the data available for stream processing with low latency. An export job can deliver data from Kafka topics into secondary storage and can query systems or into batch systems for offline analysis.

Axual Connect

We offer this functionality under the name "Axual Connect": a service wrapper around Kafka Connect
Axual Connect uses the Switching and Resolving capabilities of Axual Client Proxy extending basic Kafka Connect with functionality to support multiple kafka-clusters, tenants and environments. The service can run on multiple machines under the same Axual Instance, forming a Connect-Cluster.

Use Kafka Connect to Import/Export Data

In order to copy data between Kafka and another system, we use Connectors designed for the system we’re integrating with.
Connectors come in two flavors:

  • SourceConnectors import data from another system to Kafka (e.g. JDBCSourceConnector would import a relational database into Kafka).

  • SinkConnectors to export data from Kafka to another system (e.g. JDBCSinkConnector would export the contents of a Kafka topic to a relational database).

"Source" and "Sink" describe the integrated system, not kafka. E.g. the integrated system is the source.

Connect Sink Source

Automatic registration of avro-schemas and kafka-topics is disabled within the Axual Platform:
You have to deploy the topics before the connectors can use them.