What is "Connect"? What are "connectors"?

Glossary of terms

  • Connect-Plugin: A generic program (delivered as a JAR file) which can be configured to integrate an external system with Apache-Kafka. Making an OOP analogy, this can be seen as a "class": it has no runtime of its own, but it can create multiple instances of itself when given the required configuration.

  • Connector: A configured instance of a Connect-Plugin. Continuing the OOP analogy, this can be seen as an "Object": a runtime entity. Multiple Connectors of the same Connect-Plugin-type can exist at the same time. Connectors are preconfigured to connect to the kafka cluster, so a developer only needs to supply configuration required to reach the other system.

  • Connect-Application: A term used within the Axual ecosystem to refer to a Self-Service application which manages a group of Connectors of the same Plugin type. This resource helps with facilitating data governance, the Axual way, the same way we do with custom applications.

  • Connector-Application: A Connect-Application can start one Connector-Application (an instance of itself) per environment. This is technically just a Connector, deployed onto an Axual-Environment.

  • Connector-Task: Connectors are generally run by using multiple "tasks". This is how Connectors scale: by having multiple parallel (and distributed) processes. All instances have the same configuration.

  • Connect-Cluster: A cluster of nodes running Kafka-Connect. All Connector-Tasks run on these machines. This cluster is usually owned and operated by Axual.

All these "connect(or)" names can be confusing. To make reading this documentation a little easier, everytime the word "connect" is related to runtime, it will be bold. When it’s related to configuration/definition, the word will be written in italics.

Ask the Axual-platform operators to {xref-enable-axual-connect}, in case this functionality is not yet available for you.

Connect-Applications

Differences between Connect-Applications and custom applications:

  1. For Connect-Applications, you need to select a "Plugin type" which corresponds to the system you are integrating with (e.g. JDBC, MQTT, Cassandra, etc.), instead of choosing an "Application type" (e.g. Java, Python, Rest, etc.).

  2. Regular custom applications require a Certificate PEM file. Connector-Applications also require the Private key associated with that certificate. This is simply because the private key must be available to the running program: since the Custom applications run on the tenant’s infrastructure, the key is with them; Connectors run inside the Connect-Cluster, so the key must be made available within it.

  3. You can see the status of evert Connector-Application in the Self-Service portal. You can also start and stop them from the same place. Custom application lifecycles are handled by the developers and no information about their runtime is displayed in the portal.