What is "Connect"? What are "Connectors"?

Glossary of terms

  • Connect Cluster: A cluster of nodes running Kafka-Connect. All Connector Tasks run on these machines. This cluster is usually owned and operated by Axual.

  • Connect Plugin: A generic program (delivered as a JAR file) which can be configured to integrate an external system with Apache-Kafka. Making an OOP analogy, this can be seen as a "class": it has no runtime of its own, but it can create multiple instances of itself when given the required configuration.

  • Connect Application: A term used within the Axual ecosystem to refer to a Self-Service application that manages a group of Connectors of the same Plugin type. This resource helps with facilitating data governance, the Axual way, the same way we do with regular Kafka applications.

  • Connector: A configured instance of a Connect Plugin. Continuing the OOP analogy, this can be seen as an "Object": a runtime entity. Multiple Connectors of the same Connect Plugin-type can exist at the same time. Connectors are preconfigured to connect to the kafka cluster, so a developer only needs to supply configuration required to reach the other system.

  • Connector Application: A Connect Application can start one Connector Application (an instance of itself) per environment. This is technically just a Connector, deployed onto an Axual-Environment.

  • Connector Task: Connectors are generally run by using multiple "tasks". This is how Connectors scale: by having multiple parallel (and distributed) processes. All instances have the same configuration.

Connect Applications

Differences between Connect Applications and regular Kafka applications:

  1. For Connect Applications, you need to select a "Plugin type" which corresponds to the system you are integrating with (e.g. JDBC, MQTT, Cassandra, etc.), instead of choosing an "Application type" (e.g. Java, Python, Rest, etc.).

  2. Regular Kafka applications require a Certificate PEM file. Connector Applications also require the Private key associated with that certificate. This is simply because the private key must be available to the running program: since the Custom applications run on the tenant’s infrastructure, the key is with them; Connectors run inside the Connect Cluster, so the key must be made available within it.

  3. You can see the status of evert Connector Application in the Self-Service portal. You can also start and stop them from the same place. Custom application lifecycles are handled by the developers and no information about their runtime is displayed in the portal.