Axual Connect
This reference describes Axual Connect: the terms it uses for clusters, plugins, applications and connectors, its API and rolling restart, how it is installed, the connector settings most plugins share, and how connector logs reach Self-Service.
Type |
Reference |
Goal |
Look up what an Axual Connect concept, operation or connector setting means. |
Audience |
Platform Operators running Axual Connect, and App Owners configuring a Connect application. |
When to use |
While configuring a Connect application, or while operating an existing Axual Connect deployment. |
|
Axual Connect is deprecated. See Why Kafka Connect replaces Axual Connect for the deprecation timeline. Its replacement, Kafka Connect, gives each tenant its own Strimzi-managed cluster instead of one cluster shared by every tenant on the instance. New installations use Kafka Connect. To move an existing installation across, follow Migrating from Axual Connect to Kafka Connect. |
Axual Connect is Apache Kafka Connect integrated into Axual Self-Service, running as one cluster shared by all tenants on an instance. Connect is a framework that runs Connector Tasks, which sync data between a Kafka cluster and another system such as a data lake, an SQL database or a message queue.
Many vendors publish Connector plugins as reusable components, and most of them work with the Axual Platform.
Connect and Connectors
The terms below appear across every Axual Connect page and in the Self-Service screens.
Glossary of terms
-
Connect Cluster: A cluster of nodes running Kafka-Connect. All Connector Tasks run on these machines. This cluster is usually owned and operated by Axual.
-
Connect Plugin: A generic program (delivered as a JAR file) which can be configured to integrate an external system with Apache-Kafka. Making an OOP analogy, this can be seen as a "class": it has no runtime of its own, but it can create multiple instances of itself when given the required configuration.
-
Connect Application: An Axual term for a Self-Service application that manages a group of Connectors of the same Plugin type. It governs connector data the same way a regular Kafka application governs its own.
-
Connector: A configured instance of a Connect Plugin. Continuing the OOP analogy, this is an "Object": a runtime entity. Several Connectors of the same Connect Plugin type can exist at the same time. A Connector arrives preconfigured for the Kafka cluster, so a developer supplies only the configuration that reaches the other system.
-
Connector Application: A Connect Application can start one Connector Application (an instance of itself) per environment. Technically it is a Connector deployed onto an Axual Environment.
-
Connector Task: Connectors usually run as several "tasks". This is how Connectors scale: several parallel, distributed processes, all sharing one configuration.
Connect Applications
Connect Applications differ from regular Kafka applications in three ways:
-
For a Connect Application you select a Plugin type matching the system you integrate with (JDBC, MQTT, Cassandra, and others), instead of an Application type (Java, Python, Rest, and others).
-
A regular Kafka application needs a Certificate PEM file. A Connector Application needs the Private key belonging to that certificate as well. The running program has to hold the private key: a custom application runs on the tenant’s own infrastructure, where the key already sits, while a Connector runs inside the Connect Cluster, so the key has to reach the cluster.
-
Self-Service shows the status of every Connector Application, and you start and stop them from the same screen. Developers own the lifecycle of a custom application, and Self-Service shows nothing about its runtime.
Architecture
Deploy Connect as several Connect Nodes spread over availability zones or racks within Kubernetes, which is what gives it Resilience. Spread them with Kubernetes AffinityRules.
API and interactions
The Connect API answers requests that Self-Service normally makes on your behalf, and an operator reaches it directly only to diagnose a problem or to reload plugins.
API
The Connect framework has an API that stays unexposed, so nobody unauthorised reads Connector configuration and credentials.
Reach the Connect API through Self-Service or its API rather than directly.
For debugging, an operator can reach the Connect API by port-forwarding the Connect Service, or by creating an internal Ingress.
The Apache Kafka REST API documentation describes the endpoints.
Rolling restart
For configuration or image changes, use a GitOps & CI/CD approach. The restart below is an ad-hoc action, for reloading Connector plugins.
A Kubernetes Deployment runs Axual Connect, with the default Update Strategy of RollingUpdate. Kubernetes re-creates the Pods one by one and terminates an old Pod only once its replacement reports ready.
First get the Deployment name ending with -axual-connect:
kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
...
tenant-instance-axual-connect 3/3 3 3 15mThen restart the Deployment, and optionally watch the pods roll:
kubectl rollout restart deploy tenant-instance-axual-connect
kubectl get pods -o wide -wView output
NAME READY STATUS AGE IP
nhn-instance-axual-connect-dff54dcb-678lj 0/1 Init:1/3 1s 10.244.20.114
nhn-instance-axual-connect-dff54dcb-678lj 0/1 Init:1/3 2s 10.244.20.114
nhn-instance-axual-connect-dff54dcb-678lj 0/1 Init:2/3 30s 10.244.20.114
nhn-instance-axual-connect-dff54dcb-678lj 0/1 Init:2/3 31s 10.244.20.114
nhn-instance-axual-connect-dff54dcb-678lj 0/1 PodInitializing 32s 10.244.20.114
nhn-instance-axual-connect-dff54dcb-678lj 0/1 Running 33s 10.244.20.114
nhn-instance-axual-connect-dff54dcb-678lj 1/1 Running 40s 10.244.20.114
nhn-instance-axual-connect-6c47d98cb-x77kv 1/1 Terminating 2m44s 10.244.20.112
nhn-instance-axual-connect-dff54dcb-45wrj 0/1 Pending 0s <none>
nhn-instance-axual-connect-dff54dcb-45wrj 0/1 Pending 0s <none>
nhn-instance-axual-connect-dff54dcb-45wrj 0/1 Init:0/3 0s <none>
nhn-instance-axual-connect-6c47d98cb-x77kv 0/1 Terminating 2m45s <none>
nhn-instance-axual-connect-6c47d98cb-x77kv 0/1 Terminating 2m46s 10.244.20.112
nhn-instance-axual-connect-dff54dcb-45wrj 0/1 Init:1/3 2s 10.244.20.115
nhn-instance-axual-connect-6c47d98cb-x77kv 0/1 Terminating 2m46s 10.244.20.112
nhn-instance-axual-connect-6c47d98cb-x77kv 0/1 Terminating 2m46s 10.244.20.112
nhn-instance-axual-connect-dff54dcb-45wrj 0/1 Init:1/3 3s 10.244.20.115
nhn-instance-axual-connect-dff54dcb-45wrj 0/1 Init:2/3 30s 10.244.20.115
nhn-instance-axual-connect-dff54dcb-45wrj 0/1 Init:2/3 31s 10.244.20.115
nhn-instance-axual-connect-dff54dcb-45wrj 0/1 PodInitializing 32s 10.244.20.115
nhn-instance-axual-connect-dff54dcb-45wrj 0/1 Running 33s 10.244.20.115
nhn-instance-axual-connect-dff54dcb-45wrj 1/1 Running 40s 10.244.20.115
nhn-instance-axual-connect-6c47d98cb-xh9k5 1/1 Terminating 2m43s 10.244.20.113
nhn-instance-axual-connect-dff54dcb-xdtlj 0/1 Pending 0s <none>
nhn-instance-axual-connect-dff54dcb-xdtlj 0/1 Pending 0s <none>
nhn-instance-axual-connect-dff54dcb-xdtlj 0/1 Init:0/3 0s <none>
nhn-instance-axual-connect-6c47d98cb-xh9k5 0/1 Terminating 2m45s <none>
nhn-instance-axual-connect-dff54dcb-xdtlj 0/1 Init:1/3 2s 10.244.20.116
nhn-instance-axual-connect-6c47d98cb-xh9k5 0/1 Terminating 2m45s 10.244.20.113
nhn-instance-axual-connect-6c47d98cb-xh9k5 0/1 Terminating 2m45s 10.244.20.113
nhn-instance-axual-connect-6c47d98cb-xh9k5 0/1 Terminating 2m45s 10.244.20.113
nhn-instance-axual-connect-dff54dcb-xdtlj 0/1 Init:1/3 3s 10.244.20.116
nhn-instance-axual-connect-dff54dcb-xdtlj 0/1 Init:2/3 30s 10.244.20.116
nhn-instance-axual-connect-dff54dcb-xdtlj 0/1 Init:2/3 31s 10.244.20.116
nhn-instance-axual-connect-dff54dcb-xdtlj 0/1 PodInitializing 32s 10.244.20.116
nhn-instance-axual-connect-dff54dcb-xdtlj 0/1 Running 33s 10.244.20.116
nhn-instance-axual-connect-dff54dcb-xdtlj 1/1 Running 41s 10.244.20.116
nhn-instance-axual-connect-6c47d98cb-rvg7h 1/1 Terminating 4m55s 10.244.20.111
nhn-instance-axual-connect-6c47d98cb-rvg7h 0/1 Terminating 4m56s <none>
nhn-instance-axual-connect-6c47d98cb-rvg7h 0/1 Terminating 4m56s 10.244.20.111
nhn-instance-axual-connect-6c47d98cb-rvg7h 0/1 Terminating 4m56s 10.244.20.111
nhn-instance-axual-connect-6c47d98cb-rvg7h 0/1 Terminating 4m56s 10.244.20.111Installation
Axual Connect depends on Kafka, a plugin download location, Self-Service, and Vault, which stores the connector credentials.
Helm Charts
The Connect Helm charts are standalone. Axual Connect 0.2.1 Helm Readme lists every value they accept.
Configuration
Each Connect Application carries its own set of properties. Check the Connector’s own documentation for what it expects. To view the Connect settings, open the Connect Application and click the cog on the connector application graphic in the middle of the page.
The settings are grouped below by the connectors that accept them.
Settings shared by most connectors
The three converter settings below apply to sink and source connectors alike:
|
Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the keys in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro. Type: class Default: Valid Values: Importance: high |
|
Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro. Type: class Default: Valid Values: Importance: high |
|
HeaderConverter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the header values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro. By default, the SimpleHeaderConverter is used to serialize header values to strings and deserialize them by inferring the schemas. Type: class Default: org.apache.kafka.connect.storage.SimpleHeaderConverter Valid Values: Importance: low |
Sink connector settings
A sink connector reads from Kafka and writes to an external system, so it also accepts the topic selection and error-handling settings below:
|
Globally unique name to use for this connector. Type: string Default: Valid Values: non-empty string without ISO control characters Importance: high |
|
Maximum number of tasks to use for this connector. Type: int Default: 1 Valid Values: [1,…] Importance: high |
|
List of topics to consume, separated by commas Type: list Default: "" Valid Values: Importance: high |
|
Regular expression giving topics to consume. Under the hood, the regex is compiled to a java.util.regex.Pattern. Only one of topics or topics.regex should be specified. Type: string Default: "" Valid Values: valid regex Importance: high |
|
Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the keys in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro. Type: class Default: null Valid Values: Importance: low |
|
Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro. Type: class Default: null Valid Values: Importance: low |
|
HeaderConverter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the header values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro. By default, the SimpleHeaderConverter is used to serialize header values to strings and deserialize them by inferring the schemas. Type: class Default: null Valid Values: Importance: low |
|
The action that Connect should take on the connector when changes in external configuration providers result in a change in the connector’s configuration properties. A value of 'none' indicates that Connect will do nothing. A value of 'restart' indicates that Connect should restart/reload the connector with the updated configuration properties. The restart may be scheduled in the future if the external configuration provider indicates that a configuration value will expire in the future. Type: string Default: restart Valid Values: [none, restart] Importance: low |
|
Aliases for the transformations to be applied to records. Type: list Default: "" Valid Values: non-null string, unique transformation aliases Importance: low |
|
Aliases for the predicates used by transformations. Type: list Default: "" Valid Values: non-null string, unique predicate aliases Importance: low |
|
The maximum duration in milliseconds that a failed operation will be reattempted. The default is 0, which means no retries will be attempted. Use -1 for infinite retries. Type: long Default: 0 Valid Values: Importance: medium |
|
The maximum duration in milliseconds between consecutive retry attempts. Jitter will be added to the delay once this limit is reached to prevent thundering herd issues. Type: long Default: 60000 (1 minute) Valid Values: Importance: medium |
|
Behaviour for tolerating errors during connector operation. 'none' is the default value and signals that any error will result in an immediate connector task failure; 'all' changes the behaviour to skip over problematic records. Type: string Default: none Valid Values: [none, all] Importance: medium |
|
If true, write each error and the details of the failed operation and problematic record to the Connect application log. This is 'false' by default, so that only errors that are not tolerated are reported. Type: boolean Default: false Valid Values: Importance: medium |
|
Whether to include the Connect record in the log that resulted in a failure. This is 'false' by default, which will prevent record keys, values, and headers from being written to log files, although some information such as topic and partition number will still be logged. Type: boolean Default: false Valid Values: Importance: medium |
|
The name of the topic to be used as the dead letter queue (DLQ) for messages that result in an error when processed by this sink connector, or its transformations or converters. The topic name is blank by default, which means that no messages are to be recorded in the DLQ. Type: string Default: "" Valid Values: Importance: medium |
|
Replication factor used to create the dead letter queue topic when it doesn’t already exist. Type: short Default: 3 Valid Values: Importance: medium |
|
If true, add headers containing error context to the messages written to the dead letter queue. To avoid clashing with headers from the original record, all error context header keys start with Type: boolean Default: false Valid Values: Importance: medium |
Source connector settings
A source connector reads from an external system and writes to Kafka, so it accepts topic creation settings in place of the topic selection ones:
|
Globally unique name to use for this connector. Type: string Default: Valid Values: non-empty string without ISO control characters Importance: high |
|
Name or alias of the class for this connector. Must be a subclass of org.apache.kafka.connect.connector.Connector. If the connector is org.apache.kafka.connect.file.FileStreamSinkConnector, you can either specify this full name, or use "FileStreamSink" or "FileStreamSinkConnector" to make the configuration a bit shorter Type: string Default: Valid Values: Importance: high |
|
Maximum number of tasks to use for this connector. Type: int Default: 1 Valid Values: [1,…] Importance: high |
|
Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the keys in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro. Type: class Default: null Valid Values: Importance: low |
|
Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro. Type: class Default: null Valid Values: Importance: low |
|
HeaderConverter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the header values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro. By default, the SimpleHeaderConverter is used to serialize header values to strings and deserialize them by inferring the schemas. Type: class Default: null Valid Values: Importance: low |
|
The action that Connect should take on the connector when changes in external configuration providers result in a change in the connector’s configuration properties. A value of 'none' indicates that Connect will do nothing. A value of 'restart' indicates that Connect should restart/reload the connector with the updated configuration properties. The restart may be scheduled in the future if the external configuration provider indicates that a configuration value will expire in the future. Type: string Default: restart Valid Values: [none, restart] Importance: low |
|
Aliases for the transformations to be applied to records. Type: list Default: "" Valid Values: non-null string, unique transformation aliases Importance: low |
|
Aliases for the predicates used by transformations. Type: list Default: "" Valid Values: non-null string, unique predicate aliases Importance: low |
|
The maximum duration in milliseconds that a failed operation will be reattempted. The default is 0, which means no retries will be attempted. Use -1 for infinite retries. Type: long Default: 0 Valid Values: Importance: medium |
|
The maximum duration in milliseconds between consecutive retry attempts. Jitter will be added to the delay once this limit is reached to prevent thundering herd issues. Type: long Default: 60000 (1 minute) Valid Values: Importance: medium |
|
Behaviour for tolerating errors during connector operation. 'none' is the default value and signals that any error will result in an immediate connector task failure; 'all' changes the behaviour to skip over problematic records. Type: string Default: none Valid Values: [none, all] Importance: medium |
|
If true, write each error and the details of the failed operation and problematic record to the Connect application log. This is 'false' by default, so that only errors that are not tolerated are reported. Type: boolean Default: false Valid Values: Importance: medium |
|
Whether to include the Connect record in the log that resulted in a failure. This is 'false' by default, which will prevent record keys, values, and headers from being written to log files, although some information such as topic and partition number will still be logged. Type: boolean Default: false Valid Values: Importance: medium |
|
Groups of configurations for topics created by source connectors Type: list Default: "" Valid Values: non-null string, unique topic creation groups Importance: low |
Logging
Axual Connect can also write the logs of Connector tasks onto a Kafka topic, so App Owners who cannot reach the cluster read their connector logs from Kafka instead. See Enabling Connector Logging into Kafka.