Connector Configuration Reference
This reference lists the connector configuration properties Kafka Connect accepts: the converter settings shared by most connectors, then the settings specific to a sink connector and to a source connector, each with its type, default and importance.
Type |
Reference |
Goal |
Look up a connector configuration property and the value it accepts. |
Audience |
An application owner filling in a Connector Application’s plugin configuration. |
When to use |
While configuring a connector, alongside the connector’s own page in the catalogue. |
These are Apache Kafka Connect properties, so they apply on both Axual runtimes: Kafka Connect and the deprecated Axual Connect. A connector also accepts properties of its own, listed on its page in the Connect Plugins Catalogue.
To reach these settings in Self-Service, open the Connector Application and click the cog on the connector application graphic.
Settings shared by most connectors
Every connector reads the converter settings below, which control how keys, values and headers are serialised to and from Kafka.
On Kafka Connect, a connector that sets no key.converter or value.converter inherits the worker default from the kafka-connect-helm chart: org.apache.kafka.connect.json.JsonConverter with schemas.enable set to false.
|
Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the keys in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro. Type: class Default: Valid Values: Importance: high |
|
Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro. Type: class Default: Valid Values: Importance: high |
|
HeaderConverter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the header values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro. By default, the SimpleHeaderConverter is used to serialize header values to strings and deserialize them by inferring the schemas. Type: class Default: org.apache.kafka.connect.storage.SimpleHeaderConverter Valid Values: Importance: low |
Sink connector settings
These settings apply to a sink connector, which reads from Kafka and writes to the integrated system.
|
Globally unique name to use for this connector. Type: string Default: Valid Values: non-empty string without ISO control characters Importance: high |
|
Maximum number of tasks to use for this connector. Type: int Default: 1 Valid Values: [1,…] Importance: high |
|
List of topics to consume, separated by commas Type: list Default: "" Valid Values: Importance: high |
|
Regular expression giving topics to consume. Under the hood, the regex is compiled to a java.util.regex.Pattern. Only one of topics or topics.regex should be specified. Type: string Default: "" Valid Values: valid regex Importance: high |
|
Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the keys in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro. Type: class Default: null Valid Values: Importance: low |
|
Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro. Type: class Default: null Valid Values: Importance: low |
|
HeaderConverter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the header values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro. By default, the SimpleHeaderConverter is used to serialize header values to strings and deserialize them by inferring the schemas. Type: class Default: null Valid Values: Importance: low |
|
The action that Connect should take on the connector when changes in external configuration providers result in a change in the connector’s configuration properties. A value of 'none' indicates that Connect will do nothing. A value of 'restart' indicates that Connect should restart/reload the connector with the updated configuration properties.The restart may be scheduled in the future if the external configuration provider indicates that a configuration value will expire in the future. Type: string Default: restart Valid Values: [none, restart] Importance: low |
|
Aliases for the transformations to be applied to records. Type: list Default: "" Valid Values: non-null string, unique transformation aliases Importance: low predicates Aliases for the predicates used by transformations. Type: list Default: "" Valid Values: non-null string, unique predicate aliases Importance: low |
|
The maximum duration in milliseconds that a failed operation will be reattempted. The default is 0, which means no retries will be attempted. Use -1 for infinite retries. Type: long Default: 0 Valid Values: Importance: medium |
|
The maximum duration in milliseconds between consecutive retry attempts. Jitter will be added to the delay once this limit is reached to prevent thundering herd issues. Type: long Default: 60000 (1 minute) Valid Values: Importance: medium |
|
Behaviour for tolerating errors during connector operation. 'none' is the default value and signals that any error will result in an immediate connector task failure; 'all' changes the behaviour to skip over problematic records. Type: string Default: none Valid Values: [none, all] Importance: medium |
|
If true, write each error and the details of the failed operation and problematic record to the Connect application log. This is 'false' by default, so that only errors that are not tolerated are reported. Type: boolean Default: false Valid Values: Importance: medium |
|
Whether to the include in the log the Connect record that resulted in a failure. This is 'false' by default, which will prevent record keys, values, and headers from being written to log files, although some information such as topic and partition number will still be logged. Type: boolean Default: false Valid Values: Importance: medium |
|
The name of the topic to be used as the dead letter queue (DLQ) for messages that result in an error when processed by this sink connector, or its transformations or converters. The topic name is blank by default, which means that no messages are to be recorded in the DLQ. Type: string Default: "" Valid Values: Importance: medium |
|
Replication factor used to create the dead letter queue topic when it doesn’t already exist. Type: short Default: 3 Valid Values: Importance: medium |
|
If true, add headers containing error context to the messages written to the dead letter queue. To avoid clashing with headers from the original record, all error context header keys, all error context header keys will start with __connect.errors. Type: boolean Default: false Valid Values: Importance: medium |
Source connector settings
These settings apply to a source connector, which reads from the integrated system and writes to Kafka.
|
Globally unique name to use for this connector. Type: string Default: Valid Values: non-empty string without ISO control characters Importance: high connector.class Name or alias of the class for this connector. Must be a subclass of org.apache.kafka.connect.connector.Connector. If the connector is org.apache.kafka.connect.file.FileStreamSinkConnector, you can either specify this full name, or use "FileStreamSink" or "FileStreamSinkConnector" to make the configuration a bit shorter Type: string Default: Valid Values: Importance: high |
|
Maximum number of tasks to use for this connector. Type: int Default: 1 Valid Values: [1,…] Importance: high |
|
Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the keys in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro. Type: class Default: null Valid Values: Importance: low |
|
Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro. Type: class Default: null Valid Values: Importance: low |
|
HeaderConverter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the header values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro. By default, the SimpleHeaderConverter is used to serialize header values to strings and deserialize them by inferring the schemas. Type: class Default: null Valid Values: Importance: low |
|
The action that Connect should take on the connector when changes in external configuration providers result in a change in the connector’s configuration properties. A value of 'none' indicates that Connect will do nothing. A value of 'restart' indicates that Connect should restart/reload the connector with the updated configuration properties.The restart may be scheduled in the future if the external configuration provider indicates that a configuration value will expire in the future. Type: string Default: restart Valid Values: [none, restart] Importance: low |
|
Aliases for the transformations to be applied to records. Type: list Default: "" Valid Values: non-null string, unique transformation aliases Importance: low predicates Aliases for the predicates used by transformations. Type: list Default: "" Valid Values: non-null string, unique predicate aliases Importance: low |
|
The maximum duration in milliseconds that a failed operation will be reattempted. The default is 0, which means no retries will be attempted. Use -1 for infinite retries. Type: long Default: 0 Valid Values: Importance: medium |
|
The maximum duration in milliseconds between consecutive retry attempts. Jitter will be added to the delay once this limit is reached to prevent thundering herd issues. Type: long Default: 60000 (1 minute) Valid Values: Importance: medium |
|
Behaviour for tolerating errors during connector operation. 'none' is the default value and signals that any error will result in an immediate connector task failure; 'all' changes the behaviour to skip over problematic records. Type: string Default: none Valid Values: [none, all] Importance: medium |
|
If true, write each error and the details of the failed operation and problematic record to the Connect application log. This is 'false' by default, so that only errors that are not tolerated are reported. Type: boolean Default: false Valid Values: Importance: medium |
|
Whether to the include in the log the Connect record that resulted in a failure. This is 'false' by default, which will prevent record keys, values, and headers from being written to log files, although some information such as topic and partition number will still be logged. Type: boolean Default: false Valid Values: Importance: medium |
|
Groups of configurations for topics created by source connectors Type: list Default: "" Valid Values: non-null string, unique topic creation groups Importance: low |
Apicurio Registry properties
These properties are not ones you need to set yourself, and unlike the rest of this reference, they apply to Kafka Connect only, not the deprecated Axual Connect.
When the environment’s Instance uses Apicurio Registry with BasicAuth enabled, and the connector is deployed to a registered Kafka Connect cluster, Platform Manager injects them automatically into key.converter/value.converter, on whichever side’s converter class it recognises as Apicurio’s or Confluent’s.
See Apicurio Registry configuration for when and how this applies.
The Overridable column states whether a value you set explicitly for that property in Plugin Configuration is kept instead of the injected one. A property marked "No" is always set by Platform Manager, regardless of what your plugin configuration contains.
Apicurio converter
| Property | Value | Overridable |
|---|---|---|
|
The environment’s Apicurio Registry URL |
Yes |
|
The generated Apicurio Registry username |
No, always set by Platform Manager |
|
The generated Apicurio Registry password, injected as a secret reference rather than plaintext |
No, always set by Platform Manager |
|
|
Yes |
|
|
Yes |
|
|
Yes |
|
|
Yes |
|
A connector’s converter must agree with |
Confluent converter
| Property | Value | Overridable |
|---|---|---|
|
The environment’s Apicurio Registry URL, in Confluent’s connection format |
Yes |
|
|
No, always set by Platform Manager |
|
The generated Apicurio Registry username and password combined, injected as a secret reference rather than plaintext |
No, always set by Platform Manager |
|
|
Yes |
| In the rare case where the environment’s Apicurio Registry endpoint isn’t in a format Platform Manager can translate for Confluent’s client, the Confluent side is left unconfigured (this is logged, but not surfaced as an API error). An Apicurio-vendor converter on the same connector still receives its configuration as normal. |