Example configurations for the ADLS Gen2 Sink Connector
Examples
-
Example 1 - Minimal configuration with Account Key Authentication
-
Example 2 - Minimal configuration with Client Secret Authentication
-
Example 3 - Snappy compression with Account Key Authentication
-
Example 5 - Retry and pattern target with Account Key Authentication
-
Example 6 - Full configuration with Account Key Authentication
Example 1 - Minimal configuration with Account Key Authentication
This example will connect to the Azure Data Lake Storage account named and myexampleadlsaccount use the container test-container to create a base directory minimal-fixed-target and use the default staging and target directories.
{
"name": "minimal-fixed-target-account-key",
"config": {
"connector.class": "io.axual.connect.plugins.adls.gen2.AdlsGen2SinkConnector",
"tasks.max": "3",
"topics": "test-topic1",
"adls.endpoint": "https://myexampleadlsaccount.dfs.core.windows.net",
"adls.container.name": "test-container",
"adls.auth.method": "AccountKey",
"adls.account.name": "myexampleadlsaccount",
"adls.account.key": "get-this-from-the-azure-storage-account-settings",
"base.directory": "minimal-fixed-target"
}
}
Example 2 - Minimal configuration with Client Secret Authentication
This example will connect to the Azure Data Lake Storage account named and myexampleadlsaccount use the container test-container to create a base directory minimal-fixed-target and use the default staging and target directories.
{
"name": "minimal-fixed-target-client-secret",
"config": {
"connector.class": "io.axual.connect.plugins.adls.gen2.AdlsGen2SinkConnector",
"tasks.max": "3",
"topics": "test-topic1",
"adls.endpoint": "https://myexampleadlsaccount.dfs.core.windows.net",
"adls.container.name": "test-container",
"adls.auth.method": "ClientSecret",
"adls.tenant.id": "get.tenant.id.from.azure.ad",
"adls.client.id": "get.client.id.from.azure.ad.user.details",
"adls.client.secret": "get.client.secret.from.azure.ad.user.details",
"base.directory": "minimal-fixed-target"
}
}
Example 3 - Snappy compression with Account Key Authentication
This example will connect to the Azure Data Lake Storage account named and myexampleadlsaccount use the container test-container to create a base directory minimal-fixed-target and use the default staging and target directories.
Snappy compression is enabled to create smaller Avro Object Container files.
{
"name": "snappy-fixed-target-account-key",
"config": {
"connector.class": "io.axual.connect.plugins.adls.gen2.AdlsGen2SinkConnector",
"tasks.max": "3",
"topics": "test-topic1",
"adls.endpoint": "https://myexampleadlsaccount.dfs.core.windows.net",
"adls.container.name": "test-container",
"adls.auth.method": "AccountKey",
"adls.account.name": "myexampleadlsaccount",
"adls.account.key": "get-this-from-the-azure-storage-account-settings",
"base.directory": "minimal-fixed-target",
"compression": "snappy"
}
}
Example 4 - Pattern target with Account Key Authentication
This example will connect to the Azure Data Lake Storage account named and myexampleadlsaccount use the container test-container to create a base directory minimal-fixed-target and use custom staging and target directories.
A pattern is used as target directory to create a directory structure based on year/month/day.
The record timestamp will be the used for resolving the pattern, and the GMT timezone is used for creating the path.
{
"name": "pattern-target-account-key",
"config": {
"connector.class": "io.axual.connect.plugins.adls.gen2.AdlsGen2SinkConnector",
"tasks.max": "3",
"topics": "test-topic1",
"adls.endpoint": "https://myexampleadlsaccount.dfs.core.windows.net",
"adls.container.name": "test-container",
"adls.auth.method": "AccountKey",
"adls.account.name": "myexampleadlsaccount",
"adls.account.key": "get-this-from-the-azure-storage-account-settings",
"base.directory": "minimal-fixed-target",
"staging.directory": "staging",
"target.directory": "{yyyy}/{MM}/{dd}",
"rotation.time.source": "produced",
"rotation.time.zone": "GMT"
}
}
Example 5 - Retry and pattern target with Account Key Authentication
This example will connect to the Azure Data Lake Storage account named and myexampleadlsaccount use the container test-container to create a base directory minimal-fixed-target and use custom staging and target directories.
A pattern is used as target directory to create a directory structure based on year/month/day.
The record timestamp will be the used for resolving the pattern, and the GMT timezone is used for creating the path.
The Azure client client will use 10 retries, a call timeout of 20 seconds, with exponential backoff enabled.
The retry interval is 2 seconds with a maximum of 30 seconds.
The overall retry will use 8 retries, exponential backoff, an interval of 1 seconds with a maximum of 1 minute.
{
"name": "retrying-pattern-target-account-key",
"config": {
"connector.class": "io.axual.connect.plugins.adls.gen2.AdlsGen2SinkConnector",
"tasks.max": "3",
"topics": "test-topic1",
"adls.endpoint": "https://myexampleadlsaccount.dfs.core.windows.net",
"adls.container.name": "test-container",
"adls.auth.method": "AccountKey",
"adls.account.name": "myexampleadlsaccount",
"adls.account.key": "get-this-from-the-azure-storage-account-settings",
"base.directory": "minimal-fixed-target",
"staging.directory": "staging",
"target.directory": "{yyyy}/{MM}/{dd}",
"rotation.time.source": "produced",
"rotation.time.zone": "GMT",
"adls.client.retry.maximum.tries": "10",
"adls.client.timeout.seconds": "20",
"adls.client.retry.exponential": "true",
"adls.client.retry.interval": "2000",
"adls.client.retry.maximum.interval": "60000",
"retry.maximum.tries": "8",
"retry.exponential": "true",
"retry.interval": "1000",
"retry.maximum.interval": "60000"
}
}
Example 6 - Full configuration with Account Key Authentication
This example will connect to the Azure Data Lake Storage account named and myexampleadlsaccount use the container test-container to create a base directory minimal-fixed-target and use custom staging and target directories.
A pattern is used as target directory to create a directory structure based on year/month/day.
The record timestamp will be the used for resolving the pattern, and the GMT timezone is used for creating the path.
The Azure client client will use 10 retries, a call timeout of 20 seconds, with exponential backoff enabled.
The retry interval is 2 seconds with a maximum of 30 seconds.
The overall retry will use 8 retries, exponential backoff, an interval of 1 seconds with a maximum of 1 minute.
{
"name": "full-config-account-key",
"config": {
"connector.class": "io.axual.connect.plugins.adls.gen2.AdlsGen2SinkConnector",
"tasks.max": "3",
"topics": "test-topic1",
"adls.endpoint": "https://myexampleadlsaccount.dfs.core.windows.net",
"adls.container.name": "test-container",
"adls.auth.method": "AccountKey",
"adls.account.name": "myexampleadlsaccount",
"adls.account.key": "get-this-from-the-azure-storage-account-settings",
"base.directory": "minimal-fixed-target",
"staging.directory": "staging",
"target.directory": "{yyyy}/{MM}/{dd}",
"rotation.time.source": "produced",
"rotation.time.zone": "GMT",
"adls.client.retry.maximum.tries": "10",
"adls.client.timeout.seconds": "20",
"adls.client.retry.exponential": "true",
"adls.client.retry.interval": "2000",
"adls.client.retry.maximum.interval": "60000",
"retry.maximum.tries": "8",
"retry.exponential": "true",
"retry.interval": "1000",
"retry.maximum.interval": "60000",
"rotation.record.count": "20000",
"rotation.inactivity": "3600000",
"rotation.filesize": "50000000",
"commit.rotated.only": "true",
"commit.record.count": "5000",
"compression": "snappy",
"sync.interval": "64000"
}
}