Example configurations for the ADLS Gen2 Sink Connector

Example 1 - Minimal configuration with Account Key Authentication

This example will connect to the Azure Data Lake Storage account named and myexampleadlsaccount use the container test-container to create a base directory minimal-fixed-target and use the default staging and target directories.

{
  "name": "minimal-fixed-target-account-key",
  "config": {
    "connector.class": "io.axual.connect.plugins.adls.gen2.AdlsGen2SinkConnector",
    "tasks.max": "3",
    "topics": "test-topic1",
    "adls.endpoint": "https://myexampleadlsaccount.dfs.core.windows.net",
    "adls.container.name": "test-container",
    "adls.auth.method": "AccountKey",
    "adls.account.name": "myexampleadlsaccount",
    "adls.account.key": "get-this-from-the-azure-storage-account-settings",
    "base.directory": "minimal-fixed-target"
  }
}

Example 2 - Minimal configuration with Client Secret Authentication

This example will connect to the Azure Data Lake Storage account named and myexampleadlsaccount use the container test-container to create a base directory minimal-fixed-target and use the default staging and target directories.

{
  "name": "minimal-fixed-target-client-secret",
  "config": {
    "connector.class": "io.axual.connect.plugins.adls.gen2.AdlsGen2SinkConnector",
    "tasks.max": "3",
    "topics": "test-topic1",
    "adls.endpoint": "https://myexampleadlsaccount.dfs.core.windows.net",
    "adls.container.name": "test-container",
    "adls.auth.method": "ClientSecret",
    "adls.tenant.id": "get.tenant.id.from.azure.ad",
    "adls.client.id": "get.client.id.from.azure.ad.user.details",
    "adls.client.secret": "get.client.secret.from.azure.ad.user.details",
    "base.directory": "minimal-fixed-target"
  }
}

Example 3 - Snappy compression with Account Key Authentication

This example will connect to the Azure Data Lake Storage account named and myexampleadlsaccount use the container test-container to create a base directory minimal-fixed-target and use the default staging and target directories.
Snappy compression is enabled to create smaller Avro Object Container files.

{
  "name": "snappy-fixed-target-account-key",
  "config": {
    "connector.class": "io.axual.connect.plugins.adls.gen2.AdlsGen2SinkConnector",
    "tasks.max": "3",
    "topics": "test-topic1",
    "adls.endpoint": "https://myexampleadlsaccount.dfs.core.windows.net",
    "adls.container.name": "test-container",
    "adls.auth.method": "AccountKey",
    "adls.account.name": "myexampleadlsaccount",
    "adls.account.key": "get-this-from-the-azure-storage-account-settings",
    "base.directory": "minimal-fixed-target",
    "compression": "snappy"
  }
}

Example 4 - Pattern target with Account Key Authentication

This example will connect to the Azure Data Lake Storage account named and myexampleadlsaccount use the container test-container to create a base directory minimal-fixed-target and use custom staging and target directories.
A pattern is used as target directory to create a directory structure based on year/month/day.
The record timestamp will be the used for resolving the pattern, and the GMT timezone is used for creating the path.

{
  "name": "pattern-target-account-key",
  "config": {
    "connector.class": "io.axual.connect.plugins.adls.gen2.AdlsGen2SinkConnector",
    "tasks.max": "3",
    "topics": "test-topic1",
    "adls.endpoint": "https://myexampleadlsaccount.dfs.core.windows.net",
    "adls.container.name": "test-container",
    "adls.auth.method": "AccountKey",
    "adls.account.name": "myexampleadlsaccount",
    "adls.account.key": "get-this-from-the-azure-storage-account-settings",
    "base.directory": "minimal-fixed-target",
    "staging.directory": "staging",
    "target.directory": "{yyyy}/{MM}/{dd}",
    "rotation.time.source": "produced",
    "rotation.time.zone": "GMT"
  }
}

Example 5 - Retry and pattern target with Account Key Authentication

This example will connect to the Azure Data Lake Storage account named and myexampleadlsaccount use the container test-container to create a base directory minimal-fixed-target and use custom staging and target directories.
A pattern is used as target directory to create a directory structure based on year/month/day. The record timestamp will be the used for resolving the pattern, and the GMT timezone is used for creating the path.
The Azure client client will use 10 retries, a call timeout of 20 seconds, with exponential backoff enabled. The retry interval is 2 seconds with a maximum of 30 seconds.
The overall retry will use 8 retries, exponential backoff, an interval of 1 seconds with a maximum of 1 minute.

{
  "name": "retrying-pattern-target-account-key",
  "config": {
    "connector.class": "io.axual.connect.plugins.adls.gen2.AdlsGen2SinkConnector",
    "tasks.max": "3",
    "topics": "test-topic1",
    "adls.endpoint": "https://myexampleadlsaccount.dfs.core.windows.net",
    "adls.container.name": "test-container",
    "adls.auth.method": "AccountKey",
    "adls.account.name": "myexampleadlsaccount",
    "adls.account.key": "get-this-from-the-azure-storage-account-settings",
    "base.directory": "minimal-fixed-target",
    "staging.directory": "staging",
    "target.directory": "{yyyy}/{MM}/{dd}",
    "rotation.time.source": "produced",
    "rotation.time.zone": "GMT",
    "adls.client.retry.maximum.tries": "10",
    "adls.client.timeout.seconds": "20",
    "adls.client.retry.exponential": "true",
    "adls.client.retry.interval": "2000",
    "adls.client.retry.maximum.interval": "60000",
    "retry.maximum.tries": "8",
    "retry.exponential": "true",
    "retry.interval": "1000",
    "retry.maximum.interval": "60000"
  }
}

Example 6 - Full configuration with Account Key Authentication

This example will connect to the Azure Data Lake Storage account named and myexampleadlsaccount use the container test-container to create a base directory minimal-fixed-target and use custom staging and target directories.
A pattern is used as target directory to create a directory structure based on year/month/day. The record timestamp will be the used for resolving the pattern, and the GMT timezone is used for creating the path.
The Azure client client will use 10 retries, a call timeout of 20 seconds, with exponential backoff enabled. The retry interval is 2 seconds with a maximum of 30 seconds.
The overall retry will use 8 retries, exponential backoff, an interval of 1 seconds with a maximum of 1 minute.

{
  "name": "full-config-account-key",
  "config": {
    "connector.class": "io.axual.connect.plugins.adls.gen2.AdlsGen2SinkConnector",
    "tasks.max": "3",
    "topics": "test-topic1",
    "adls.endpoint": "https://myexampleadlsaccount.dfs.core.windows.net",
    "adls.container.name": "test-container",
    "adls.auth.method": "AccountKey",
    "adls.account.name": "myexampleadlsaccount",
    "adls.account.key": "get-this-from-the-azure-storage-account-settings",
    "base.directory": "minimal-fixed-target",
    "staging.directory": "staging",
    "target.directory": "{yyyy}/{MM}/{dd}",
    "rotation.time.source": "produced",
    "rotation.time.zone": "GMT",
    "adls.client.retry.maximum.tries": "10",
    "adls.client.timeout.seconds": "20",
    "adls.client.retry.exponential": "true",
    "adls.client.retry.interval": "2000",
    "adls.client.retry.maximum.interval": "60000",
    "retry.maximum.tries": "8",
    "retry.exponential": "true",
    "retry.interval": "1000",
    "retry.maximum.interval": "60000",
    "rotation.record.count": "20000",
    "rotation.inactivity": "3600000",
    "rotation.filesize": "50000000",
    "commit.rotated.only": "true",
    "commit.record.count": "5000",
    "compression": "snappy",
    "sync.interval": "64000"
  }
}