Example configurations for the ADLS Gen2 Sink Connector
Examples
-
Example 1 - Minimal configuration with Account Key Authentication
-
Example 2 - Minimal configuration with Client Secret Authentication
-
Example 3 - Snappy compression with Account Key Authentication
-
Example 5 - Retry and pattern target with Account Key Authentication
-
Example 6 - Full configuration with Account Key Authentication
Example 1 - Minimal configuration with Account Key Authentication
This example will connect to the Azure Data Lake Storage account named and myexampleadlsaccount use the container test-container to create a base directory minimal-fixed-target and use the default staging and target directories.
Unresolved include directive in modules/ROOT/pages/connect-plugins-catalog/adls-gen2-sink/examples.adoc - include::example$/adls-gen2-sink/minimal-fixed-target-account-key.json[]Example 2 - Minimal configuration with Client Secret Authentication
This example will connect to the Azure Data Lake Storage account named and myexampleadlsaccount use the container test-container to create a base directory minimal-fixed-target and use the default staging and target directories.
Unresolved include directive in modules/ROOT/pages/connect-plugins-catalog/adls-gen2-sink/examples.adoc - include::example$/adls-gen2-sink/minimal-fixed-target-client-secret.json[]Example 3 - Snappy compression with Account Key Authentication
This example will connect to the Azure Data Lake Storage account named and myexampleadlsaccount use the container test-container to create a base directory minimal-fixed-target and use the default staging and target directories.
Snappy compression is enabled to create smaller Avro Object Container files.
Unresolved include directive in modules/ROOT/pages/connect-plugins-catalog/adls-gen2-sink/examples.adoc - include::example$/adls-gen2-sink/snappy-fixed-target-account-key.json[]Example 4 - Pattern target with Account Key Authentication
This example will connect to the Azure Data Lake Storage account named and myexampleadlsaccount use the container test-container to create a base directory minimal-fixed-target and use custom staging and target directories.
A pattern is used as target directory to create a directory structure based on year/month/day.
The record timestamp will be the used for resolving the pattern, and the GMT timezone is used for creating the path.
Unresolved include directive in modules/ROOT/pages/connect-plugins-catalog/adls-gen2-sink/examples.adoc - include::example$/adls-gen2-sink/pattern-target-account-key.json[]Example 5 - Retry and pattern target with Account Key Authentication
This example will connect to the Azure Data Lake Storage account named and myexampleadlsaccount use the container test-container to create a base directory minimal-fixed-target and use custom staging and target directories.
A pattern is used as target directory to create a directory structure based on year/month/day.
The record timestamp will be the used for resolving the pattern, and the GMT timezone is used for creating the path.
The Azure client client will use 10 retries, a call timeout of 20 seconds, with exponential backoff enabled.
The retry interval is 2 seconds with a maximum of 30 seconds.
The overall retry will use 8 retries, exponential backoff, an interval of 1 seconds with a maximum of 1 minute.
Unresolved include directive in modules/ROOT/pages/connect-plugins-catalog/adls-gen2-sink/examples.adoc - include::example$/adls-gen2-sink/retrying-pattern-target-account-key.json[]Example 6 - Full configuration with Account Key Authentication
This example will connect to the Azure Data Lake Storage account named and myexampleadlsaccount use the container test-container to create a base directory minimal-fixed-target and use custom staging and target directories.
A pattern is used as target directory to create a directory structure based on year/month/day.
The record timestamp will be the used for resolving the pattern, and the GMT timezone is used for creating the path.
The Azure client client will use 10 retries, a call timeout of 20 seconds, with exponential backoff enabled.
The retry interval is 2 seconds with a maximum of 30 seconds.
The overall retry will use 8 retries, exponential backoff, an interval of 1 seconds with a maximum of 1 minute.
Unresolved include directive in modules/ROOT/pages/connect-plugins-catalog/adls-gen2-sink/examples.adoc - include::example$/adls-gen2-sink/full-config-account-key.json[]