Amazon S3-Bucket sink connector, version 2.12.0

Amazon S3-Bucket

This document makes the following assumptions:

  • You already have access to an AWS subscription

  • You have an S3 bucket (optionally named my-s3-kafka-connect-target) within this AWS subscription

  • You already have access to an AWS (service) account, which has read/write permissions on the bucket

  • You have a secret access key and key ID of the (service) account

  • You have access to a kafka producer, which you can configure and run whenever you want

Configuring a new source Connector

  1. Follow the Creating streams documentation in order to create one stream and deploy it onto an environment.
    The name of the stream will be my_s3_connector_source.
    The key/value types will be String/String.

  2. Follow the Configure and install a connector documentation to set up a new Connector-Application.
    Let’s call it my_s3_sink.
    The plugin name is io.aiven.kafka.connect.s3.AivenKafkaConnectS3SinkConnector.

    For advanced configuration, see the official connector documentation.

  3. Provide the following minimal configuration in order to connect to the previously configured Amazon S3-Bucket instance.

    aws.access.key.id

    Example value
    AYEM7RPD4TAXLHLPM333

    aws.secret.access.key

    Example value
    Hc89Cnwp3MnNvmYdJRzlOPOe2WZFWtXt7FndjRCi

    aws.s3.region

    Use the bucket’s real region
    eu-central-1

    aws.s3.bucket.name

    my-s3-kafka-connect-target

    format.output.type

    json

    topics

    my_s3_connector_source

    key.converter

    org.apache.kafka.connect.storage.StringConverter

    value.converter

    org.apache.kafka.connect.storage.StringConverter

  4. Authorize the my_s3_sink sink Connector-Application to consume the my_s3_connector_source stream.

  5. You can now start the sink Connector-Application.

  6. Produce some String/String events to this stream. Follow the producer documentation and examples if needed.

  7. You can now check the S3 bucket to see the events published by the Connector. The bucket will contain one or more compressed archives which you can download and decompress. Their contents will be JSON files containing the values you produced.

Cleanup

Once you are done, stop the Connector-Application and cleanup the unused axual resources.
Don’t forget to return to AWS and delete your service account and S3 bucket.

License

Amazon S3-Bucket sink-Connector is licensed under the Apache License, Version 2.0.

Source code

The source code for the Connect-Plugin can be found at github.com/aiven/s3-connector-for-apache-kafka.

© Copyright 2022, Axual B.V. All other trademarks, servicemarks, and copyrights are the property of their respective owners.