Azure Cosmos DB sink connector, version 1.14.2

Azure Cosmos DB

This document makes the following assumptions:

  • You already have an Azure CosmosDB Account

  • You have an Azure CosmosDB container (optionally named 'cosmosdb-testing').

  • You have access to the master key (Azure CosmosDB primary key).

  • You have access to a kafka producer, which you can configure and run whenever you want

Configuring a new sink Connector

  1. Follow the Creating streams documentation in order to create one stream and deploy it onto an environment.
    The name of the stream will be hotels.
    The key/value types will be JSON/JSON.

  2. Follow the Configure and install a connector documentation to set up a new Connector-Application.
    Let’s call it my-custom-cosmosdb-instance.
    The plugin name is com.azure.cosmos.kafka.connect.sink.CosmosDBSinkConnector.
    If a plugin isn’t available, ask a platform operator to install plugins.

  3. Provide the following minimal configuration in order to connect to the previously configured Azure Cosmos DB instance.
    For advanced configuration, see the official connector documentation.

    key.converter

    org.apache.kafka.connect.json.JsonConverter3

    topics

    hotels

    value.converter

    org.apache.kafka.connect.json.JsonConverter

    connect.cosmos.connection.endpoint

    (https://<cosmosinstance-name>.documents.azure.com:443/)

    connect.cosmos.master.key

    Insert Azure CosmosDB Master Key

    connect.cosmos.containers.topicmap

    Format is topic#containername. Example: hotels#cosmosdbtesting

    connect.cosmos.databasename

    Insert name of database container. Example: cosmosdbtesting

  4. Authorize the my-custom-cosmosdb-instance sink Connector-Application to consume the hotels stream.

  5. You can now start the sink Connector-Application.

  6. Produce some JSON/JSON events to this stream. Follow the producer documentation and examples if needed.

  7. You can now check the Azure container to see the events published by the Connector.

Cleanup

Once you are done, stop the Connector-Application and cleanup the unused axual resources.
Don’t forget to return to Azure and delete your service account and container.

License

Azure Cosmos DB sink-Connector is licensed under the MIT license.

Source code

The source code for the Connect-Plugin can be found at https://github.com/microsoft/kafka-connect-cosmosdb.