Azure Cosmos DB source connector, version 1.14.2

Azure Cosmos DB

This document makes the following assumptions:

  • You already have an Azure CosmosDB Account

  • You have an Azure CosmosDB container (optionally named 'cosmosdb-testing').

  • You have access to the master key (Azure CosmosDB primary key).

  • You have access to a kafka producer, which you can configure and run whenever you want

Configuring a new source Connector

  1. Follow the Creating streams documentation in order to create one stream and deploy it onto an environment.
    The name of the stream will be apparels.
    The key/value types will be JSON/JSON.

  2. Follow the Configure and install a connector documentation to set up a new Connector-Application.
    Let’s call it my-custom-cosmosdb-instance.
    The plugin name is com.azure.cosmos.kafka.connect.source.CosmosDBSourceConnector.
    If a plugin isn’t available, ask a platform operator to install plugins.

  3. Provide the following minimal configuration in order to connect to the previously configured Azure Cosmos DB instance.
    For advanced configuration, see the https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/kafka-connector-source#install-the-source-connector.

    key.converter

    org.apache.kafka.connect.json.JsonConverter

    topics

    apparels

    value.converter

    org.apache.kafka.connect.json.JsonConverter

    connect.cosmos.connection.endpoint

    (https://<cosmosinstance-name>.documents.azure.com:443/)

    connect.cosmos.master.key

    Insert Azure CosmosDB Master Key

    connect.cosmos.containers.topicmap

    Format is topic#containername. Example: apparels#cosmosdbtesting

    connect.cosmos.databasename

    Insert name of database container. Example: cosmosdbtesting

  4. Authorize the my-custom-cosmosdb-instance source Connector-Application to produce to the apparels stream.

  5. You can now start the source Connector-Application.

  6. Produce to some JSON/JSON events to this stream. Follow the producer documentation and examples if needed.

  7. You can now check the Azure container to see the events published by the Connector.

Cleanup

Once you are done, stop the Connector-Application and cleanup the unused axual resources.
Don’t forget to return to Azure and delete your service account and container.

License

Azure Cosmos DB source-Connector is licensed under the MIT license.

Source code

The source code for the Connect-Plugin can be found at https://github.com/microsoft/kafka-connect-cosmosdb.