Couchbase source and sink Connector, version 4.1.9

Couchbase

If you already have a publicly accessible deployment of Couchbase, you can skip ahead and Configure and install a connector, using the Axual Self-Service UI. Use the configuration instructions below.
If you don’t have one available, the following section instructs you to deploy a publicly available instance of Couchbase.

Deploying a Couchbase instance

Sign up a new account with couchbase in order to deploy a trial cloud cluster.
Follow the instructions provided by Couchbase, and import the travel-sample dataset. While the dataset is getting imported, let’s set up the connectivity to the cluster:

  1. Click "Clusters" in the left-side menu

  2. Click on the name of your cluster

  3. Click the Connect tab

  4. Note down the Connection → Wide Area Network URL :cb cb.exampleuuid.cloud.couchbase.com

  5. Click Manage allowed IP

  6. Add permanent IP: 0.0.0.0/0

  7. Click BACK in the top-left to return to the previous page of the Connect tab.
    Open Database Access → Database Credentials → Manage Credentials

  8. Click Create Database Credentials

  9. Provide a username and password. Example: mytest / ABCdef123!

  10. Bucket level access: All Buckets / All Scopes / Read/Write

  11. Click Create

Configuring a new source Connector

  1. Follow the Creating streams documentation in order to create one stream and deploy it onto an environment.
    The name of the stream will be travel-sample.inventory.airport.
    The key/value types will be String/String.

    The stream name follows the following naming convention: <bucketName>.<scopeName>.<collectionName>.
    One such stream is needed for every collection we’ll be watching.

  2. Follow the Configure and install a connector documentation to set up a new Connector-Application.
    Let’s call it my_couchbase_source_app.
    The plugin name is com.couchbase.connect.kafka.CouchbaseSourceConnector.
    If a plugin isn’t available, ask a platform operator to install plugins.

    For advanced configuration, see the official source connector documentation, as well as the source configuration options and this source-connector example file.

  3. Provide the following minimal configuration in order to connect to the previously configured Couchbase instance.

    couchbase.bucket

    travel-sample

    couchbase.enable.tls

    true

    couchbase.trust.certificate.path

    /home/kafka/certificates/couchbase.pem

    Axual operators have whitelisted Couchbase’s self-signed certificate in advance and made it available to users at this fixed location within the Axual Connect-Cluster.

    couchbase.username

    mytest

    couchbase.password

    ABCdef123!

    couchbase.seed.nodes

    The Connection-Wide Area Network URL you noted down at the beginning.

    couchbase.source.handler

    com.couchbase.connect.kafka.handler.source.RawJsonSourceHandler

    couchbase.collections

    inventory.airport

    topic.creation.enable

    This property-key is not prefilled in the configuration by default, so you’ll have to add it as a custom property.

    false

  4. Authorize the my_couchbase_source_app source Connector-Application to produce to the travel-sample.inventory.airport stream.

  5. You can now start the source Connector-Application.

  6. You can now check the travel-sample.inventory.airport stream to see the events published by the Connector.

Configuring a new sink Connector

To keep things simple, we’ll consume the topic we just used for the source connector and publish everything into a new bucket in couchbase.
We’ll start by creating a new bucket.

  1. Go to you couchbase project.

    • Click Clusters and then the Buckets tab, and click the Create Bucket button.
      Let’s call it travel-copy. Use all the default settings.

    • Within the bucket, create a scope called inventory

    • Within the scope, create a collection called airport with a TTL of 2629800 (1 month)

  2. Follow the Configure and install a connector documentation to set up a new Connector-Application.
    Let’s call it my_couchbase_source_app.
    The plugin name is com.couchbase.connect.kafka.CouchbaseSourceConnector.
    If a plugin isn’t available, ask a platform operator to install plugins.

  3. Provide the following minimal configuration in order to connect to the previously configured Couchbase instance.
    For advanced configuration, see the official sink connector documentation, as well as the sink configuration options and this source-connector example file.

    topics

    travel-sample.inventory.airport

    couchbase.bucket

    travel-copy

    couchbase.topic.to.collection

    travel-sample.inventory.airport=inventory.airport

    couchbase.enable.tls

    true

    couchbase.trust.certificate.path

    /home/kafka/certificates/couchbase.pem

    Axual operators have whitelisted Couchbase’s self-signed certificate in advance, and made it available to users at this location within the Axual Connect-Cluster.

    couchbase.username

    mytest

    couchbase.password

    ABCdef123!

    couchbase.seed.nodes

    The Connection-Wide Area Network URL you noted down at the beginning.

  4. Authorize the my_couchbase_source_app sink Connector-Application to consume the travel-sample.inventory.airport stream.

  5. You can now start the sink Connector-Application

  6. You can now check the bucket in couchbase to see the events published by the Connector.

Cleanup

Once you are done, stop the Connector-Applications and cleanup the unused axual resources.
Don’t forget to return to the Cluster overview in Couchbase and delete your cluster if you’re not using it anymore.

License

Couchbase connectors are licensed under the Apache License, Version 2.0.

Source code

The source code for the Connect-Plugin can be found at github.com/couchbase/kafka-connect-couchbase.