Debezium - MongoDB source-connector

Debezium - MongoDB

If you already have a publicly accessible deployment of a MongoDB, then you can already configure and install a connector using the Axual Self-Service UI. Use the configuration instructions below.

If you don’t have one available, follow the next section to deploy a publicly available instance of MongoDB server.

Deploying a MongoDB instance

Deploy a free shared MongoDB cloud cluster.

During the "security quickstart" step:

You will create a set of credentials used to authenticate against the cluster . Remember these credentials.
You will authorize IP addresses to access the cluster. It’s simplest if you authorize "0.0.0.0/0"

Go to "Database". You will see your cluster. There’s 4 buttons after the cluster name, click the … (three dots) and then click "Load Sample dataset".

Follow MongoDB instructions and import the dataset.

Once that is complete, go to Database → Click on the name of your deployment (e.g. "Cluster0"), and click on the primary node.

Save the name, i.e. ab-cdef1gh-shard-00-01.i2j3klm.mongodb.net. You will use this later as the mongodb.hosts connector-configuration value.

[Optional] If you haven’t done it during the quickstart, you can Go to Network Access and authorize the '0.0.0.0/0' CIDR range.

Configuring a new connector

Follow the 2022.1@axual::self-service/stream-management.html.adoc#creating-streams documentation in order to create one stream.

The name of the stream will be <mongo_nickname>.<db_name>.<collection_name>, where each field corresponds to the table we’ll be watching. In our example, that name turns out to be mongo_nickname.sample_mflix.comments.

When you deploy the stream onto an environment, you will use the following configuration:

retention time and segment.ms of at least 86400000 (1 day).
a single (one) partition.

Follow the Configuring Connector-Applications documentation to set up a new Connect-Application. Let’s call it my_comments_app. The plugin name is "io.debezium.connector.mongodb.MongoDbConnector". Configure the security certificate as instructed. The values you will need to supply as configuration will be listed in this section.

For advanced configuration, see the official connector documentation.

`mongodb.hosts` _The hostname of the primary mongodb node, which you saved a few minutes ago_

`mongodb.hosts`	_The hostname of the primary mongodb node, which you saved a few minutes ago_
Example value: `ac-abcd1ef-shard-00-02.abcdefg.mongodb.net`	`mongodb.ssl.enabled`
`true`	`mongodb.user` The user you created, NOT your account name
Example value: `username`	`mongodb.password` Password for the created user
Example value: `password1234`	`mongodb.name`
`mongo_nickname`	`database.include.list`
`sample_mflix`	`collection.include.list`

Example value:

ac-abcd1ef-shard-00-02.abcdefg.mongodb.net

mongodb.ssl.enabled

true

mongodb.user

The user you created, NOT your account name

Example value:

username

mongodb.password

Password for the created user

Example value:

password1234

mongodb.name

mongo_nickname

database.include.list

sample_mflix

collection.include.list

We need to supply additional (custom) properties, with keys which are not automatically available in the self-service UI. We disable automatic topic creation. The first property should be enough. The fact that the following 2 properties are required is a bug within the connect-plugin itself.

topic.creation.enable

false

topic.creation.default.partitions

1

topic.creation.default.replication.factor

3

Authorize the Connector-Application to produce to the fulfillment.inventory.customers topic.

You can now start the Connector-Application

Cleanup

Once you are done, stop the connector application and cleanup the unused axual resources.

In case you deployed your mongo-cluster via cloud.mongodb.com, don’t forget to terminate your cluster: click the cluster, click the … three dots on the right side of the screen, then click terminate and follow the instructions.