OpenSearch sink Connector, version 2.0.1

OpenSearch

OpenSearch is an open-source search and analytics suite that makes it easy to ingest, search, visualize, and analyze data.
Common OpenSearch use cases are application search, log analytics, data observability, data ingestion, and more.

Note: Although the Connect-Plugin is open-sourced, this is not the "official" Connect-Plugin from Aiven.
The "Official" Connect-Plugin is not open sourced nor free.

The Connector currently contains a bug which prevents configuration-discovery from happening. The self-service UI relies on this functionality to offer suggestions for the properties you need to supply to the Connector.
A pull request was created to fix this bug with Aiven, but until it gets merged, you are invited to use the jar file from the git repository you will shortly be asked to clone.

Deploying a OpenSearch instance

This documentation page is a work in progress and as such, we don’t have a solution for obtaining a publicly available service exposing the OpenSearch API, for you to test your Connector against.

For developers who have the means to deploy the axual platform locally, we’ll outline the steps to deploy OpenSearch locally.

Follow the operator documentation to deploy the axual platform locally, as well as enabling connect.
Clone the gitlab.com/axual/bitnami-kafka-connect-setup git repository → branch opensearch-axualplatform .
Deploy a OpenSearch single-node cluster and dashboard by running docker-compose up.
Confirm the deployment was successful by opening the OpenSearch dashboard.
Login using admin : admin as credentials.
You’re done for now. You’ll get back to the dashboard after starting the Connector.

Configuring a new sink Connector

Follow the Creating streams documentation in order to create one stream and deploy it onto an environment.
The name of the stream will be my_opensearch_stream.
The key/value types will be JSON/JSON.
Produce some JSON/JSON events to this stream. You can follow the producer documentation and examples.
Example JSON message:
```
{
    "name": "john",
    "age": 28
}
```
Follow the Configure and install a connector documentation to set up a new Connector-Application.
Let’s call it my_opensearch_sink.
The plugin name is io.aiven.kafka.connect.opensearch.OpensearchSinkConnector.

For advanced configuration, see the official sink connector documentation.

Provide the following minimal configuration in order to connect to the previously configured OpenSearch instance.

connection.url

https://platform.local:9200

connection.username

admin

connection.password

admin

topics

my_opensearch_stream

key.ignore

true

schema.ignore

true

errors.log.enable

true

errors.log.include.messages

true

errors.tolerance

all

behavior.on.malformed.documents

warn

Authorize the my_opensearch_sink sink Connector-Application to consume the my_opensearch_stream stream.
You can now start the sink Connector-Application.
Log back into the OpenSearch dashboard.
Go to the index-pattern creation page
- Click the menu button (three stripes) in the top-left corner of the screen → Management → Stack management
- Click Index Patterns (although it may look like it’s already selected)
Click + Create index pattern
- Index pattern name: my_opensearch_stream
- Click Next step
Click Create index pattern
Go to the Discover page.
- Click the menu button (three stripes) in the top-left corner of the screen → OpenSearch Dashboards → Discover
You should now see your records in OpenSearch

Cleanup

Once you are done, stop the Connector-Application and cleanup the unused axual resources.
Run docker-compose down in the same directory you ran the up command in order to stop and remove the OpenSearch containers.

License

OpenSearch sink-Connector is licensed under the Apache License, Version 2.0.

Source code

The source code for the Connect-Plugin can be found at github.com/aiven/opensearch-connector-for-apache-kafka