OpenSearch sink Connector, version 2.0.3

OpenSearch

OpenSearch is an open-source search and analytics suite that makes it easy to ingest, search, visualize, and analyze data.
Common OpenSearch use cases are application search, log analytics, data observability, data ingestion, and more.

Note: Although the Connect-Plugin is open-sourced, this is not the "official" Connect-Plugin from Aiven.
The "Official" Connect-Plugin is not open sourced nor free.

The Connector currently contains a bug which prevents configuration-discovery from happening. The self-service UI relies on this functionality to offer suggestions for the properties you need to supply to the Connector.
A pull request was created to fix this bug with Aiven, but until it gets merged, you are invited to use the jar file from the git repository you will shortly be asked to clone.

Deploying a OpenSearch instance

This documentation page is a work in progress and as such, we don’t have a solution for obtaining a publicly available service exposing the OpenSearch API, for you to test your Connector against.

For developers who have the means to run the axual platform locally, we’ll outline the steps to deploy OpenSearch locally.

  1. Follow the operator documentation to deploy the axual platform locally, as well as enabling connect.

  2. Clone the gitlab.com/axual/bitnami-kafka-connect-setup git repository → branch opensearch-axualplatform .

  3. Deploy a OpenSearch single-node cluster and dashboard by running docker-compose up.

  4. Confirm the deployment was successful by opening the OpenSearch dashboard.
    Login using admin : admin as credentials.
    You’re done for now. You’ll get back to the dashboard after starting the Connector.

Configuring a new sink Connector

  1. Follow the Creating streams documentation in order to create one stream and deploy it onto an environment.
    The name of the stream will be my_opensearch_stream.
    The key/value types will be JSON/JSON.

  2. Produce some data as JSON/JSON events to this stream.

  3. Follow the Configure and install a connector documentation to set up a new Connector-Application.
    Let’s call it my_opensearch_sink.
    The plugin name is io.aiven.kafka.connect.opensearch.OpensearchSinkConnector.
    If a plugin isn’t available, ask a platform operator to install plugins.

  4. Provide the following minimal configuration in order to connect to the previously configured OpenSearch instance.
    For advanced configuration, see the official sink connector documentation.

    connection.url

    https://platform.local:9200

    connection.username

    admin

    connection.password

    admin

    topics

    my_opensearch_stream

    key.ignore

    true

    schema.ignore

    true

    errors.log.enable

    true

    errors.log.include.messages

    true

    errors.tolerance

    all

    behavior.on.malformed.documents

    warn

  5. Authorize the my_opensearch_sink sink Connector-Application to consume the my_opensearch_stream stream.

  6. You can now start the sink Connector-Application.

  7. Log back into the OpenSearch dashboard.

  8. Go to the index-pattern creation page

    • Click the menu button (three stripes) in the top-left corner of the screen → Management → Stack management

    • Click Index Patterns (although it may look like it’s already selected)

  9. Click + Create index pattern

    • Index pattern name: my_opensearch_stream

    • Click Next step

  10. Click Create index pattern

  11. Go to the Discover page.

    • Click the menu button (three stripes) in the top-left corner of the screen → OpenSearch Dashboards → Discover

  12. You should now see your records in OpenSearch

Cleanup

Once you are done, stop the Connector-Application and cleanup the unused axual resources.
Run docker-compose down in the same directory you ran the up command in order to stop and remove the OpenSearch containers.

License

OpenSearch sink-Connector is licensed under the Apache License, Version 2.0.

Source code

The source code for the Connect-Plugin can be found at github.com/aiven/opensearch-connector-for-apache-kafka.