Google Cloud Storage Sink Connector

Type

sink

Class

io.lenses.streamreactor.connect.gcp.storage.sink.GCPStorageSinkConnector

Target System

Google Cloud Storage

Maintainer

Lenses.io

License

Apache License 2.0

Project

github.com/lensesio/stream-reactor

Download

Lenses.io stream-reactor Releases

This page documents version 9.0.2. Newer versions should be compatible unless there are breaking changes, but field names or default values may differ. If you notice discrepancies, please contact Axual Support.

Description

The Google Cloud Storage (GCS) Sink Connector consumes records from Kafka topics and writes them as objects to a GCS bucket.

It is maintained by Lenses.io as part of the open-source github.com/lensesio/stream-reactor (stream-reactor) project.

The connector batches Kafka records and writes them to GCS objects, using a GCP service account for authentication. Objects are organized by topic and partition.

Features

  • Write Kafka records to GCS objects

  • Configurable object prefix and path structure

  • Authenticates via GCP service account JSON key

  • Supports JSON and Avro output formats

When to Use

  • You need to archive Kafka stream data in GCS for long-term storage or batch processing.

  • You are building a data lake pipeline with GCS as the storage layer.

When NOT to Use

  • You need bidirectional sync — this connector is sink only.

  • You need real-time query access to the data — consider BigQuery sink instead.

Installation

The connector is available from the Lenses.io stream-reactor Releases.

  1. Navigate to the Lenses.io stream-reactor releases page.

  2. Download the archive for version 9.0.2 and extract the GCS connector JAR.

For installation steps, see Installing Connector Plugins.

Configuration

For the complete configuration reference, see the official sink connector documentation.

To configure a connector in Axual Self-Service, see Starting Connectors.

Getting Started

Prerequisites

GCS bucket

You need a GCS bucket in your GCP project and a service account with the Storage Object Creator role on that bucket.

  1. create a bucket.

  2. Create a GCP service account and download the JSON key file.

  3. Grant the service account Storage Object Creator on the bucket.

Axual stream with records

The stream this connector consumes must already exist in Axual Self-Service and contain records. See Creating streams if you need to create a stream, and Produce some data to produce test events.

Steps

Step 1 — Create a connector application

  1. Follow the Configure and install a connector documentation to set up a new Connector-Application.
    Let’s call it my_gcs_sink_app.
    The plugin name is io.lenses.streamreactor.connect.gcp.storage.sink.GCPStorageSinkConnector.
    If a plugin isn’t available, ask a platform operator to install plugins.

Step 2 — Configure the connector

  1. Provide the following minimal configuration:

    connector.class

    io.lenses.streamreactor.connect.gcp.storage.sink.GCPStorageSinkConnector

    topics

    Your Axual stream name

    gcp.credentials

    Contents of your GCP service account JSON key file

    gcs.bucket

    Your GCS bucket name

    gcs.prefix

    kafka/

    value.converter

    org.apache.kafka.connect.json.JsonConverter

    value.converter.schemas.enable

    false

    For advanced options, see the official sink connector documentation.

  2. Authorize the my_gcs_sink_app sink Connector-Application to consume the my_gcs_topic stream.

Step 3 — Start the connector

Start the connector application from Axual Self-Service.

Step 4 — Verify

In the Google Cloud Console, navigate to your GCS bucket and confirm objects are appearing under the configured prefix.

Cleanup

When you are done:

  1. Stop the connector application in Axual Self-Service.

  2. Remove stream access for the application if no longer needed.

  3. Delete your Cloud Storage bucket if it was created only for testing.

Known limitations

  • GCP service account credentials are passed as a JSON string in the connector configuration — store them securely using Axual’s keyvault reference syntax.

  • Objects are written in batches — records may not appear in GCS immediately after being consumed from Kafka.

Examples

Minimal configuration

{
  "name": "my-gcs-sink",
  "config": {
    "connector.class": "io.lenses.streamreactor.connect.gcp.storage.sink.GCPStorageSinkConnector",
    "topics": "<your-axual-stream-name>",
    "gcp.credentials": "<contents-of-service-account-json>",
    "gcs.bucket": "my-gcs-bucket",
    "gcs.prefix": "kafka/",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false"
  }
}

License

Google Cloud Storage sink connector is licensed under the Apache License, Version 2.0.