Google Pub/Sub Lite Sink Connector

Type

Sink

Class

com.google.pubsublite.kafka.sink.PubSubLiteSinkConnector

Target System

Google Cloud Pub/Sub Lite

Maintainer

Google

License

Apache License 2.0

Project

github.com/googleapis/java-pubsub-group-kafka-connector

Download

Maven Central

This page documents version 1.3.3. Newer versions should be compatible unless there are breaking changes, but field names or default values may differ. If you notice discrepancies, please contact Axual Support.

Description

The Google Pub/Sub Lite Sink Connector consumes records from a Kafka topic and publishes them as messages to a Google Cloud Pub/Sub Lite topic.

It is maintained by Google as part of the open-source github.com/googleapis/java-pubsub-group-kafka-connector.

Features

  • Consume Kafka records and publish to a Google Cloud Pub/Sub Lite topic

  • Supports zonal Pub/Sub Lite topics

  • Supports GCP service account authentication via environment variable

When to Use

  • You need to bridge Kafka into Google Cloud Pub/Sub Lite for downstream GCP services.

When NOT to Use

The Pub/Sub Lite connector authenticates against Google at the Connect-cluster level, not at the connector level. This means there can only be one Pub/Sub Lite connector per cluster.
Authentication uses the environment variable approach:

customEnv:
  GOOGLE_APPLICATION_CREDENTIALS: '/tmp/keys.json'

Contact Axual to set this cluster-wide environment variable and distribute the key file to every Connect node.

Installation

The connector is available from the Maven Central.

  1. Navigate to the Maven Central page and download the JAR for version 1.3.3.

For installation steps, see Installing Connector Plugins.

Configuration

For the complete configuration reference, see the official documentation.

To configure a connector in Axual Self-Service, see Starting Connectors.

Getting Started

Prerequisites

Google Cloud project, service account, and Pub/Sub Lite resources

Ensure you have completed the setup described in the Pub/Sub Lite Source Connector prerequisites.

Create a second Lite topic named my_pubsub_lite_google_target in the same reservation.

Axual stream

The Kafka stream this connector will consume must already exist and contain records in Axual Self-Service.

Steps

Step 1 — Create a connector application

  1. Follow the Configure and install a connector documentation to set up a new Connector-Application.
    Let’s call it my_pubsub_lite_sink.
    The plugin name is com.google.pubsublite.kafka.sink.PubSubLiteSinkConnector.
    If a plugin isn’t available, ask a platform operator to install plugins.

Step 2 — Configure the connector

topics

my_pubsub_lite_kafka_topic

pubsublite.topic

my_pubsub_lite_google_target

pubsublite.location

europe-west1-b

pubsublite.project

Paste the project number noted after creating the Lite topic.

For advanced options, see the official documentation.

  1. Authorize the my_pubsub_lite_sink sink Connector-Application to consume the {stream-name} stream.

Step 3 — Start the connector

Contact Axual to set the cluster-wide GOOGLE_APPLICATION_CREDENTIALS environment variable and distribute the key file to every Connect node before starting the connector.

Once the cluster is configured, start the connector application from Axual Self-Service.

Step 4 — Verify

Verify that records have arrived in the target Pub/Sub Lite topic using the Google Cloud console or the Python publisher/subscriber samples.

Cleanup

When you are done:

  1. Stop the connector application in Axual Self-Service.

  2. Remove stream access for the application if no longer needed.

  3. Return to your Pub/Sub Lite Topic List and delete your Google topics.

  4. Delete the my-pubsub-admin service account.

  5. Check your IAM principals and remove any principals created as part of this example.

Known limitations

  • Authentication is done at the cluster level — only one Pub/Sub Lite connector is supported per Connect cluster. Contact Axual for multi-connector scenarios.

Examples

Minimal configuration

{
  "name": "my-pubsub-lite-sink",
  "config": {
    "connector.class": "com.google.pubsublite.kafka.sink.PubSubLiteSinkConnector",
    "topics": "my_pubsub_lite_kafka_topic",
    "pubsublite.topic": "my_pubsub_lite_google_target",
    "pubsublite.location": "europe-west1-b",
    "pubsublite.project": "<your-gcp-project-number>"
  }
}

License

Google Pub/Sub Lite source and sink connectors are licensed under the Apache License, Version 2.0.