Google Pub/Sub Lite Sink Connector

Type

Sink

Class

com.google.pubsublite.kafka.sink.PubSubLiteSinkConnector

Target System

Google Cloud Pub/Sub Lite

Maintainer

Google

License

Apache License 2.0

Project

github.com/GoogleCloudPlatform/pubsub

Download

GitHub Releases

This page documents version 2.8.2. Newer versions should be compatible unless there are breaking changes, but field names or default values may differ. If you notice discrepancies, please contact Axual Support.
The GoogleCloudPlatform/pubsub repository used by this connector has been deprecated by Google since 2022. The successor project is googleapis/java-pubsub-group-kafka-connector. Migration involves updating the connector class names and download location. This documentation page has not yet been updated to reflect the new project.

Description

The Google Pub/Sub Lite Sink Connector consumes records from a Kafka topic and publishes them as messages to a Google Cloud Pub/Sub Lite topic.

It is maintained by Google as part of the open-source github.com/GoogleCloudPlatform/pubsub.

Features

  • Consume Kafka records and publish to a Google Cloud Pub/Sub Lite topic

  • Supports zonal Pub/Sub Lite topics

  • Supports GCP service account authentication via environment variable

When to Use

  • You need to bridge Kafka into Google Cloud Pub/Sub Lite for downstream GCP services.

When NOT to Use

The Pub/Sub Lite connector authenticates against Google at the Connect-cluster level, not at the connector level. This means there can only be one Pub/Sub Lite connector per cluster.
Authentication uses the environment variable approach:

customEnv:
  GOOGLE_APPLICATION_CREDENTIALS: '/tmp/keys.json'

Contact Axual to set this cluster-wide environment variable and distribute the key file to every Connect node.

Installation

The connector is available from the GitHub Releases.

  1. Navigate to the releases page and select the version matching your Kafka Connect installation.

  2. Download the JAR file.

For installation steps, see Installing Connector Plugins.

Configuration

For the complete configuration reference, see the official sink connector documentation.

To configure a connector in Axual Self-Service, see Starting Connectors. TIP: For Infrastructure-as-Code deployment, see the Axual Kafka Connect Boilerplates for Terraform and Management API boilerplates.

Getting Started

Prerequisites

Google Cloud project, service account, and Pub/Sub Lite resources

Ensure you have completed the setup described in the Pub/Sub Lite Source Connector prerequisites.

Create a second Lite topic named my_pubsub_lite_google_target in the same reservation.

Axual stream

The Kafka stream this connector will consume must already exist and contain records in Axual Self-Service.

Steps

Step 1 — Create a connector application

  1. Follow the Configure and install a connector documentation to set up a new Connector-Application.
    Let’s call it my_pubsub_lite_sink.
    The plugin name is com.google.pubsublite.kafka.sink.PubSubLiteSinkConnector.
    If a plugin isn’t available, ask a platform operator to install plugins.

Step 2 — Configure the connector

pubsublite.topic

my_pubsub_lite_google_target

pubsublite.location

europe-west1-b

pubsublite.project

Paste the project number noted after creating the Lite topic.

For advanced options, see the official sink connector documentation.

  1. Authorize the my_pubsub_lite_sink sink Connector-Application to consume the {stream-name} stream.

Step 3 — Start the connector

Contact Axual to set the cluster-wide GOOGLE_APPLICATION_CREDENTIALS environment variable and distribute the key file to every Connect node before starting the connector.

Once the cluster is configured, start the connector application from Axual Self-Service.

Step 4 — Verify

Verify that records have arrived in the target Pub/Sub Lite topic using the Google Cloud console or the Python publisher/subscriber samples.

Cleanup

When you are done:

  1. Stop the connector application in Axual Self-Service.

  2. Remove stream access for the application if no longer needed.

  3. Return to your Pub/Sub Lite Topic List and delete your Google topics.

  4. Delete the my-pubsub-admin service account.

  5. Check your IAM principals and remove any principals created as part of this example.

Known limitations

  • Authentication is done at the cluster level — only one Pub/Sub Lite connector is supported per Connect cluster. Contact Axual for multi-connector scenarios.

Examples

Minimal configuration

{
  "name": "my-pubsub-lite-sink",
  "config": {
    "connector.class": "com.google.pubsublite.kafka.sink.PubSubLiteSinkConnector",
    "pubsublite.topic": "my_pubsub_lite_google_target",
    "pubsublite.location": "europe-west1-b",
    "pubsublite.project": "<your-gcp-project-number>"
  }
}

License

Google Pub/Sub Lite source and sink connectors are licensed under the Apache License, Version 2.0.