Google Pub/Sub Lite Source Connector

Type

Source

Class

com.google.pubsublite.kafka.source.PubSubLiteSourceConnector

Target System

Google Cloud Pub/Sub Lite

Maintainer

Google

License

Apache License 2.0

Project

github.com/GoogleCloudPlatform/pubsub

Download

GitHub Releases

This page documents version 2.8.2. Newer versions should be compatible unless there are breaking changes, but field names or default values may differ. If you notice discrepancies, please contact Axual Support.
The GoogleCloudPlatform/pubsub repository used by this connector has been deprecated by Google since 2022. The successor project is googleapis/java-pubsub-group-kafka-connector. Migration involves updating the connector class names and download location. This documentation page has not yet been updated to reflect the new project.

Description

The Google Pub/Sub Lite Source Connector reads messages from a Google Cloud Pub/Sub Lite subscription and publishes them as records to a Kafka topic.

It is maintained by Google as part of the open-source github.com/GoogleCloudPlatform/pubsub.

Features

  • Consume messages from a Google Cloud Pub/Sub Lite subscription and publish to Kafka

  • Supports zonal Pub/Sub Lite topics

  • Supports GCP service account authentication via environment variable

When to Use

  • You need to bridge Google Cloud Pub/Sub Lite into Kafka for downstream processing.

When NOT to Use

The Pub/Sub Lite connector authenticates against Google at the Connect-cluster level, not at the connector level. This means there can only be one Pub/Sub Lite connector per cluster.
Authentication uses the environment variable approach:

customEnv:
  GOOGLE_APPLICATION_CREDENTIALS: '/tmp/keys.json'

Contact Axual to set this cluster-wide environment variable and distribute the key file to every Connect node.

Installation

The connector is available from the GitHub Releases.

  1. Navigate to the releases page and select the version matching your Kafka Connect installation.

  2. Download the JAR file.

For installation steps, see Installing Connector Plugins.

Configuration

For the complete configuration reference, see the official source connector documentation.

To configure a connector in Axual Self-Service, see Starting Connectors. TIP: For Infrastructure-as-Code deployment, see the Axual Kafka Connect Boilerplates for Terraform and Management API boilerplates.

Getting Started

Prerequisites

Google Cloud project and service account

We’ll deploy a Google Pub/Sub Lite instance using Google Cloud Services.
If you have a Google account, you can sign up for a free trial of Google cloud.

You can read more about Pub/Sub Lite here.

  1. Create a new service account. (You might get prompted to select your Google project first).
    This service account can be used for both Google Pub/Sub and Google Pub/Sub Lite.

    • At the top of the page, click +CREATE SERVICE ACCOUNT. A new page will open. Fill in the following:

      • Service account name: my-pubsub-admin

      • Service account ID: my-pubsub-admin

      • Service account description: "Temporary admin account to manage Google Pub/Sub"

    • Click CREATE AND CONTINUE

    • We’ll grant this ServiceAccount access to the project, and specifically two roles.

      • Click the dropdown called "Select a role" and, in the filter box, type Pub/Sub Lite Admin. Click "Pub/Sub Lite Admin".

      • Click +ADD ANOTHER ROLE. Repeat the previous step with Pub/Sub Admin as a filter.

    • We are not granting users access to this service account, so click Done instead.

  2. Click the newly created service account. A new page will open.

  3. Click the KEYS tab.

  4. Click the ADD KEY button and select "Create a new key". Use "JSON" Key type and click CREATE.
    A key will be created and automatically downloaded on your machine. We’ll refer to it later as "the downloaded key-file".
    You can now close the "Service accounts" tab.

Pub/Sub Lite reservation, topic, and subscription

  1. Enable Pub/Sub Lite for your project by visiting this page.

  2. Click + CREATE LITE RESERVATION, fill in the details, and create a reservation named my-pubsub-lite.

  3. On the reservation page, click + CREATE LITE TOPIC:

    • Location: Zonal Lite Topic, zone europe-west1-b

    • Name: my_pubsub_lite_google_topic

      Note the Lite topic name — it begins with projects/123456789012/…​. Note down the project number.

  4. Click CREATE LITE SUBSCRIPTION:

    • Lite subscription ID: my_pubsub_lite_google_topic-sub

    • Starting Offset: Beginning

  5. Produce at least one message to the topic (see the Google publishing docs).

Axual stream

The stream where the connector will produce events must already exist in Axual Self-Service. See Creating streams if you need to create it.

Steps

Step 1 — Create a connector application

  1. Follow the Creating streams documentation in order to create one stream and deploy it onto an environment.
    The name of the stream will be my_pubsub_lite_kafka_topic.
    The key/value types will be String/String.

  2. Follow the Configure and install a connector documentation to set up a new Connector-Application.
    Let’s call it my_pubsub_lite_source.
    The plugin name is com.google.pubsublite.kafka.source.PubSubLiteSourceConnector.
    If a plugin isn’t available, ask a platform operator to install plugins.

Step 2 — Configure the connector

kafka.topic

my_pubsub_lite_kafka_topic

pubsublite.location

europe-west1-b

pubsublite.project

Paste the project number noted after creating the Lite topic.

pubsublite.subscription

my_pubsub_lite_google_topic-sub

For advanced options, see the official source connector documentation.

  1. Authorize the my_pubsub_lite_source source Connector-Application to produce to the my_pubsub_lite_kafka_topic stream.

Step 3 — Start the connector

Contact Axual to set the cluster-wide GOOGLE_APPLICATION_CREDENTIALS environment variable and distribute the key file to every Connect node before starting the connector.

Once the cluster is configured, start the connector application from Axual Self-Service.

Step 4 — Verify

In Axual Self-Service, use stream-browse on my_pubsub_lite_kafka_topic to confirm events are arriving. Note: the values will be base64 encoded.

Cleanup

When you are done:

  1. Stop the connector application in Axual Self-Service.

  2. Remove stream access for the application if no longer needed.

  3. Return to your Pub/Sub Lite Topic List and delete your Google topics.

  4. Delete the my-pubsub-admin service account.

  5. Check your IAM principals and remove any principals created as part of this example.

Known limitations

  • Authentication is done at the cluster level — only one Pub/Sub Lite connector is supported per Connect cluster. Contact Axual for multi-connector scenarios.

  • Pub/Sub Lite message values are delivered to Kafka as base64-encoded strings.

Examples

Minimal configuration

{
  "name": "my-pubsub-lite-source",
  "config": {
    "connector.class": "com.google.pubsublite.kafka.source.PubSubLiteSourceConnector",
    "kafka.topic": "my_pubsub_lite_kafka_topic",
    "pubsublite.location": "europe-west1-b",
    "pubsublite.project": "<your-gcp-project-number>",
    "pubsublite.subscription": "my_pubsub_lite_google_topic-sub"
  }
}

License

Google Pub/Sub Lite source and sink connectors are licensed under the Apache License, Version 2.0.