Google Pub/Sub Sink Connector

Type

Sink

Class

com.google.pubsub.kafka.sink.CloudPubSubSinkConnector

Target System

Google Cloud Pub/Sub

Maintainer

Google

License

Apache License 2.0

Project

github.com/GoogleCloudPlatform/pubsub

Download

GitHub Releases

This page documents version 2.8.2. Newer versions should be compatible unless there are breaking changes, but field names or default values may differ. If you notice discrepancies, please contact Axual Support.
The GoogleCloudPlatform/pubsub repository used by this connector has been deprecated by Google since 2022. The successor project is googleapis/java-pubsub-group-kafka-connector. Migration involves updating the connector class names and download location. This documentation page has not yet been updated to reflect the new project.

Description

The Google Pub/Sub Sink Connector consumes records from a Kafka topic and publishes them as messages to a Google Cloud Pub/Sub topic.

It is maintained by Google as part of the open-source github.com/GoogleCloudPlatform/pubsub.

Features

  • Consume Kafka records and publish to a Google Cloud Pub/Sub topic

  • Supports GCP service account authentication via JSON key file

When to Use

  • You need to bridge Kafka into Google Cloud Pub/Sub for downstream GCP services.

  • You want to forward Kafka topic data into Pub/Sub for fan-out delivery.

When NOT to Use

Installation

The connector is available from the GitHub Releases.

  1. Navigate to the releases page and select the version matching your Kafka Connect installation.

  2. Download the JAR file.

For installation steps, see Installing Connector Plugins.

Configuration

For the complete configuration reference, see the official sink connector documentation.

To configure a connector in Axual Self-Service, see Starting Connectors. TIP: For Infrastructure-as-Code deployment, see the Axual Kafka Connect Boilerplates for Terraform and Management API boilerplates.

Getting Started

Prerequisites

Google Cloud project, service account, and Pub/Sub topic

Ensure you have completed the setup described in the Pub/Sub Source Connector prerequisites, including creating a GCP service account and a target Pub/Sub topic.

You may optionally create a second Pub/Sub topic (e.g. my_pubsub_google_target) for the sink to write to.

Axual stream

The Kafka stream this connector will consume must already exist and contain records in Axual Self-Service.

Steps

Step 1 — Create a connector application

  1. Follow the Configure and install a connector documentation to set up a new Connector-Application.
    Let’s call it my_pubsub_sink.
    The plugin name is com.google.pubsub.kafka.sink.CloudPubSubSinkConnector.
    If a plugin isn’t available, ask a platform operator to install plugins.

Step 2 — Configure the connector

  1. Open the previously downloaded GCP service account key file. Provide the following configuration:

    cps.project

    Copy the project_id value from the downloaded key file

    Example value:
    decisive-lambda-369420

    gcp.credentials.json

    Copy the entire JSON as-is from the downloaded key file. Newlines will be automatically excluded.

    cps.topic

    my_pubsub_google_target

    topics

    For advanced options, see the official sink connector documentation.

  2. Authorize the my_pubsub_sink sink Connector-Application to consume the {stream-name} stream.

Step 3 — Start the connector

Start the connector application from Axual Self-Service.

Step 4 — Verify

Verify that records have arrived in the target Pub/Sub topic:

  1. Open the topics page.

  2. Click the my_pubsub_google_target topic.

  3. Click the MESSAGES tab.

  4. Select the available subscription from the dropdown.

  5. Click PULL.

  6. The values will be displayed as base64-encoded strings.

Cleanup

When you are done:

  1. Stop the connector application in Axual Self-Service.

  2. Remove stream access for the application if no longer needed.

  3. Return to your Pub/Sub topics list and delete your Google topics.

  4. Delete the my-pubsub-admin service account.

  5. Check your IAM principals and remove any principals created as part of this example.

Known limitations

  • Pub/Sub message values arrive as base64-encoded strings.

Examples

Minimal configuration

{
  "name": "my-pubsub-sink",
  "config": {
    "connector.class": "com.google.pubsub.kafka.sink.CloudPubSubSinkConnector",
    "cps.project": "decisive-lambda-369420",
    "gcp.credentials.json": "<contents-of-gcp-service-account-key-json>",
    "cps.topic": "my_pubsub_google_target",
    "topics": "my_pubsub_kafka_topic"
  }
}

License

Google Pub/Sub source and sink connectors are licensed under the Apache License, Version 2.0.