Google Pub/Sub Lite source and sink Connector, version 2.8.2

Google Pub/Sub Lite

If you already have a publicly accessible deployment of Google Pub/Sub Lite, you can skip ahead and Configure and install a connector, using the Axual Self-Service UI. Use the configuration instructions below.
If you don’t have one available, the following section instructs you to deploy a publicly available instance of Google Pub/Sub Lite.

Deploying a Google Pub/Sub Lite instance

We’ll deploy a Google Pub/Sub Lite instance using Google Cloud Services.
If you have a Google account, you can sign up for a free trial of Google cloud.

You can read more about Google Pub/Sub Lite on Google cloud here.

The Google Pub/Sub Lite Connector authenticates against Google.
This authentication is not done at the Connect-Plugin level, but at the Connect-Cluster level. This means there can only be one PubSub Lite Connector per cluster.
For the first Connector of this kind per cluster, we’ll authenticate using the environment variable approach:

customEnv:
  GOOGLE_APPLICATION_CREDENTIALS: '/tmp/keys.json'

If more such Connectors should be started within the same Connect-Cluster, Axual developers will have to look into alternative authentication options and provide you with advice. This is out scope for this example.

Let’s get started.

  1. Create a new service account. (You might get prompted to select your Google project first).
    This service account can be used for both Google Pub/Sub and Google Pub/Sub Lite.

    • At the top of the page, click +CREATE SERVICE ACCOUNT. A new page will open. Fill in the following:

      • Service account name: my-pubsub-admin

      • Service account ID: my-pubsub-admin

      • Service account description: "Temporary admin account to manage Google Pub/Sub"

    • Click CREATE AND CONTINUE

    • We’ll grant this ServiceAccount access to the project, and specifically two roles.

      • Click the dropdown called "Select a role" and, in the filter box, type Pub/Sub Lite Admin. Click "Pub/Sub Lite Admin".

      • Click +ADD ANOTHER ROLE. Repeat the previous step with Pub/Sub Admin as a filter.

    • We are not granting users access to this service account, so click Done instead.

  2. Click the newly created service account. A new page will open.

  3. Click the KEYS tab.

  4. Click the ADD KEY button and select "Create a new key". Use "JSON" Key type and click CREATE.
    A key will be created and automatically downloaded on your machine. We’ll refer to it later as "the downloaded key-file".
    You can now close the "Service accounts" tab.

Create a PubSub Lite topic

  1. Enable Google Pub/Sub Lite for your project by visiting this page.

  2. At the top of the page, click + CREATE LITE RESERVATION

    • Location: Choose europe-west1 (Belgium)

    • Click CONTINUE

    • Name: my-pubsub-lite

    • Click CONTINUE

    • Throughput: 1

    • Click CONTINUE

    • Review: Just click CREATE
      Your new lite reservation page will show up.

  3. At the bottom of the page, click + CREATE LITE TOPIC

    • Location: Select Zonal Lite Topic and select europe-west1-b from the dropdown

    • Click CONTINUE

    • Name: my_pubsub_lite_google_topic

    • Click CONTINUE

    • Throughput:

      • Peak Publish Throughput (MiB/s): 4 (default)

      • Peak Subscribe Throughput (MiB/s): 4 (default)

    • Click CONTINUE

    • Message storage:

      • Storage per partition (GiB): 30 (default, minimum)

      • Retention period: Drop messages after specified duration even if storage is available

        • Days: 1

    • Click CONTINUE

    • Review: Just click CREATE

  4. The topic page will open now.
    The first listed property is Lite topic name. This property value begins with: projects/123456789012/…​
    Note down the number (e.g. 123456789012). We’ll call it "the project number". We’ll use it when configuring the Connector-Application, for the pubsublite.project configuration value.

  5. While still on the lite topic page, click CREATE LITE SUBSCRIPTION:

    • Lite subscription ID: my_pubsub_lite_google_topic-sub

    • Delivery requirement (required): Deliver messages immediately (default)

    • Starting Offset (required): Beginning

    • Click CREATE

Publish messages to the PubSub topic

  1. We’ll now prepare to publish a message to the topic using Google Cloud shell.
    At the top-right of your screen, next to your picture, are a few buttons.
    One of them is "Activate Cloud Shell". Click it.
    A shell will open at the bottom of your screen. Copy-Paste the following code-block:

    pip3 install --upgrade google-cloud-pubsublite
wget -O ~/produce-to-lite-topic.py https://raw.githubusercontent.com/googleapis/python-pubsublite/v1.4.2/samples/snippets/publisher_example.py
export GOOGLE_APPLICATION_CREDENTIALS=~/keys.json
cat << } > ~/keys.json

  2. Now Paste the contents of your downloaded key file.

  3. Then paste this command

    echo '}' >> ~/keys.json

  4. Produce to the topic using the following command:

    python3 ~/produce-to-lite-topic.py    \
        $(gcloud projects list | grep "$(gcloud config get-value project)" | awk '{print $(NF)}') \
        'europe-west1'                \
        'b'                           \
        'my_pubsub_lite_google_topic' \
        ''

  5. [Optional] Repeat the previous step and publish multiple messages.
    For additional documentation about publishing messages to PubSub Lite topics see this page from Google.

  6. [Optional] Create a second lite topic called my_pubsub_lite_google_target:
    Go to your lite reservations and click on the my-pubsub-lite reservation, then create a second topic the same way you did before.

    You don’t need to produce any events to it. This topic will be used later, when configuring a sink Connector.

Configuring a new source Connector

  1. Follow the Creating streams documentation in order to create one stream and deploy it onto an environment.
    The name of the stream will be my_pubsub_lite_kafka_topic.
    The key/value types will be String/String.

  2. Follow the Configure and install a connector documentation to set up a new Connector-Application.
    Let’s call it my_pubsub_lite_source.
    The plugin name is com.google.pubsublite.kafka.source.PubSubLiteSourceConnector.
    If a plugin isn’t available, ask a platform operator to install plugins.

  3. Configuring the Connector-Application deployment:
    Open the previously downloaded key-file. We’ll use values from this file as part of our configuration.
    For advanced configuration, see the official source connector documentation.

    kafka.topic

    my_pubsub_lite_kafka_topic

    pubsublite.location

    europe-west1-b

    pubsublite.project

    Paste the project number you noted down after creating the my_pubsub_lite_google_topic topic.

    pubsublite.subscription

    my_pubsub_lite_google_topic-sub

  4. Authorize the my_pubsub_lite_source source Connector-Application to produce to the my_pubsub_lite_kafka_topic stream.

  5. You cannot start the source Connector-Application. Contact Axual to set the Cluster-wide GOOGLE_APPLICATION_CREDENTIALS environment variable for you and include the key json file in every connect node.

  6. Start the source Connector-Application.

  7. You can now check the stream to see the events published by the connector. The values will be base64 encoded.

Configuring a new sink Connector

  1. Follow the Configure and install a connector documentation to set up a new Connector-Application.
    Let’s call it my_pubsub_lite_sink.
    The plugin name is com.google.pubsublite.kafka.sink.PubSubLiteSinkConnector.
    If a plugin isn’t available, ask a platform operator to install plugins.

  2. Configuring the Connector-Application deployment:

    • Open the previously downloaded key-file. We’ll use values from this file as part of our configuration.

    • Provide the following minimal configuration in order to connect to the previously configured Google Pub/Sub Lite instance.
      For advanced configuration, see the official sink connector documentation.

      pubsublite.topic

      my_pubsub_lite_google_target

      pubsublite.location

      europe-west1-b

      pubsublite.project

      Paste the project number you noted down after creating the my_pubsub_lite_google_target topic.

  3. Authorize the my_pubsub_lite_sink sink Connector-Application to consume the my_pubsub_lite_kafka_topic stream.

  4. You cannot start the sink Connector-Application yet.
    Contact Axual to set the Cluster-wide GOOGLE_APPLICATION_CREDENTIALS environment variable for you and include the key json file in every connect node.

  5. Start the sink Connector-Application.

  6. You can now check the Google topic to see the events published by the Connector.

Cleanup

Once you are done, stop the Connector-Applications and cleanup the unused axual resources.

Don’t forget to return to your Pub/Sub Lite Topic List and delete your Google Topics, as well as the my-pubsub-admin service account. Do also check your IAM principals: see if you can remove any principal which was created as a byproduct of this example.

License

Google Pub/Sub Lite source and sink Connectors are licensed under the Apache License, Version 2.0.

Source code

The source code for the Connect-Plugin can be found at github.com/GoogleCloudPlatform/pubsub.