Google Pub/Sub Lite source and sink connectors

Create a service account here (You might get prompted to select your Google project).
You can reuse the service account you created for Google Pub/Sub, if you have it.

Click the newly created service account. A new page will open.

Click the KEYS tab.

Click the ADD KEY button and select "Create a new key". Use "JSON" Key type and click CREATE.
A key will be created and automatically downloaded on your machine. We’ll refer to it later as "the downloaded key-file".
You can now close the "Service accounts" tab.

Enable Google Pub/Sub Lite for your project by visiting this page.

At the top of the page, click + CREATE LITE RESERVATION

At the bottom of the page, click + CREATE LITE TOPIC

The topic page will open now.
The first listed property is Lite topic name. This property value begins with: projects/123456789012/…​
Note down the number (e.g. 123456789012). We’ll call it "the project number". We’ll use it when configuring the connector-application, as the pubsublite.project configuration value.

While still on the lite topic page, click CREATE LITE SUBSCRIPTION:

We’ll now prepare to publish a message to the topic using Google Cloud shell.
At the top-right of your screen, next to your picture, are a few buttons.
One of them is "Activate Cloud Shell". Click it.
A shell will open at the bottom of your screen. Copy-Paste the following code-block:

pip3 install --upgrade google-cloud-pubsublite
wget -O ~/produce-to-lite-topic.py https://raw.githubusercontent.com/googleapis/python-pubsublite/v1.4.2/samples/snippets/publisher_example.py
export GOOGLE_APPLICATION_CREDENTIALS=~/keys.json
cat << } > ~/keys.json

Now Paste the contents of your downloaded key file.

Paste this command

echo '}' >> ~/keys.json

Produce to the topic using the following command:

python3 ~/produce-to-lite-topic.py    \
        $(gcloud projects list | grep "$(gcloud config get-value project)" | awk '{print $(NF)}') \
        'europe-west1'                \
        'b'                           \
        'my_pubsub_lite_google_topic' \
        ''

[Optional] Repeat the previous step and publish multiple messages.
These steps were similar to this documentation page from Google.

[Optional] Create a second lite topic called my_pubsub_lite_google_target. Go to your lite reservations and click on the my-pubsub-lite reservation, then follow the same steps you did before to finalize the creation of the topic.
You don’t need to produce any events to it. This topic will be used later, when configuring a sink connector.

Follow the 2022.3@axual::self-service/stream-management.html.adoc#creating-streams documentation in order to create one stream and deploy it onto an environment.
The name of the stream will be my_pubsub_lite_kafka_topic.
The key/value types will be String.

Follow the Configuring Connector-Applications documentation to set up a new connector application.
Let’s call it my_pubsub_lite_source. The plugin name is "com.google.pubsublite.kafka.source.PubSubLiteSourceConnector".
The values you will need to supply as configuration will be listed in this section.
Configure the security certificate as instructed.

Configuring the Connector-application deployment:

Authorize the my_pubsub_lite_source Connector-Application to produce to the my_pubsub_lite_kafka_topic topic.

You cannot start the Connector-Application.

Contact Axual to find a solution for authenticating against google with your service account key.

Start the Connector-Application

You can now check the stream to see the events published by the connector. The values will be base64 encoded.

Follow the Configuring Connector-Applications documentation to set up a new Connector application.
Let’s call it my_pubsub_lite_sink. The plugin name is "com.google.pubsublite.kafka.sink.PubSubLiteSinkConnector".
The values you will need to supply as configuration will be listed in this section.
Configure the security certificate as instructed.

Configuring the Connector-application deployment:

Authorize the my_pubsub_lite_sink Connector-Application to consume the my_pubsub_kafka_topic topic (or an alternative topic).

You cannot start the Connector-Application.

Contact Axual to find a solution for authenticating against google with your service account key.

Start the Connector-Application

You can now check the Google topic to see the events published by the connector.