Google BigQuery Sink Connector

Type

sink

Class

io.lenses.streamreactor.connect.gcp.bigquery.sink.BigQuerySinkConnector

Target System

Google BigQuery

Maintainer

Lenses.io

License

Apache License 2.0

Project

github.com/lensesio/stream-reactor

Download

Lenses.io stream-reactor Releases

This page documents version 9.0.2. Newer versions should be compatible unless there are breaking changes, but field names or default values may differ. If you notice discrepancies, please contact Axual Support.

Description

The Google BigQuery Sink Connector consumes records from Kafka topics and writes them to Google BigQuery tables.

It is maintained by Lenses.io as part of the open-source github.com/lensesio/stream-reactor (stream-reactor) project.

The connector writes Kafka records to BigQuery tables, using a GCP service account for authentication. Target datasets and tables must exist before the connector starts.

Features

  • Write Kafka records to Google BigQuery tables

  • Authenticates via GCP service account JSON key

  • Configurable target dataset and table

When to Use

  • You need to load Kafka stream data into BigQuery for analytics or reporting.

  • You are building a data pipeline with BigQuery as the destination warehouse.

When NOT to Use

  • You need bidirectional sync — this connector is sink only.

  • Your records do not have a compatible schema for BigQuery column mapping.

Installation

The connector is available from the Lenses.io stream-reactor Releases.

  1. Navigate to the Lenses.io stream-reactor releases page.

  2. Download the archive for version 9.0.2 and extract the BigQuery connector JAR.

For installation steps, see Installing Connector Plugins.

Configuration

For the complete configuration reference, see the official sink connector documentation.

To configure a connector in Axual Self-Service, see Starting Connectors.

Getting Started

Prerequisites

Google Cloud project with BigQuery enabled

You need a GCP project with the BigQuery API enabled and a service account with the BigQuery Data Editor and BigQuery Job User roles.

  1. Create a GCP service account and download the JSON key file.

  2. Create the target BigQuery dataset and table.

Axual stream with records

The stream this connector consumes must already exist in Axual Self-Service and contain records. See Creating streams if you need to create a stream, and Produce some data to produce test events.

Steps

Step 1 — Create a connector application

  1. Follow the Configure and install a connector documentation to set up a new Connector-Application.
    Let’s call it my_bigquery_sink_app.
    The plugin name is io.lenses.streamreactor.connect.gcp.bigquery.sink.BigQuerySinkConnector.
    If a plugin isn’t available, ask a platform operator to install plugins.

Step 2 — Configure the connector

  1. Provide the following minimal configuration:

    connector.class

    io.lenses.streamreactor.connect.gcp.bigquery.sink.BigQuerySinkConnector

    topics

    Your Axual stream name

    gcp.credentials

    Contents of your GCP service account JSON key file

    project

    Your GCP project ID

    dataset

    Your BigQuery dataset name

    value.converter

    org.apache.kafka.connect.json.JsonConverter

    value.converter.schemas.enable

    false

    For advanced options, see the official sink connector documentation.

  2. Authorize the my_bigquery_sink_app sink Connector-Application to consume the my_bigquery_topic stream.

Step 3 — Start the connector

Start the connector application from Axual Self-Service.

Step 4 — Verify

In the Google Cloud Console, open your BigQuery dataset and confirm rows are appearing in the target table.

Cleanup

When you are done:

  1. Stop the connector application in Axual Self-Service.

  2. Remove stream access for the application if no longer needed.

  3. Delete the BigQuery dataset if it was created only for testing.

Known limitations

  • The target BigQuery dataset must exist before the connector starts — the connector does not create datasets automatically.

  • GCP service account credentials are passed as a JSON string in the connector configuration — store them securely using Axual’s keyvault reference syntax.

Examples

Minimal configuration

{
  "name": "my-bigquery-sink",
  "config": {
    "connector.class": "io.lenses.streamreactor.connect.gcp.bigquery.sink.BigQuerySinkConnector",
    "topics": "<your-axual-stream-name>",
    "gcp.credentials": "<contents-of-service-account-json>",
    "project": "my-gcp-project",
    "dataset": "my_dataset",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false"
  }
}

License

Google BigQuery sink connector is licensed under the Apache License, Version 2.0.