How to Produce Sample Data for a Sink Connector

This guide shows you how to put sample records on a topic through the Rest Proxy, so a sink connector has something to consume when you test it.

Type

How-to guide

Goal

Produce a record onto a topic, from a terminal or from Postman, so a sink connector can be tried end to end.

Audience

An application owner testing a sink Connector Application.

When to use

Before starting a sink connector for the first time, when the source topic is still empty.

A sink connector reads from a topic and writes to the integrated system, so it does nothing at all while the topic is empty. Producing a couple of records first is the quickest way to prove the connector works.

Contents

The sections below cover each task in this guide:

Prerequisites

Confirm the following before you begin.

Access and permissions required

You need the following:

Resources that must exist before starting

The following must already be in place:

  • The topic the sink connector consumes from, deployed onto the environment you are testing in.

  • The topic name and environment short name.

  • Your Rest Proxy URL, which depends on how the platform was installed:

Installation Rest Proxy URL

Axual Cloud

Of the form https://rest-ams01.cloud.axual.io:<port>;. The port differs per tenant, so take it from your internal documentation.

SaaS trial

Stated in the README.txt included in your care package.

Axual Platform on your own infrastructure

Ask the Axual Platform operators for the address.

Local Helm deployment

https://platform.<domain>:18111
Replace every <VALUE> placeholder with your own value before running a command.

Produce from a terminal

Use curl when you only need to send a record or two.

The example assumes the producer’s certificate and key sit in the working directory as auth.cert and auth.key. Update the first three lines with your own values, then run it.

REST_PROXY_URL="https://rest-ams01.cloud.axual.io:99999"
AXUAL_ENVIRONMENT="example"
AXUAL_STREAM_NAME="my_test_stream"

curl -k --request POST \
  --url "${REST_PROXY_URL}/stream/${AXUAL_ENVIRONMENT}/${AXUAL_STREAM_NAME}" \
  --header "axual-application-id: irrelevant-for-producers" \
  --header "axual-producer-uuid: ${AXUAL_STREAM_NAME}-rest-producer" \
  --header "Content-Type: application/json" \
  --key ./auth.key \
  --cert ./auth.cert \
  --data '
  {
     "keyMessage":{
        "type":"STRING",
        "message":"{\"name\": \"john\"}"
     },
     "valueMessage":{
        "type":"STRING",
        "message":"{\"name\": \"john\", \"age\": \"28\"}"
     }
  }'
The Rest Proxy produce path is /stream/<environment>/<topic>. The stream segment is part of the API and stays as it is, even though the resource it addresses is called a topic in Self-Service.

A successful produce returns HTTP 200 and a ProduceResponse naming the cluster, topic, partition and offset the record landed on.

Produce from Postman

Use Postman when you want to keep the request around and send it repeatedly.

  1. Download Postman if you do not already have it installed.

  2. Acquire the certificate and key file you used when registering your producer. Configure Postman to send them as a client certificate for mutual TLS, following Manage Postman Certificates.

  3. Import the following as a Postman collection:

    {
	"info": {
		"_postman_id": "b04bd56b-49cf-4c23-8198-2f22fc8d5d72",
		"name": "Axual Platform produce example",
		"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
	},
	"item": [{
			"name": "Produce event",
			"protocolProfileBehavior": {
				"disabledSystemHeaders": {
					"user-agent": true,
					"accept": true,
					"accept-encoding": true,
					"connection": true
				},
				"strictSSL": false
			},
			"request": {
				"method": "POST",
				"header": [
					{
						"key": "axual-application-id",
						"value": "irrelevant-for-producers",
						"type": "default"
					},
					{
						"key": "axual-producer-uuid",
						"value": "{{AXUAL_STREAM_NAME}}-rest-producer",
						"type": "default"
					}
				],
				"body": {
					"mode": "raw",
					"raw": "{\n  \"keyMessage\":{\n    \"type\":\"STRING\",\n    \"message\":\"{\\\"name\\\": \\\"john\\\"}\"\n  },\n  \"valueMessage\":{\n    \"type\":\"STRING\",\n    \"message\":\"{\\\"name\\\": \\\"john\\\", \\\"age\\\": \\\"28\\\"}\"\n  }\n}\n",
					"options": { "raw": { "language": "json" }}
				},
				"url": {
					"raw": "{{REST_PROXY_URL}}/stream/{{AXUAL_ENVIRONMENT}}/{{AXUAL_STREAM_NAME}}",
					"host": [
						"{{REST_PROXY_URL}}"
					],
					"path": [
						"stream",
						"{{AXUAL_ENVIRONMENT}}",
						"{{AXUAL_STREAM_NAME}}"
					]
				}
			},
			"response": []
	}],
	"event": [],
	"variable": [
		{
			"key": "REST_PROXY_URL",
			"value": "https://rest-ams01.cloud.axual.io:99999",
			"type": "default"
		},
		{
			"key": "AXUAL_ENVIRONMENT",
			"value": "example",
			"type": "default"
		},
		{
			"key": "AXUAL_STREAM_NAME",
			"value": "my_test_stream",
			"type": "default"
		}
	]
}

  4. Open the Variables tab of the collection and set the three variables to your own values. Save the collection.

  5. Send the Produce event request to publish a record.