Example: Producing data with Rest Proxy

This guide shows you how to produce a record to a String key and value topic through the Rest Proxy with curl: the access you need first, authenticating with TLS or with OAuth, and the full request that puts the two together.

Type

How-to guide

Goal

Produce a record to a topic through the Rest Proxy from the command line.

Audience

A developer holding produce access to the topic, and the certificate or client credentials the application was registered with.

When to use

Use this guide to check produce access to a topic, or as a template for a producing client.

For a wider set of curl requests covering every available feature, see the rest-proxy-examples repository.

Producing via curl

A produce request carries the environment and topic in its path, an authentication method the Rest Proxy accepts, and the key and value in its body. The subsections below cover each part, then assemble them into one request.

Prerequisites

Confirm the following before you begin:

An application registered in Self-Service, with produce access to the target topic requested and granted.
A topic whose key type and value type are both String, which is what the example below produces.

Security context

Client requests authenticate with either TLS or OAuth, depending on the Rest Proxy’s Security configuration.

TLS

Acquire the certificate and key file you used when registering your producer. In this example, the two files are in the working directory and are named auth.key and auth.cert.

OAuth

Obtaining a valid JSON Web Token (JWT) and refreshing it before it expires is the caller’s job. The example below retrieves a token from Keycloak:

curl -X POST \
  https://keycloak.cloud.axual.com/auth/realms/axual/protocol/openid-connect/token \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'client_id=<producer_client_id>' \
  -d 'grant_type=client_credentials' \
  -d 'client_secret=<your_client_secret>'

Extract the JWT from the response. The token must carry a claim named after your configured security.oauth.enabled.principal-claim-name string. Kafka authenticates and authorises the request against the contents of that claim, so the topic you produce to needs Access Control Lists (ACLs) matching it.

Prepare your request header following the format:

  --header "Authorization: Bearer <obtained_token>"

Putting it all together

Replace the values in the first few lines of the script below with your own deployment information, then run the commands.

REST_PROXY_URL="https://rest-ams01.cloud.axual.com:99999"
AXUAL_ENVIRONMENT="example" (1)
AXUAL_TOPIC_NAME="my_test_topic" (2)
UUID="unique_producer_$RANDOM" (3)

curl -k --request POST \
  --url "${REST_PROXY_URL}/topic/${AXUAL_ENVIRONMENT}/${AXUAL_TOPIC_NAME}" \
  --header "axual-application-id: irrelevant-for-producers" \
  --header "axual-producer-uuid: ${UUID}" \
  --header "Content-Type: application/json" \
  --key ./auth.key \ (4)
  --cert ./auth.cert \
  --data '
  {
     "keyMessage":{
        "type":"STRING", (5)
        "message":"{\"name\": \"axl\"}"
     },
     "valueMessage":{
        "type":"STRING", (6)
        "message":"{\"employer\": \"Axual\", \"age\": \"21\"}"
     }
  }'

1	Environment `Short name` as it appears in Self-Service
2	Topic `Name` as it appears in Self-Service
3	Associates the session with a Producer object. `$RANDOM` is a bash builtin that yields a different value on each run. Later calls that reuse the same UUID avoid the cost of creating another Producer
4	Depending on your configuration, use the OAuth header here instead of the TLS configuration
5	Must match the Topic key type
6	Must match the Topic value type

You can also import any of the curl examples into Postman with the Raw Text option, pasting the command.

Verify the record arrived

A successful produce returns HTTP 200 and a Rest Proxy Definitions: ProduceResponse body naming the cluster, topic, partition and offset the record landed on:

{
    "cluster": "amsterdam-01",
    "offset": 0,
    "timestamp": 1569911914135,
    "stream": "general-test",
    "partition": 0
}

A 403 means the application has no produce access to the topic, and a 404 means the environment short name or topic name in the path matches nothing in Self-Service. To read the record back, open the topic in Topic Browse (Messages).