Rest Proxy

Functionality

Overview

The REST Proxy can be used to interact with Kafka without using Kafka libraries. It is possible to produce or consume data in a secure way using API calls over HTTPS.
Due to the HTTPS overhead, the throughput is much lower compared to native Kafka clients.

Rest Proxy Overview

README & Changelog

More details are in the REST Proxy 1.10.5 Readme and the REST Proxy 1.10.5 Changelog

Example use

Suppose in your company a tool is used that could write relevant event data on a topic but does not have any way to connect to Kafka. If the tool allows for publishing data via HTTPS, the REST Proxy can be used to get the correct data onto Kafka topics.

This repository contains how-to examples for Rest Proxy

API

Overview

Applications using the Rest Proxy should be registered in Self Service and use Self Service to get access to streams. This will result in Access Control Lists being created on the clusters using these resources. Rest Proxy will perform authentication and authorisation using Mutual TLS, and verify that the client has access to the resources by checking the cluster Access Control Lists.

Avro

Streams can use Avro schema definitions for keys and/or values. These schemas need to be provided with every produce call. This will usually result in a relatively high overhead in data. The schemas are stored in the backend and ids are provided for these schemas. The produce call also accepts these ids. The Avro service can provide the id of a specified schema for a specific stream and environment.

See Avro Schema Service for more information.

Produce

The produce service enables clients to produce messages to specific streams. The Rest Proxy supports multiple datatypes for key and value.

See Produce Service for more information.

Consume

The consume service enables clients to consume messages from specific streams. The Rest Proxy supports multiple datatypes for key and value. The expected key and value type should be static per stream

See Consume Service for more information

Installation

The rest-proxy is a standalone component that is not dependent on another component. Without Kafka and Axual Self-Service it would not function however.

Helm Charts

As part of the Streaming Helm charts, the REST Proxy can be installed following the guide Axual Streaming & Governance Installation.

Configuration

Helm configuration

All details of the Helm Chart configurations are in REST Proxy 1.10.5 Helm Readme

Overriding Rest Proxy Producer/Consumer Settings

You can override any Rest proxy producer/consumer configuration that are supported by Kafka. It’s fully dynamic and flexible.

  1. Create a values.yaml file in your working directory.

    Replace kafka configuration dots . with dashes -
    values.yaml
    streaming:
  restproxy:
    axual:
      producer:
        # Rest-Proxy producer configuration
        config:
          # Overrides kafka producer configuration
          metadata-max-age-ms: 180000
          connections-max-idle-ms: 180000
          request-timeout-ms: 120000
          retries: 3
          max-block-ms: 60000
          acks: all
          batch-size: 10
          linger-ms: 1
          max-in-flight-requests-per-connection: 5
          send-buffer-bytes: 10000
          receive-buffer-bytes: 10000
      consumer:
        # Rest-Proxy consumer configuration
        numberOfThreads: 10
        config:
          # Overrides kafka consumer configuration
          metadata-max-age-ms: 180000
          connections-max-idle-ms: 180000

If you want to override configuration per produce/consume request, then it’s possible but there is limited configuration options. You can add the config in request HTTP Headers.

Produce request supported HTTP Headers
Consume request supported HTTP Headers

Static Kafka and Schema Registry configuration

From version 1.9.2 of the Rest Proxy, it is possible to configure the Rest Proxy in a way to exclude the dependencies from the now obsolete Discovery Service.

This can be achieved with proper configuration in the values.yml file.

The relevant configurations are the following:

Configuration Required / Optional Description Default Value

config-mode

Optional

Configuration via discovery or static

discovery

staticConfig:

Required if config-mode is static

Static configuration for the Rest Proxy

<no default>

Below an example of a static configuration:

values.yaml
axual-streaming:
  restproxy:
    axual:
    config-mode: "static"
    staticConfig: # Only used if `config-mode` = `static`
      tenant: "axual"
      instance: "local"
      cluster: "local"
      bootstrap.servers: "platform.local:31757"
      schema.registry.url: "https://platform.local:25000"
      group.id.resolver: "io.axual.common.resolver.GroupPatternResolver"
      group.id.pattern: "{tenant}-{instance}-{environment}-{group}"
      topic.resolver: "io.axual.common.resolver.TopicPatternResolver"
      topic.pattern: "{tenant}-{instance}-{environment}-{topic}"
      transactional.id.resolver: "io.axual.common.resolver.TransactionalIdPatternResolver"
      transactional.id.pattern: "{tenant}-{instance}-{environment}-{transactional.id}"
      acl.principal.builder: io.axual.security.principal.AdvancedAclPrincipalBuilder

Logging

Change the logging configuration by following these steps.

Metrics

The REST Proxy exposes the default Prometheus metrics via the Spring Boot Actuator, with some additional custom metrics:

  • consumers.open - Number of currently active consumers

  • producers.open - Number of currently active consumers

  • consumer.create.timer - Total consumer creation time

  • producer.create.timer - Total producer creation time

  • endpoint.<endpoint>.total - Total <endpoint> calls

  • endpoint.<endpoint>.success - Total successful <endpoint> calls

  • endpoint.<endpoint>.fail - Total failed <endpoint> calls

  • endpoint.<endpoint>.forbidden - Total forbidden <endpoint> calls

More info about Monitoring & Metrics here.