REST Proxy

Functionality

Overview

The REST Proxy can be used to interact with Kafka without using Kafka client libraries. It is possible to produce or consume data in a secure way using API calls over HTTPS.
Due to the HTTP overhead, the throughput is much lower compared to native Kafka clients.

README & Changelog

More details are in the REST Proxy 1.12.0 Readme and the REST Proxy 1.12.0 Changelog.

Example use

Suppose in your company a tool is used that could write relevant event data on a topic but does not have any way to connect to Kafka. If the tool allows for publishing data via HTTP, the REST Proxy can be used to get the correct data onto Kafka topics.

For an extensive set of curl requests demonstrating all available features, you can refer to the rest-proxy-examples repository.

API

Overview

Applications using the REST Proxy should be registered in Self Service and use Self Service to get access to topics. This will result in Access Control Lists being created on the clusters using these resources. REST Proxy will perform authentication and authorization using Mutual TLS, and verify that the client has access to the resources by checking the Kafka cluster Access Control Lists.

Avro

Avro schemas are supported for keys and/or values. These schemas need to be provided with every produce call either in String form or by ID reference. The latter is preferred since passing the entire schema string will result high overhead that can be avoided. The Avro service can provide the id of a specified schema for a specific topic and environment.

See Avro Schema Service for more information.

Produce

The Produce service enables clients to produce messages to specific topics. REST Proxy supports multiple datatypes for key and value.

See Produce Service for more information.

Consume

The Consume service enables clients to consume messages from specific topics. Multiple datatypes for key and value are supported. The expected key and value type should match the topic definition on Self Service.

See Consume Service for more information.

Security

Before performing any operation against Kafka, client requests need to be authenticated and authorized. To achieve this, the request must contain a security context that can be used for authorization. As stated, the client request must match the ACLs present on the Kafka topic, so Self Service registration of producer or consumer access should be in place.

REST Proxy supports two ways to match Kafka ACLs: MTLS or OAuth (since version 1.12.0). The distinction is made through startup configuration.

MTLS

The client is required to present a valid certificate and key as part of their request. The REST Proxy extracts the string representation of the certificate chain from the request in the form of a distinguished name containing the entire certificate chain. This chain is, in turn, used to verify that the request is authorized to access the Topic and/or Group.

OAuth

The client request should contain an authorization header in the form of Bearer <encrypted token>, as obtained from the external identity provider.

Getting this token is outside the scope of REST Proxy, the client needs to run the authentication flow against their own identity provider to get the token and be sure the token is valid and not expired.

The operator must configure the server that REST Proxy uses to verify the validity of request tokens. The operator must also configure a claim name that is used to extract the principal which is used to authorize requests against Kafka. For authenticated requests, REST Proxy extracts the predefined claim from the token by accessing the claim defined as a string. The extracted string is used to verify that the request is authorized to access the Topic and/or Group, so it needs to match the ACL on the resource of the request.

Installation

REST Proxy is a standalone component that is not dependent on another component. Without Kafka and Axual Self-Service it would not function, however.

Helm Charts

As part of the Streaming Helm charts, the REST Proxy can be installed following the guide Axual Streaming & Governance Installation.

Configuration

Helm configuration

All details of the Helm Chart configurations are in REST Proxy 1.12.0 Helm Readme

Overriding REST Proxy Producer/Consumer Settings

You can override any REST proxy producer/consumer configuration supported by Kafka clients. It’s fully dynamic and flexible.

Create a values.yaml file in your working directory.

Replace kafka configuration dots . with dashes -

values.yaml

axual-streaming:
  rest-proxy:
    axual:
      producer:
        # producer configuration
        config:
          # Overrides kafka producer configuration
          metadata-max-age-ms: 180000
          connections-max-idle-ms: 180000
          request-timeout-ms: 120000
          retries: 3
          max-block-ms: 60000
          acks: all
          batch-size: 10
          linger-ms: 1
          max-in-flight-requests-per-connection: 5
          send-buffer-bytes: 10000
          receive-buffer-bytes: 10000
      consumer:
        # consumer configuration
        numberOfThreads: 10
        config:
          # Overrides kafka consumer configuration
          metadata-max-age-ms: 180000
          connections-max-idle-ms: 180000

If you want to override configuration per produce/consume request, then it’s possible, but there are limited configuration options. You can add the config in request HTTP Headers.

Produce request supported HTTP Headers
Consume request supported HTTP Headers

Static Kafka and Schema Registry configuration

Since version 1.12.0, REST Proxy no longer relies on the deprecated Discovery Service. The Axual static configuration can be set in the values.yaml file.

Example of Static Configuration

values.yaml

axual-streaming:
  rest-proxy:
    axual:
      static-configuration:
        tenant: "axual"
        instance: "local"
        cluster: "local"
        bootstrap.servers: "platform.local:31757"
        schema.registry.url: "https://platform.local:25000"
        group.id.resolver: "io.axual.common.resolver.GroupPatternResolver"
        group.id.pattern: "{tenant}-{instance}-{environment}-{group}"
        topic.resolver: "io.axual.common.resolver.TopicPatternResolver"
        topic.pattern: "{tenant}-{instance}-{environment}-{topic}"
        transactional.id.resolver: "io.axual.common.resolver.TransactionalIdPatternResolver"
        transactional.id.pattern: "{tenant}-{instance}-{environment}-{transactional.id}"
        acl.principal.builder: "io.axual.security.principal.AdvancedAclPrincipalBuilder"

Security

Since version 1.12.0, REST Proxy adds support for OAuth next to TLS. Clients are now able to authenticate and authorize their requests using an Oauth Authentication header as an alternative to certificate and key.

To enable this, the operator must provide some additional security configuration.

values.yaml

axual-streaming:
  rest-proxy:
    spring:
      security:
        oauth2:
          resourceserver:
            jwt:
              issuer-uri: https://<keyloak.url>/auth/realms/<realm> (1)
    security:
      oauth:
        enabled: true
        principal-claim-name: axual_application_custom_claim (2)
      mtls:
        enabled: false (3)

1	Used by spring to validate the JWT tokens provided in requests
2	Required if oauth is enabled. Name of the claim from the JWT authentication token which will be used to extract the user principal.
3	Default enabled, both are supported simultaneously, TLS takes precedence

Logging

Change the logging configuration by following these steps.

Metrics

The REST Proxy exposes the default Prometheus metrics via the Spring Boot Actuator, with some additional custom metrics:

consumers.open - Number of currently active consumers
producers.open - Number of currently active consumers
consumer.create.timer - Total consumer creation time
producer.create.timer - Total producer creation time
endpoint.<endpoint>.total - Total <endpoint> calls
endpoint.<endpoint>.success - Total successful <endpoint> calls
endpoint.<endpoint>.fail - Total failed <endpoint> calls
endpoint.<endpoint>.forbidden - Total forbidden <endpoint> calls

More info about Monitoring & Metrics here.