Configuration

Contents

Rest Proxy Configuration

The default values for Rest proxy are in platform-deploy/include/defaults/rest-proxy.sh

To override them for a specific Rest proxy instance, you will need to modify platform-config/tenants/{tenant-name}/instances/{instance-name}/rest-proxy.sh file.

Configuration Required / Optional Description Default Value Since

REST_PROXY_ENABLED

Optional

Rest-Proxy enabled

false

1.0.0

RESTPROXY_VERSION

Optional

Determines the docker image version to use for rest-proxy

false

1.0.0

REST_PROXY_LOGLEVEL_ROOT

Optional

Root logging level

WARN

1.0.0

REST_PROXY_LOGLEVEL_APACHE_KAFKA

Optional

Kafka logging level

ERROR

1.0.0

REST_PROXY_LOGGING_LEVEL_IO_AXUAL_PROXY_REST

Optional

Rest-Proxy logging level

INFO

1.0.0

REST_PROXY_LOGBACK_CONFIG

Optional

Logging configuration

/home/kafka/logback.xml

1.0.0

REST_PROXY_ACL_RETRY_SLEEP

Optional

Interval (in ms) between AdminClient method call retries

100

1.0.0

REST_PROXY_ACL_ADMIN_CLIENT_REQUEST_TIMEOUT_MS

Optional

Provided to AdminClient on creation as AdminClientConfig.REQUEST_TIMEOUT_MS_CONFIG

2000

1.0.0

REST_PROXY_AVRO_MAX_SCHEMAS_PER_SUBJECT

Optional

Serde config: Maximum number of schemas to create or cache locally

10

1.0.0

REST_PROXY_REGISTRY_CLEANUP_INTERVAL

Optional

Consumer cleanup interval (in ms)

60000

1.0.0

REST_PROXY_REGISTRY_KEEPALIVE_INTERVAL

Optional

Active Consumers will call poll() every such interval (in ms) so that they are not marked dead

3000

1.0.0

REST_PROXY_CONSUMER_THREADS

Optional

The number of threads that the server spawns to process messages per consume request

15

1.0.0

REST_PROXY_OPEN_ENDPOINTS

Optional

Prometheus open endpoints

/actuator/prometheus, /actuator/health

1.0.0

REST_PROXY_MGMT_ENDPOINTS_INCLUDE

Optional

Management open endpoints

health,prometheus

1.0.0

REST_PROXY_MGMT_SERVER_SSL_ENABLED

Optional

Management server SSL enable

false

1.0.0

REST_PROXY_MGMT_SECURITY_ENABLED

Optional

Management security enable

false

1.0.0

REST_PROXY_SECURITY_SUPERUSERS_PRINCIPALS

Optional

Superusers are always allowed to impersonate

<no default>

1.0.0

REST_PROXY_SECURITY_SUPERUSERS_PRINCIPALS_CHAIN

Optional

This will be used instead of REST_PROXY_SECURITY_SUPERUSERS_PRINCIPALS in case USE_PRINCIPAL_CHAIN is true on the cluster

<no default>

1.0.0

REST_PROXY_SECURITY_IMPERSONATORS

Optional

Impersonators are allowed to call the Proxy on behalf of another application. The certificate DN or chain of DNs must be provided using the X-SSL-Client-DN header

<no default>

1.0.0

REST_PROXY_SECURITY_IMPERSONATORS_CHAIN

Optional

This will be used instead of REST_PROXY_SECURITY_IMPERSONATORS in case USE_PRINCIPAL_CHAIN is true on the cluster

<no default>

1.0.0

REST_PROXY_SECURITY_PROTOCOL

Required

The security protocol used by Rest proxy, SSL is advised

<no default>

1.0.0

REST_PROXY_PRINCIPAL

Required

Rest-Proxy Principal

<no default>

1.0.0

REST_PROXY_PRINCIPAL_CHAIN

Required

This will be used instead of REST_PROXY_PRINCIPAL_CHAIN in case USE_PRINCIPAL_CHAIN is true on the cluster

<no default>

1.0.0

REST_PROXY_SERVER_SSL_SECURITY_PROTOCOL

Required

Server SSL Protocol. TLSv1.2 is suggested

<no default>

1.0.0

REST_PROXY_SERVER_SSL_KEYSTORE_PASSWORD

Required

Server keystore password

<no default>

1.0.0

REST_PROXY_SERVER_SSL_KEY_PASSWORD

Required

Server key password

<no default>

1.0.0

REST_PROXY_SERVER_SSL_CLIENT_AUTH

Required

Client authentication

want

1.0.0

REST_PROXY_CLIENT_SSL_KEY_STORE_PASSWORD

Required

Client keystore password

<no default>

1.0.0

REST_PROXY_CLIENT_SSL_KEY_PASSWORD

Required

Client key password

<no default>

1.0.0

REST_PROXY_JMX_ENABLED

Required

JMX enabled

false

1.0.0

REST_PROXY_PROMETHEUS_ENABLED

Optional

Prometheus enabled

false

1.0.0

REST_PROXY_PORT

Required

Port to be used to connect to Rest-Proxy

<no default>

1.0.0

REST_PROXY_PROMETHEUS_PORT

Required

Prometheus scrapes the metrics from this port

<no default>

1.0.0

REST_PROXY_JMX_PORT

Required

Metrics are exposed on this port

<no default>

1.0.0

Overriding Rest Proxy Producer/Consumer Settings

You can override any Rest proxy producer/consumer configuration that are supported by kafka. It’s fully dynamic and flexible.

Axual CLI

To override configuration in Axual CLI, you have to add producer/consumer prefix following the configuration key and a value.

REST_PROXY_PRODUCER_OVERRIDE_ is a prefix for producer. And REST_PROXY_CONSUMER_OVERRIDE_ is a prefix for consumer config and rest is a configuration key and a value.

{Producer/Consumer prefix}_{kafka configuration}
For kafka configuration replace dots . with underscores _

Example:

platform-config/tenants/{tenant-name}/instances/{instance-name}/rest-proxy.sh
# Rest-Proxy Producer Config
REST_PROXY_PRODUCER_OVERRIDE_METADATA_MAX_AGE_MS=180000
REST_PROXY_PRODUCER_OVERRIDE_CONNECTIONS_MAX_IDLE_MS=180000
REST_PROXY_PRODUCER_OVERRIDE_REQUEST_TIMEOUT_MS=15000
REST_PROXY_PRODUCER_OVERRIDE_RETRIES=1
REST_PROXY_PRODUCER_OVERRIDE_MAX_BLOCK_MS=15000
REST_PROXY_PRODUCER_OVERRIDE_ACKS=all
REST_PROXY_PRODUCER_OVERRIDE_BATCH_SIZE=16384
REST_PROXY_PRODUCER_OVERRIDE_LINGER_MS=50
REST_PROXY_PRODUCER_OVERRIDE_MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION=10
REST_PROXY_PRODUCER_OVERRIDE_SEND_BUFFER_BYTES=131072
REST_PROXY_PRODUCER_OVERRIDE_RECEIVE_BUFFER_BYTES=32768

# Rest-Proxy Consumer Config
REST_PROXY_CONSUMER_OVERRIDE_METADATA_MAX_AGE_MS=180000
REST_PROXY_CONSUMER_OVERRIDE_CONNECTIONS_MAX_IDLE_MS=180000
REST_PROXY_CONSUMER_OVERRIDE_AUTO_OFFSET_RESET=latest
REST_PROXY_CONSUMER_OVERRIDE_MAX_POLL_RECORDS=20

Helm Charts

With k8s deployments are getting easier. To achieve that we have developed our Axual Helm Charts to run the Axual Platform on your local machine. Check Using the HELM charts section.

  1. Create a values.yaml file in your working directory.

    Replace kafka configuration dots . with dashes -
    values.yaml
    instance:
  restproxy:
    axual:
      producer:
        # Rest-Proxy configuration
        config:
          # Overrides kafka producer configuration
          metadata-max-age-ms: 180000
          connections-max-idle-ms: 180000
          request-timeout-ms: 120000
          retries: 3
          max-block-ms: 60000
          acks: all
          batch-size: 10
          linger-ms: 1
          max-in-flight-requests-per-connection: 5
          send-buffer-bytes: 10000
          receive-buffer-bytes: 10000
      consumer:
        # Rest-Proxy configuration
        numberOfThreads: 10
        config:
          # Overrides kafka consumer configuration
          metadata-max-age-ms: 180000
          connections-max-idle-ms: 180000

  2. Issue a helm upgrade command with the values.yaml as input file to upgrade Axual Platform configuration.

    helm upgrade --install platform axual-stable/platform -f ./values.yaml -n kafka

  3. Alternatively you can override a specific property with the --set option

    helm upgrade --install platform axual-stable/platform --set instance.restproxy.axual.producer.config.metadata-max-age-ms=200000 -n kafka

If you want to override configuration per produce/consume request, then it’s possible but there is limited configuration options. You can add the config in request HTTP Headers.

Produce request supported HTTP Headers
Consume request supported HTTP Headers

Static Kafka and Schema Registry configuration

With Helm deployments, it is possible to configure the Rest Proxy in a way to exclude the dependencies from the Discovery Service.

This can be achieved with proper configuration in the values.yml file.

The relevant configurations are the following:

Configuration Required / Optional Description Default Value

config-mode

Optional

Configuration via discovery or static

discovery

staticConfig:

Required if config-mode is static

Static configuration for the Rest Proxy

<no default>

Below an example of a static configuration:

values.yaml
instance:
  restproxy:
    axual:
    config-mode: "static"
    staticConfig: # Only used if `config-mode` = `static`
      tenant: "axual"
      instance: "local"
      cluster: "local"
      bootstrap.servers: "platform.local:31757"
      schema.registry.url: "https://platform.local:25000"
      group.id.resolver: "io.axual.common.resolver.GroupPatternResolver"
      group.id.pattern: "{tenant}-{instance}-{environment}-{group}"
      topic.resolver: "io.axual.common.resolver.TopicPatternResolver"
      topic.pattern: "{tenant}-{instance}-{environment}-{topic}"
      transactional.id.resolver: "io.axual.common.resolver.TransactionalIdPatternResolver"
      transactional.id.pattern: "{tenant}-{instance}-{environment}-{transactional.id}"
      acl.principal.builder: io.axual.security.principal.AdvancedAclPrincipalBuilder