Runtime Provisioner 0.8.0 Readme

Request Body:

Returns 200 OK if deployment is successful.

Returns the status of KSML deployment(s). This endpoint supports two modes:

Detailed Status (single deployment)

Query Parameters: - tenant (required): The tenant name - instance (required): The instance name - environment (required): The environment name - application (required): The application name - deploymentMode (optional): Either StatefulSet or Job

Example: GET /status?tenant=dizzl&instance=sandbox&environment=dev&application=sensor-inspector

Returns the status of a specific KSML deployment including Helm and Kubernetes details.

The status can be one of Running, Starting, Failing, or Completed (for Jobs).

Bulk Status (all deployments for tenant/instance)

Query Parameters: - tenant (required): The tenant name - instance (required): The instance name

Example: GET /status?tenant=dizzl&instance=sandbox

Returns the status of all KSML deployments for the given tenant and instance combination. This searches for pods with matching axual.io/tenant and axual.io/instance labels. Each status includes resource limits (CPU in cores, memory in MB) from the KSML container.

Returns or streams logs for the KSML deployment. The response format is determined by the Accept request header.

Query Parameters: - tenant (required): The tenant name - instance (required): The instance name - environment (required): The environment name - application (required): The application name - deploymentMode (optional): Either StatefulSet or Job - tailLines (optional): Number of log lines to return or replay at stream start (1–1000, default 100)

Batch mode (default — no Accept header, or Accept: application/json):

Returns the last N lines of logs for the KSML deployment as a single JSON response.

Sample response:

Streaming mode (Accept: application/x-ndjson):

Streams real-time logs as NDJSON over a chunked HTTP response. The connection stays open and each log line is forwarded to the client as it arrives, making it suitable for live log tailing. A keepalive message is emitted every 15 seconds on idle connections to prevent intermediate proxies from closing the connection.

Each line in the response body is a self-contained JSON object:

Sample response (each line is a separate JSON object):

Returns the current CPU and memory usage metrics for the KSML deployment. This endpoint requires the Kubernetes Metrics Server to be installed in the cluster.

Query Parameters: - tenant (required): The tenant name - instance (required): The instance name - environment (required): The environment name - application (required): The application name - deploymentMode (optional): Either StatefulSet or Job

Sample response:

Note: If the Metrics Server is not installed or metrics are not yet available for a newly started pod, the endpoint will return a 404 error with an appropriate message.

Request Body:

Returns 200 OK if removal of deployment is successful

Reads the worker pods of one Kafka Connect cluster and merges their logs into one answer. Pods are found by the axual.io/tenant, axual.io/instance and axual.io/connect-cluster labels, so connectCluster must match the label, not the Helm release name.

tailLines is how many lines of history the answer holds in total, across the workers, from 0 to 1000. 0 means live lines only, which only a stream can serve. It bounds the answer and not the read: every worker is read over the whole total, and the merge then keeps the newest tailLines across the cluster, so one busy worker can supply the whole answer. It means the same in both modes, so a stream shows the same window a batch request would answer with, within the one limit described under streaming below.

connector keeps only the lines naming that connector. A JSON entry is matched on its connector.context field, which is exact. Any other line falls back to looking for the name in the text, which is still needed: Kafka’s own producer and consumer threads log outside the task’s logging context and name the connector in their client id instead, and 7 lines in 618 on a live worker were of that shape. Either way it runs after the read, so the read window is widened first, see “Filtering and the read window”.

There is no way to read the log of a container that has already stopped. That was tried and taken out: the node keeps the old log on its own schedule, so it was usually gone, and a worker killed outright writes no reason into its log anyway. To look at a crashed worker use kubectl logs --previous and kubectl describe pod, which also shows the exit code.

minLevel keeps only the lines at that severity or above, from TRACE, DEBUG, INFO, WARN, ERROR and FATAL. Any other value is a 400. It widens the read window for the same reason connector does, including when no connector is given: warnings and errors are 1.4% of a worker’s log, measured over 5995 lines, so a window sized to the request comes back nearly empty.

A kept line keeps the lines that belong to it. Two thirds of a worker’s lines are indented frames under a Java error and carry no level of their own, so a filter judging each line alone would keep Error while starting connector and drop the ninety lines explaining why.

A line whose own severity cannot be read follows the line above it, exactly as an indented frame does, because that is what such a line nearly always is. Keeping it regardless would leak everything a worker prints before its logging starts, and that is not a small amount: the first few hundred lines of a Kafka Connect worker are shell tracing and a dumped configuration file, none of which carry a level. A reader who asked for errors would get those instead of an empty window saying nothing matched. A blank line follows the line above it for the same reason.

So a log format this reader cannot classify at all disappears under a threshold. That is the honest outcome and it explains itself, because the empty window says how far it looked and which filter emptied it, and both layouts the chart writes always carry a level.

The severity is read from either log format: the log.level field of a JSON entry, or a level word near the start of a plain-text line. Both are what the chart’s two layouts produce.

Both stay supported even though kafka-connect-helm 0.5.0 writes JSON and no longer offers a choice. A cluster upgraded to that chart keeps hours of plain text in the same file until it rotates out, so one answer routinely mixes the two, and a cluster on an older chart, or one this chart did not create, can send plain text indefinitely. Note that the KSML endpoint is a different reader entirely: it streams lines without parsing them, so KSML has no severity filter and is unaffected by any of this.

Answers with one JSON document by default:

entries is the answer, in time order, oldest first. linesRead counts every raw line taken off the workers and linesReturned how many the answer holds, which is what tells “the cluster has nothing more” apart from “plenty was read and none of it matched”. truncated means the byte budget cut the answer short. degraded names the worker pods that could not be read at all, so the answer covers only the rest of the cluster.

Send Accept: application/x-ndjson to stream instead. Each line is then one JSON object, and every log line carries origin, the pod it came from, and timestamp, when the node stamped it:

A stream sends the history first and then follows the log. Both come from one read per worker, opened with the window and follow together, so the kubelet seeks back over the window and then keeps following the same file. So a client needs one request, not two.

Do not fetch the history with a second batch request. Two requests read two different windows that overlap at the recent end, and the client then receives the same lines from both with no way to tell them apart. A client that already holds the history opens the stream with tailLines=0, which asks for new output only. Platform Manager sends a fixed tailLines on its own stream, so through Platform Manager a stream always carries history: ask it once and do not fetch the history yourself.

The kubelet’s join is exact. Where the history ends is our judgement. One read cannot lose a line between the window and the new output, and cannot send the same line twice. But the stream carries no marker between the two, so the provisioner decides where the history stops: when the window has been read, when a line stamped later than the request arrives, when the read falls silent for 400 ms, or after 5 seconds, whichever comes first. Lines that miss the merged block are still sent, straight after it, in their own worker’s order. So a busy cluster can show a few lines out of order at the seam, and a stream forced to end its history early can send more lines than tailLines asked for. No line is lost, and none arrives twice.

That last point is the one difference between the two modes: tailLines bounds a batch answer exactly, while on a stream it bounds the merged block and the rest of the window still follows it.

The history lines arrive in time order across the workers, exactly as entries does, and each carries origin and timestamp, so a stream describes a line the same way the batch answer does. timestamp is absent on a line the node never stamped, which is a stack trace frame or another continuation of the line above it. New output cannot be ordered across workers at all, because holding lines back to sort them would delay a live tail, so each worker’s new lines are sent as they arrive.

A stream stays open while at least one worker still sends lines, and closes after 30 minutes.

When a filtered stream reads a wide window and matches nothing, the silent-stream notice says so, for example No lines for this connector in the last 4000 lines read from its workers. Waiting for output from the application…​. A reader can then tell a quiet connector from a window that did not look far enough back.

The notice names the filter that emptied the window, because a reader told the wrong one lifts the wrong control and still sees nothing. With minLevel alone it reads No lines at WARN or above in the last 4000 lines read from the workers. …​, and with both filters No lines for this connector at WARN or above in the last 4000 lines read from its workers. …​. A minLevel of TRACE keeps every level, so it is not named, exactly as it does not widen the read window.

When every worker returns fewer raw lines than the window asked for, the read reached the start of the log the node still has, and the notice says that instead: No lines for this connector in the whole log the workers still have, 340 lines. …​, or without a filter This is the whole log the workers still have, 340 lines. …​. That is the difference between “ask for more” and “there is no more”, which a line count alone does not show. The node’s retention is the wall no request can pass: the kubelet keeps containerLogMaxSize per file and containerLogMaxFiles of them, 10Mi and 5 by default.

The reader that finds a level in a plain-text line exists for three reasons, and all three end:

So it becomes safe to delete once every cluster runs 0.5.0 or later and their log files have rotated. Written down because otherwise it is either deleted too early or kept for ever by default.

The KSML endpoint never reaches this reader: it streams lines without parsing them. Platform Manager’s and the console’s ability to show a plain-text line is a separate matter and must stay, because KSML sends plain text permanently.

Most logging contexts are short, 21 to 31 bytes, such as [payments-sink|task-0]. A connector that copies between two clusters produced a 98-byte one, because it carries a whole client id:

It appeared on 4 lines out of thousands, so it costs nothing today. It could matter if a customer runs many cross-cluster connectors and one logs heavily, since the field is on every line and counts against the read window as well as disk. The fix, if it is ever needed, is a shorter field such as the connector name alone. Do not remove the field: it is what the connector filter matches on.

The split between history and new output is the provisioner’s own judgement, so three constants decide it. They are constants on purpose: no cluster has needed a different value, and a setting is a contract to keep.

Reading any of them too early costs the merge for the rest of the replay, or a little latency. None of them can lose or repeat a line. If a cluster ever needs different values, they follow the CONNECT_FILTER_WINDOW_PERCENT pattern: an environment variable, a chart value and a row in the settings table.

Answers 404 when no worker pod matches the labels, 400 for a bad parameter, 503 when the stream limit is full, and 501 in Docker mode, where there is no Kubernetes API to read.

Sends every worker’s current log file as one plain-text attachment, in labelled sections (===== worker <pod> =====). It shares nothing with the endpoint above: no filtering, no merging, no line budget. Bytes go from the Kubernetes stream to the socket in 64 KiB chunks, so the provisioner’s memory does not grow with the log.

connector, minLevel and tailLines are ignored, because reading everything is the point.

What it can return is bounded by the container runtime, not by this service. Kubernetes serves the container’s current log file only. Rotated files stay on the node and no API reaches them. Measured on the test cluster: a container that had written 59 MiB held 83 MB across five files on disk, docker logs on the node returned all 62 MB of it, and the Kubernetes API returned 2.4 MB, the current file. So this endpoint returns one rotation window, and how much that is depends on when the last rotation happened.

For scale, the runtime’s rotation size is the ceiling: 10Mi with the kubelet’s containerLogMaxSize default, 20MB with the Docker json-file driver’s max-size. Reading a whole file measured about 19 MiB/s through the Kubernetes API, so a few workers take a few seconds.

CONNECT_DOWNLOAD_CAP_BYTES caps the total across every worker, 64 MiB by default. It is a safety net rather than a feature limit: a real cluster never comes near it, and it exists for the case where somebody configures a much larger rotation size. Hitting it still produces a valid file, with a final line saying it was truncated and how many workers were copied.

Kubelet timestamps are deliberately off. The worker writes one JSON object per line and a timestamp prefix would stop the file being readable as NDJSON; every entry already carries its own @timestamp.

A worker that cannot be read is named in the file and the others are still copied. If no worker can be read the answer is a status code, not a file whose every line is an error message.

/healthz reports only that the process runs, and makes no Kubernetes call. /readyz lists one pod in each configured namespace, so an ok also proves the service account may read pods. Both answer {"status":"ok"} and need no authentication.

Every KSML endpoint above is also served under /ksml/, for example /ksml/logs. Both spellings behave identically. The flat paths are deprecated and will be removed in a later major version.

When a /deploy request arrives the provisioner: 1. Writes a ksml-runner.yaml configuration file (with inline Kafka settings, schema registry config, and KSML definition) into a per-container directory on the host. 2. Starts a Docker container from KSML_IMAGE, mounting that directory so the KSML runner can read its config. 3. Attaches the container to any Docker networks listed in DOCKER_CONFIG_NETWORKS so it can reach Kafka brokers and schema registries running in the same Compose project.

On /undeploy the container is stopped and removed. Config files written for that container are also cleaned up.

The /logs and /status endpoints work the same way as in Kubernetes mode. The /metrics endpoint is not supported in Docker mode and always returns 404.

Set these environment variables to run the Provisioner in Docker mode.

These environment variables change the default behaviour in Docker mode.

Below is a minimal example showing the provisioner alongside Kafka. The config directory is shared as a named volume so it works on both Linux and Docker Desktop for Mac.