Kafka Streams Prefixed ACLs

The sections that follow build up from the runtime behaviour that creates the need for prefixed ACLs, through the collision check that guards them, to what happens when the underlying grants change:

A Kafka Streams application does more than read and write the topics you declare in Self-Service. To hold state between runs, the Kafka Streams client conditionally creates its own internal topics at runtime, for example repartition and changelog topics that back stateful operations. These topics are transparent to you: they never appear in Self-Service, yet a running instance depends on them.

Because those topics do not exist when the application is registered, their access rights cannot be granted one topic at a time. Instead, the authentication a user configures produces a prefixed topic ACL, a single grant that covers every topic name sharing a common prefix. This grant is created as part of authentication configuration, the same step that provisions the application’s credentials, so the internal topics are authorised the moment the client creates them.

The prefix is derived from the application ID rather than chosen freely. Internal topics created through Kafka Streams clients follow an illustrative naming pattern:

Every such name begins with the application ID, so a single ACL on the prefix <application.id>- covers all of them. That is why the application ID chosen at the application ID field is not merely a label: it defines the exact set of topic names the application will be permitted to access, and it is the reason the value has to be verified before the ACL is created.

A prefixed ACL is broad by design. The same breadth that authorises unknown future topic names also carries a risk: a poorly chosen prefix could grant an application access to topics you never intended to expose. If one application’s prefix is a prefix of another topic’s name, the first application gains access to that topic through its ACL alone.

To close that gap, Axual checks the application ID per environment when authentication is created for a Kafka Streams application. The check rejects an application ID when either of the following is true:

When either condition holds, the application ID is rejected and a new value has to be supplied. The check runs per environment because ACLs and topics are scoped to an environment, so a prefix that is safe in one environment may collide in another.

The prefixed ACL exists only as long as the authentication that produced it. Deleting the credentials or principals of a Kafka Streams application revokes the ACLs that grant access to its internal topics.

The consequence is immediate for anything still running. An instance that depends on internal topics on that environment loses its authorisation and surfaces authorisation errors, because the topics it needs to read and write its own state are no longer within a grant it holds.

The application ID field remains editable after registration, but because the prefix is derived from it, changing the ID changes the prefixed ACL. A new ID means a new prefix, and a new prefix has to be validated the same way the original one was.

Modifying the application ID therefore triggers collision detection across every environment where the application has authentication configured. If a collision is detected in any one of those environments, the new application ID is rejected, and the old value stays in effect. When the new ID passes the check in all environments, the prefixed topic ACLs of the old application ID are replaced by those of the new one.