Managing Applications

An application registration via the Self-Service portal is necessary in order to provide access to the kafka-cluster. This operation can be done before the application is actually developed and/or released.

The portal also uses this information to provide a great visual overview of all the active streaming data pipelines on the streaming platform.

Application Types

Applications have been grouped to reflect their management style. The new types are

Custom to Self Managed
Connector to Managed Connector
KSML to Managed KSML

When creating an application using the Self-Service portal, you need to define the Application Type, which can be one of "Self Managed", "Managed Connector" or "Managed KSML":

Self Managed: Standalone (Java, .NET, Kafka Streams etc.) applications hosted by the user using kafka clients
- Kafka Streams applications are newly introduced with 2024.4. Using this type, it is possible to run a fully functional Kafka Streams application managed by Axual. The additional steps that take place in the background to achieve this involve creating prefixed Kafka ACLs, which allow for internal topic operations Kafka Streams Applications rely on for persisting their state. These steps take place when the said Application has its authentication configured and are transparent to the user.
Managed Connector: A Connector is a generic pluggable piece of code which can communicate with a kafka-cluster and an external system (like a database). Because of how generic it is, a user is only required to provide configuration in order to connect the two systems. Connector applications run on Axuals infra (within a "Connect" cluster). The integrated system can be a data source or a data sink. There are different Connector Plugins for different external systems.
Managed KSML: KSML is a wrapper around Kafka Streams that allows for development of low code stream processing applications. It provides an easy way for developers to write streaming applications without requiring Java programming.

This page will describe the registration of Custom application.

If you want to register Connector applications, go to: Starting Connectors

Creating a Self Managed Application

To register your application, visit the Applications page and click the New Application button.
You will see a screen as below:

Fill out the form with the required information:

ID: This is a string that uniquely identifies your application. We recommend using a fully qualified package/class name as it is unique enough within the organization, e.g. com.company.division.AlertingApp. The maximum length is 255 and the value should be alphanumeric with no spaces and underscores are allowed.

The Application ID is used by consuming applications to keep track of where it left off consuming messages from a topic. Consumers sharing the same ID share the events on a topic. So one event can be only read by one consumer with the same Application ID (read more about consumer groups).

Name: The name of the application. It should not be more than 50 characters. Spaces are allowed.
Short Name: A unique human-readable short name of the application. This name is used as a label in Self Service. Only alphanumeric characters and underscore are allowed. Max length should not exceed 60 characters.
Owner: Choose the group which would be the owner of this application. Choose your DevOps team’s group. Is it not available yet? Please refer to Creating a group.
Application Type: choose Self Managed
Type: Choose the type of application, serves as metadata.
Visibility: see Application visibility below
Description: A short summary describing the purpose of this application, e.g. "Enriching transaction events with transaction categories.". Must not exceed 200 characters.
Viewer Groups: (Optional) Choose the groups which would be able to view this application configuration. Please refer to Viewer Groups section.
Properties: Allows you to add additional metadata as Key Value pairs to describe the application.

At this point, you have created your Application in Self Service. Once the application is successfully created, you will be directed to the application detail page.

Creating a Self Managed Kafka Streams Application

Application Registration

To register your application, visit the Applications page and click the New Application button.
You will see a screen as below:

Fill out the form with the required information:

ID: This is a string that uniquely identifies your application. It corresponds to the (unresolved) application.id field that you need to provide in your KafkaStreams client application (read more about Kafka Streams). We recommend using a fully qualified package/class name as it is unique enough within the organization, e.g. com.company.division.AlertingApp. The maximum length is 255 and the value should be alphanumeric with no spaces and underscores are allowed.

The application.id you provide here needs to not collide with any existing Kafka Streams based application or Topic deployment within the environment you deploy. More details follow in Managing Prefixed ACLs.

Name: The name of the application. It should not be more than 50 characters. Spaces are allowed.
Short Name: A unique human-readable short name of the application. This name is used as a label in Self Service. Only alphanumeric characters and underscore are allowed. Max length should not exceed 60 characters.
Owner: Choose the group which would be the owner of this application. Choose your DevOps team’s group. Is it not available yet? Please refer to Creating a group.
Application Type: choose Self Managed
Type: Choose Kafka Streams.
Visibility: see Application visibility below
Description: A short summary describing the purpose of this application, e.g. "Enriching transaction events with transaction categories.". Must not exceed 200 characters.
Viewer Groups: (Optional) Choose the groups which would be able to view this application configuration. Please refer to Viewer Groups section.
Properties: Allows you to add additional metadata as Key Value pairs to describe the application.

Self-Service Add Kafka Streams Application

Press the Add Application button. Once the application is successfully created, you will be directed to the application detail page.

At this point, you have created your Application in Self Service. Once the application is successfully created, you will be directed to the application detail page.

Self-Service Kafka Streams Application Detail

Managing Prefixed Topic ACLs

Additional to the Topic Access rights which are viewable and manageable through the Self-Service, Kafka Streams applications may require creation of topics used for managing internal state. Those topics are transparent to the user, they do not appear in Self-Service, however the application conditionally creates them at runtime and uses them to store its state in subsequent runs.

To enable this, the authentication configuration provided by the user will be used to generate Prefixed Topic ACLs, which grant access to the required topics.

Topics created internally through Kafka Streams clients follow the naming pattern:

<application.id>-<operatorName>-<suffix>

Consequently, the Prefixed Topic ACLs we attach to the application during creation of the Authentication are on the pattern <application.id>-. This in turn means that the selection of the Application Id needs to be verified such that collisions can be averted.

Creating Authentication

Once the application registration is complete, authentication configuration follows.

Prefixed Topic ACL Collision Prevention

Prefixed Topic ACL creations run the risk of allowing applications to access Topics that have not been approved explicitly by the user. To alleviate this, the Application Id provided during registration of the Application will be checked per Environment during creation of Authentication for the Kafka Streams based Applications.

If any of the following conditions are met:

An existing Kafka Streams-based application is already configured where the Prefixed ACL would overlap with the one being created
A configured Topic exists whose name would fall under the prefixed ACL being created.

then the provided Application ID will be rejected, and the user will need to provide a new value.

Deleting Authentication

Deletion of previously created Credentials or Principals results in revoking of the ACLs required to access internal topics. This may result in authorization errors of any running instance requiring internal Topics on the given environment.

Modifying Application Id

Application Id field is open to modification, however, doing so will trigger Prefixed ACL collision detections through all the Environments where the application has Authentications configured. In case any collision is detected in any Environment the new Application Id may be rejected.

Once successful, the Prefixed Topic ACLs of the old Application Id will be replaced by the ones of the new.

Using Schema SerDes

The following applies only to Instances where Apicurio Schema Registry is used. To use Schemas in your Kafka Streams application, ensure that your Tenant Admin has created an Instance with Apicurio Keycloak details.

The Kafka Streams Topology you want to create may require Schema support. In that case, the internal topics may also consist of Schema serialized messages. To achieve this, your application needs to have automatic Schema read and write rights against Apicurio.

This is now possible through the creation of Schema Registry credentials in the Application Credentials modal. If the Instance is configured correctly, the modal should show this button which allows creating such credentials like below:

Create application schema-registry credentials model

The credentials created allow for the dynamic creation of Schemas in Apicurio. First, be sure to enable subject auto-registration by the application. Then copy the key and password values generated and provide them to your Kafka Streams Application alongside any other related configuration.

props.put(org.apache.kafka.streams.StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, AvroSerde.class.getName());
props.put(org.apache.kafka.streams.StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, AvroSerde.class.getName());
props.put(io.apicurio.registry.serde.REGISTRY_URL, schemaRegistryUrl);
props.put(io.apicurio.registry.serde.SerdeConfig.AUTO_REGISTER_ARTIFACT, true);
props.put(io.apicurio.registry.serde.SerdeConfig.AUTH_USERNAME, username);
props.put(io.apicurio.registry.serde.SerdeConfig.AUTH_PASSWORD, password);

Your application is now set to use Schemas to Serialize and Deserialize messages.

Creating a Managed KSML Application

To register your KSML application, visit the Applications page and click the New Application button.
You will see a screen as below:

Fill out the form with the required information:

ID: This represents the Application ID of your KSML application.This is a string that uniquely identifies your application.We recommend using a fully qualified package/class name as it is sufficiently unique within the organization, e.g. com.company.division.AlertingApp.The maximum length is 255 and the value should be alphanumeric with no spaces and underscores are allowed.

As mentioned, Kafka Streams based applications require creating prefixed Kafka ACLs which allow for creation of topics used for managing internal state. The application.id you provide here needs to not collide with any existing Kafka Streams based application or Topic deployment within the environment you deploy. This check will be performed during authentication configuration.

Name: The name of the KSML application.It should not be more than 50 characters.Spaces are allowed.
Short Name: A unique human-readable short name of the application.This name is used as a label in Self Service.Only alphanumeric characters and underscore are allowed.Max length should not exceed 60 characters.
Owner: Choose the group which would be the owner of this application.Choose your DevOps team’s group.Is it not available yet?Please refer to Creating a group.
Application Type: choose Managed KSML.
Visibility: see Application visibility below.
Description: A short summary describing the purpose of this application, e.g. "Enriching transaction events with transaction categories.".Must not exceed 200 characters.
Properties: Allows you to add additional metadata as Key Value pairs to describe the application.

Press the Add Application button.Once the application is successfully created, you will be directed to the application detail page.

The Application ID is used by consuming applications to keep track of where it left off consuming messages from a topic.Consumers sharing the same ID share the events on a topic.So one event can be only read by one consumer with the same Application ID (read more about consumer groups).

Configuring Application Security/Authentication

There are two methods of to authenticating an application to use a topic.

SASL
SSL

SASL

SASL must be enabled as one of the Authentication methods on the Tenant and the Instance.

Application Credential

To generate a SASL credential for the selected environment:

Visit the detail page of the application.
Select the environment for which you want to generate the credentials.
Click on the lock button in the application box (hovering over it will show 'Authentication').
You will see a modal including the section below:
Click +Generate new Pair
A new Username Password Pair is shown
Copy the Username and Password and add them to your application

Copy the password and store it in a safe place as soon as it is displayed. It shall only be displayed right after the generation. Any later views of the Credentials will only show the Username.

You can generate as many credentials as you need

Delete Credentials

Click the Bin Icon to delete a set of credentials.

SASL Authentication Available on Multiple Clusters

For multiple clusters of the same provider type connected to an Instance in an Environment, one application credential is created and it is available to all the clusters of the Instance in the Environment. The clusters names which this credential is applied to, is displayed as shown below:

When no clusters are shown, it means that the SASL credential was applied to all clusters that were active at the time of creation.

Application Custom Principal

To use a SASL clientID on the selected environment:

Visit the detail page of the application.
Select the environment for which you want to use a clientID.
Click on the Padlock button in the application box (hovering over it will show 'Authentication').

You will see a modal including the section below:

Provide your ClientID registered in the IAM Provider
Click Apply

Use the custom principal in your application

You can add as many custom principals as you need

Delete Custom Principal

Click the Bin Icon to delete a custom principal.

SSL (Application Principals)

The application principal is used to authorize the application to use a topic (either producing or consuming) in an environment. The application principal consists of the Distinguished Name (DN) of the certificate used by the application, combined with the full chain of authority (ie. This starts with your application certificate, followed by the certificate of the entity that signed your certificate request, followed by the certificate of the entity that signed the previous one etc. This goes until it reaches the entity that is proclaimed as principal of the Instance you will run your application in).

The chain of signing authorities are used to allow the applications to access the brokers while the Distinguished Name (DN) name is used to verify the application’s access rights (ACL) to a particular topic. Both of these elements are required to make a successful connection to an Axual topic.

For every distinct environment, an application principal needs to be defined. A principal could be re-used, but we strongly advice to have a unique one per environment. Without a principal, an application cannot have consume/produce-access to a topic.

To configure the application principal on the selected environment:

Visit the detail page of the application.
Select the environment for which you want to configure the principal.
Click on the Padlock button in the application box (hovering over it will show 'Authentication'). You will see a modal as below:

Upload the PEM file of your application certificate, see Generating A Certificate PEM File
If the upload is successful, it’ll show the certificate chain starting from the subject to the different levels of signing authorities. Confirm that the uploaded certificate matches the certificate that you will use for your application in the corresponding environment.
Click on Apply.
The application box name will turn green indicating that the application is configured properly on that environment. Above steps can be repeated for different environments where application is required to access the topic.

If you are replacing an application certificate, you can register two principals at the same time, and choose to delete the old one once the application is not using it anymore.

Generating A Certificate PEM File

To reduce human error and to validate the uploaded certificates, we have opted to let the user upload a .pem file for his/her certificate-chain.

Most of the time, you are using a .jks file for your application already, so you can use the following instructions to extract a .pem from a .jks file, ready to be used in Self Service.

The following is taking into account that you are using a bash terminal and not ZSH or another custom terminal to run the following commands.

Export the .jks keystore into the PKCS12 format using Keytool, which generally comes with every JRE installment.

Be sure to replace yourAlias with the alias of your application certificate, sourceKeyStore with the path of your .jks file, and password with your own password.

keytool -importkeystore \
  -alias yourAlias \
  -srckeystore sourceKeyStore.jks \
  -srcstoretype jks \
  -srcstorepass password \
  -destkeystore destFile.p12 \
  -deststoretype PKCS12 \
  -deststorepass password

Retrieve the application certificate to the target .pem file, using the openssl command

openssl pkcs12 \
  -in destFile.p12 \
  -nokeys \
  -passin pass:password \
  -passout pass:password \
  | grep -v -e '^\s' | grep -v '^\(Bag\|subject\|issuer\)' > destKeyStore.pem

This results in a .pem file which can be uploaded as an application principal.

Update an expired certificate

When a user wants to update an existing certificate, they can access the Self-Service and add the new certificate alongside the expired certificate; then they can remove the expired certificate.

In Self-Service, the user can perform update and delete operations before pressing the apply button.

When the user is interacting with the PlatformManager over REST API calls, we advise to follow the following flow:

Add the new certificate (POST /application_principals)
Remove the expired certificate (DELETE /application_principals/{uid})

ACLs created in the Kafka topics do not take the expiry date into consideration.

Requesting Topic Access

Before your application can produce to, or consume from, a topic, a Topic Access Request has to be done. If you are making a request for an environment for testing purposes (private), an approval will not be necessary. For all other (public) environments, approval is necessary. The request is then received by the data owner of the topic.

Requesting access to a topic is straightforward.

Follow the steps below:

Before you request topic access, make sure you have configured the Application Principal or the Application Credentials for the specific environment in which you request access (see above)

Visit the detail page of your application.
Scroll to the bottom of the page where + Add request button is visible.
Click on the button to open the Request approval modal:
Select the Application type by clicking on Consumer or Producer button.
Select the Topic from the Topic dropdown. If there are already pending and/or approved requests for this topic a list of them will appear after selection.
Submit the request by clicking the Request approval button.

The data owner will receive your request and has to approve your ‘Topic Access Request’.

Resetting Consumer Offset

Resetting the consumer offset allows you to control the starting point for message consumption for a specific consumer group within a topic. This is especially useful for use cases such as reprocessing existing data or skipping to the latest available messages.

Reset Offset Options

You can reset consumer offsets to either the Latest or Earliest position:

Reset to Latest: This resets the consumer offset to the most recent position, effectively skipping all messages currently available in the topic. As a result, the consumer lag will be reduced to 0.
Reset to Earliest: This repositions the consumer offset to the beginning of the topic, allowing the consumer to reprocess all available messages from the topic. As a result, the consumer lag will be set to the total message count in the topic.

You may alternate between these steps as needed to adjust the consumer’s offset to suit your requirements.

Performing an Offset Reset

Requirements

This feature is valid only under the following conditions:

Only application owners are authorized to reset the consumer offset for their applications.
The application must not be of the types KSML, Self-Managed KSML or Kafka Streams.
The application has authentication configured and consume access for at least one topic.
The topic contains messages, which can be verified by browsing the topic.

Procedure

To reset the consumer offset for an application, navigate to the Applications page and select the application you want to manage. In the application detail, you can view the consumer lag number in records.

Stop the consumer application currently consuming from the topic.

Ensure that the consumer application currently consuming from the topic is stopped. If the consumer is active, the reset operation will fail as offsets cannot be modified for an active consumer group.

Click the Offset-Reset button on the application card
This opens a modal where you can configure the offset reset
- Application ID: Select the application id you used in your consumer application to consume the data
- Topic: Choose the topic for which the offset reset will be applied
- Reset offset to: Select either Latest or Earliest based on your requirements
Click Apply to execute the offset reset

Monitoring Changes After Reset

This offset reset operation takes place across all instance clusters.

After performing the reset, start your consumer application to resume data consumption.

For reset to latest: The consumer will start from the most recent messages in the topic.
For reset to earliest: The consumer will reprocess all messages from the beginning of the topic.

Deleting An Application

You can use Self Service Portal to delete your application.

An application can always be deleted; in case the application has authorization configured and it’s connected to topics, those resources will be deleted as well.

This operation might result in failing clients since the Kafka topic will be removed from the Kafka Cluster.

Visit the Application Detail page
Click Edit Application.
There is a Delete button on the bottom left of the page.
Click on that Delete button
1. If all constraints are met and deletion is possible will show a confirmation modal as below:
2. If not, a warning is shown with information that the application is actively mapped to topics, as below:
Clicking on Delete Application will delete the application.

Application visibility

By setting the application visibility, you control whether applications will end up in the Applications page for everyone, or is just visible to the owner.

Public: choose this option if you are working on a normal application that will be eventually deployed in production, and is not confidential in any way
Private: choose for private if this is a test application that will never be deployed in production or an application that should not end up in the Applications page.