Managing topics

Contents

Creating topics

Visit the Topics page and click on New Topic button. You will see the below page:

New topic form

Fill the form with required information. If you are unsure about certain values, please contact the team supporting the Axual Platform within your organization.

Make sure to be very descriptive when creating your topic, because that helps your colleagues finding your topic and implementing their data driven use case faster.

  • Name: The name of the topic. The topic name is usually discussed and finalized as part of the Intake session or a follow-up.

  • Description: Text describing the purpose of the topic.

  • Owner: The group which will own this topic. All members of that group will be able to modify the topic. This group should be created before creating the topic. See also: Creating A Group.

The list of groups shown are only the ones which the user is a member of. you cannot create a topic for a group that you are not part of

  • Key Type: The type of data that will be stored in the KEY part of the topic.

  • Key Schema: If Key type is selected as AVRO, Protobuf or JSON Schema, a new input field Key Schema will appear. This will have a list of available schemas. Choose the correct schema. As part of Schema design and development, these values would be known. If the required schema is not available, please contact support.

  • Value Type: The type of data that will be stored in the VALUE part of the topic.

  • Value Schema: If Value type is selected as AVRO, Protobuf or JSON Schema, a new input field Value Schema will appear. This will have a list of available schemas. Choose the correct schema. As part of Schema design and development, these values would be known. If the required schema is not available, please contact support.

  • Retention Policy: This field determines if the topic will retain messages forever or the messages will be deleted after a certain period. This would have been discussed in the Intake session.

    • Remove all messages when the retention period has passed: Choose this option if the messages should not be stored forever. The actual retention time will be configured later per environment.

    • Keep the latest value for each key forever: Choose this option if messages should be retained forever. This is also called COMPACTION topic.

  • Properties: This is a set of key:value pairs for setting various properties of a topic. For now this can be kept empty.

    1. Click on Create Topic. Once the topic is created, you will end up on Topic detail page.
      At this point, you have created a topic in Self Service Portal. The topic is not ready for use yet. It needs to be configured for an environment.

  • Viewer Groups: (Optional) Choose the groups which would be able to view this topic configuration. Please refer to Viewer Groups section.

Configuring A Topic For An Environment

Once the Topic has been created, the next step to actually configure the topic for any environment is to configure the topic.

  1. Visit the Topic page

  2. Select the environment from the dropdown for which the topic needs to be configured

  3. If the topic has not yet been configured for the environment, a blue circle will appear on the gear icon in the topic box. Click on it

  4. A pop-up will show up with some information to be filled:

Configure topic modal

  1. Retention time: Determines how long the messages should be available on a topic. There should be an agreed value most likely discussed in Intake session with the team supporting Axual Platform. In most cases, it is 7 days.

  2. Key Schema Version: If the Topic Key Type is AVRO, Protobuf or JSON Schema, then this field will show up. Choose the version of the key schema that should be used by the topic.

  3. Value Schema Version: If the Topic Value Type is AVRO, Protobuf or JSON Schema, then this field will show up. Choose the version of the value schema that should be used by the topic.

  4. Number of partitions: Choose the number of partitions in which the topic will be broken down and distributed over the brokers. This value defines the maximum number of consumer instances that can read from the topic.

The number of partitions cannot be changed using Self Service after a topic configuration has been saved, due to technical limitations. However, there is a workaround: delete the topic configuration and re-configure the topic in a particular environment with the correct number of partitions.

  1. Click on Save button. The topic is now configured for the specific environment. Repeat this process for any other environment.

Supported Kafka Properties

When you need fine-tuning your topic config you may want to use some kafka properties, find below the full list of kafka properties that we are currently supporting.

  • max.compaction.lag.ms

  • message.timestamp.difference.max.ms

  • message.timestamp.type

  • min.compaction.lag.ms

  • retention.bytes

  • segment.ms

This list of properties will always be updated with every release and is also present in the API documentation

Inherited Kafka Properties

These kafka properties might be defined in different resources in the Self-Service so the TopicConfig just inherits from a different resource.

  • cleanup.policy

  • min.insync.replicas

  • replication.factor

The replication.factor property defined the number of brokers on which a kafka message will be replicated, this property depends on the number of brokers available for the Tenant so the value used by the TopicConfig comes from the Tenant resource. Same for min.insync.replicas that is defined in the Tenant resource.

The cleanup.policy defines if the kafka topic is a compact or a delete topic, this property is defined in the Topic and all TopicConfig inherits from it.

Unsupported Kafka Properties

These kafka properties cannot be changed, we used the broker default to have consistency in the platform.

  • compression.type

  • delete.retention.ms

  • file.delete.delay.ms

  • flush.messages

  • flush.ms

  • follower.replication.throttled.replicas

  • index.interval.bytes

  • leader.replication.throttled.replicas

  • message.downconversion.enable

  • message.format.version

  • min.cleanable.dirty.ratio

  • preallocate

  • segment.bytes

  • segment.index.bytes

  • segment.jitter.ms

  • unclean.leader.election.enable

Deleting Topic Configuration

Topic configuration can be deleted from the Configure Topic modal.

This can be done if there are no active producer/consumer application connections in the chosen environment.

Make sure you don’t have active topic connections in the environment for which you are deleting the topic configuration

  1. Visit the Topic Detail page and click on Configure button inside the topic box. The Configure Topic modal opens as below:

topic configuration modal

  1. Hover the Delete Topic Configuration button on the bottom left of the modal.

    1. If not all constraints are met, the Delete button is disabled and hovering on it shows a tooltip with a reason as shown below:

      Inform topic configuration delete modal

    2. If all constraints are met and deletion is possible, the button will be active.

      Delete topic configuration confirm

  1. Click "Confirm" to delete the topic configuration.

When you delete a topic config, recreating it may be blocked for a period configured in the Instance properties. You can configure per instance how long you want to wait before recreating a Topic Configuration, use the property create-topic.disable-time (the value is in minutes), see Instance Properties.

If not set in the Instance, the default value comes from Management API that is 0; that means you can recreate a Topic Configuration immediately.

Viewing and Downloading Schema

To view and download schemas used on a topic see Viewing and Downloading Schemas used on a topic

Viewing and Downloading AsyncAPI Spec

You can view and download the generated AsyncAPI spec for a configured Topic, in an environment.

To view and download the generated AsyncAPI spec

  1. Navigate to the Topic Detail page.

  2. On the Topic Card, click on the AsyncAPI icon (s-shape icon)

  3. A modal will show the generated AsyncAPI spec.

  4. A download button is visible at the bottom right of the modal. Click on the button to download.

Topic AsyncAPI Spec

Deleting A Topic

A topic can always be deleted; in case the topic has deployed topic configurations, those resources will be deleted as well.

This operation might result in failing clients since the Kafka topic will be removed from the Kafka Cluster.

  1. Navigate to topic page, press the Edit Topic button and then click the Delete Button

Note: The delete a topic button will only be visible to those who have the correct access which is typically a topic owner or administrator.

  1. Click Edit Topic button

  2. Click Delete button

Topic Delete Button

  1. If there are deployed Topic Configurations, the following modal will be displayed:

Topic Delete Modal

Once you have confirmed that you would like to delete the Topic, it will be removed and no longer accessible by any Application.