Managing topics

Contents

Creating topics

Visit the Topics page and click on New Topic button. You will see the below page:

New stream form

  1. Start filling the form with required information. If you are unsure about certain values, please contact the team supporting the Axual Platform within your organization.

Make sure to be very descriptive when creating your stream, because that helps your colleagues finding your stream and implementing their data driven use case faster.

  • Name: The name of the topic. This must be in the format string-string (Needs to contain exactly one dash). The stream name is usually discussed and finalized as part of the Intake session or a follow up.

  • Description: A text describing the purpose of the topic.

  • Owner: The group which will own this topic. All members of that group will be able to modify the topic. This group should be created before creating the topic. See also: Creating A Group.

The list of groups shown are only the ones which the user is a member of. you cannot create a stream for a group that you are not part of

  • Key Type: The type of data that will be stored in the KEY part of the topic. This is usually AVRO.

  • Key Schema: If Key type is selected as AVRO, a new input field Key Schema will appear. This will have a list of available AVRO schemas. Choose the correct schema. As part of Schema design and development, these values would be known. If the required schema is not available, please contact support.

  • Value Type: The type of data that will be stored in the VALUE part of the topic. This is usually AVRO.

  • Value Schema: If Value type is selected as AVRO, a new input field Value Schema will appear. This will have a list of available AVRO schemas. Choose the correct schema. As part of Schema design and development, these values would be known. If the required schema is not available, please contact support.

  • Retention Policy: This field determines if the stream will retain messages forever or the messages will be deleted after a certain period. This would have been discussed in the Intake session.

    • Remove all messages when the retention period has passed: Choose this option if the messages should not be stored forever. The actual retention time will be configured later per environment.

    • Keep latest value for each key forever: Choose this option if messages should be retained forever. This is also called COMPACTION topic.

  • Properties: This is a set of key:value pair for setting various properties of a topic. For now this can be kept empty.

    1. Click on Create Topic. Once the stream is created, you will end up on Topic detail page.
      At this point, you have created a Topic in Self Service Portal. The stream is not ready for use yet. It needs to be configured for an environment.

Configuring A Topic For An Environment

Once the Topic has been created, the next step to actually configure the stream for any environment is to configure the topic.

  1. Visit the Topic page.

  2. Select the environment from the dropdown for which the stream needs to be configured

  3. If the stream has not yet been configured for the environment, a big bold + Configure Topic button will appear in the stream box. Click on it.

  4. A pop-up will show up with some information to be filled:

Configure stream modal

  1. Retention time (milliseconds): Determine how long the messages should be available on a topic. There should be an agreed value most likely discussed in Intake session with the team supporting Axual Platform. In most cases, it is 7 days.

  2. Key Schema Version: If the Topic Key Type is AVRO, then this field will show up. Choose the version of the key schema that should be used by the topic.

  3. Value Schema Version: If the Topic Value Type is AVRO, then this field will show up. Choose the version of the value schema that should be used by the topic.

  4. Number of partitions: Choose the number of partitions in which the stream must be split. If unsure, leave it empty. This values defines the maximum number of consumer instances that can read from the topic.

The number of partitions can not be changed using Self Service after a stream configuration has been saved, due to technical limitations. However, there is a workaround: delete the topic configuration and re-configure the topic in a particular environment with the correct number of partitions.

  1. Click on Save button. The stream is now configured for the specific environment. Repeat this process for any other environment.

Supported Kafka Properties

When you need fine-tuning your stream config you may want to use some kafka properties, find below the full list of kafka properties that we are currently supporting.

  • max.compaction.lag.ms

  • message.timestamp.difference.max.ms

  • message.timestamp.type

  • min.compaction.lag.ms

  • retention.bytes

  • segment.ms

This list of properties will always be updated with every release and is also present in the API documentation

Inherited Kafka Properties

These kafka properties might be defined in different resources in the Self-Service so the TopicConfig just inherits from a different resource.

  • cleanup.policy

  • min.insync.replicas

  • replication.factor

The replication.factor property defined the number of brokers on which a kafka message will be replicated, this property depends on the number of brokers available for the Tenant so the value used by the TopicConfig comes from the Tenant resource. Same for min.insync.replicas that is defined in the Tenant resource.

The cleanup.policy defines if the kafka topic is a compact or a delete topic, this property is defined in the Topic and all TopicConfig inherits from it.

Unsupported Kafka Properties

These kafka properties can not be changed, we used the broker default to have consistency in the platform.

  • compression.type

  • delete.retention.ms

  • file.delete.delay.ms

  • flush.messages

  • flush.ms

  • follower.replication.throttled.replicas

  • index.interval.bytes

  • leader.replication.throttled.replicas

  • message.downconversion.enable

  • message.format.version

  • min.cleanable.dirty.ratio

  • preallocate

  • segment.bytes

  • segment.index.bytes

  • segment.jitter.ms

  • unclean.leader.election.enable

Viewing and Downloading Schema

To view and download avro schemas used on a stream see Viewing and Downloading Schemas used on a topic

Viewing Topic Messages

When you are producing data on a stream and you would like to see whether messages have ended up on the platform correctly, you can use Browse & Search. This functionality is found on the Topic Detail in the top right navigation.

Topic Messages Navigation

  1. Open the Topic Detail page

  2. Click Browse & Search in the top right. The following screen will be shown

    Topic Messages Controls

  3. Select the environment on which you would like to search

  4. Enter a keyword (optional)

  5. Narrow down the search to a particular time window using Timestamp from (default: now - 1h) and Timestamp to (default: now)

  6. Choose how many results you want to see per page, by choosing a Page size

  7. Click Search to execute the search

Stream Browse search controls
If you don’t have privileges to search on these topics' messages, the Search button will be greyed out.

Providing there are messages available you will then be shown your results in a paginated table view such as below.

Search results

Every row shows:

  1. Technical message details: technical details about the message, see also Technical message details

  2. Key: the contents of the message Key (abbreviated with an ellipsis)

  3. Value: the contents of the message Value (abbreviated with an ellipsis)

Click a row to expand the view and open a viewer that shows the full message details, for which you can expand or collapse individual fields.

Search result (row) expanded

Pagination

Use the pagination controls in the bottom to jump to individual pages or use the arrow buttons to traverse search results. You can adjust the Page size before you search, to show more results per page.

Technical message details

Every row shows some technical details, which can be handy for debugging purposes:

  1. Partition: the stream’s partition the message is stored on

  2. Offset: the sequence number of the message on the partition

  3. timestamp: the producer timestamp

Deleting Topic Configuration

Topic configuration can be deleted from the configure stream modal. This can be done if there are no active producer/consumer application connections in the chosen environment.

Make sure you don’t have active topic connections in the environment for which you are deleting the stream configuration

  1. Visit the Topic Detail page and click on Configure button inside the topic box. The topic configuration modal opens as below:

Topic configuration modal

  1. Click on the Delete Topic Configuration button on the bottom left of the modal. If all constraints are met and deletion is possible, a confirmation modal opens, as below:

Delete topic configuration confirm

  1. Click "Confirm" to delete the stream configuration.

When you delete a stream config, recreating it may be blocked for a period that is configured in the Instance properties. You can configure per instance how long you want to wait before recreating a stream config, use the property create-stream.disable-time (the value is in minutes), see Instance Properties.

If not set in the Instance, the default value comes from Management API that is 0; that means you can recreate a stream-config immediately.

If not all constraints are met, an info modal opens as below:

Inform stream configuration delete modal

Deleting A Topic

A stream can be deleted providing that the stream does not have any active stream configurations currently active on the stream itself.

  1. Navigate to the edit a stream page and then click the 'Delete Topic' Button

Note: The delete a stream button will only be visible to those who have the correct access which is typically a stream owner or administrator.

  1. Click "Edit a Topic"

  2. Click "Delete Topic"

Topic Delete Button

  1. Providing there are no active stream configuration present the following modal will be displayed:

Topic Delete Modal

Once you’ve confirmed that you would like to delete the stream it will be removed and no longer accessible by any application.