Managing streams

Contents

Creating streams

Visit the Streams page and click on New Stream button. You will see the below page:

New stream form

  1. Start filling the form with required information. If you are unsure about certain values, please contact the team supporting the Axual Platform within your organization.

Make sure to be very descriptive when creating your stream, because that helps your colleagues finding your stream and implementing their data driven use case faster.

  • Name: The name of the stream. This must be in the format string-string (Needs to contain exactly one dash). The stream name is usually discussed and finalized as part of the Intake session or a follow up.

  • Description: A text describing the purpose of the stream.

  • Owner: The group which will own this stream. All members of that group will be able to modify the stream. This group should be created before creating the stream. See also: Creating A Group.

The list of groups shown are only the ones which the user is a member of. you cannot create a stream for a group that you are not part of

  • Key Type: The type of data that will be stored in the KEY part of the stream. This is usually AVRO.

  • Key Schema: If Key type is selected as AVRO, a new input field Key Schema will appear. This will have a list of available AVRO schemas. Choose the correct schema. As part of Schema design and development, these values would be known. If the required schema is not available, please contact support.

  • Value Type: The type of data that will be stored in the VALUE part of the stream. This is usually AVRO.

  • Value Schema: If Value type is selected as AVRO, a new input field Value Schema will appear. This will have a list of available AVRO schemas. Choose the correct schema. As part of Schema design and development, these values would be known. If the required schema is not available, please contact support.

  • Retention Policy: This field determines if the stream will retain messages forever or the messages will be deleted after a certain period. This would have been discussed in the Intake session.

    • Remove all messages when the retention period has passed: Choose this option if the messages should not be stored forever. The actual retention time will be configured later per environment.

    • Keep latest value for each key forever: Choose this option if messages should be retained forever. This is also called COMPACTION topic.

  • Properties: This is a set of key:value pair for setting various properties of a stream. For now this can be kept empty.

    1. Click on Create Stream. Once the stream is created, you will end up on Stream detail page.
      At this point, you have created a Stream in Self Service Portal. The stream is not ready for use yet. It needs to be configured for an environment.

Configuring A Stream For An Environment

Once the Stream has been created, the next step to actually configure the stream for any environment is to configure the stream.

  1. Visit the Stream page.

  2. Select the environment from the dropdown for which the stream needs to be configured

  3. If the stream has not yet been configured for the environment, a big bold + Configure Stream button will appear in the stream box. Click on it.

  4. A pop-up will show up with some information to be filled:

Configure stream modal

  1. Retention time (milliseconds): Determine how long the messages should be available on a stream. There should be an agreed value most likely discussed in Intake session with the team supporting Axual Platform. In most cases, it is 7 days.

  2. Key Schema Version: If the Stream Key Type is AVRO, then this field will show up. Choose the version of the key schema that should be used by the stream.

  3. Value Schema Version: If the Stream Value Type is AVRO, then this field will show up. Choose the version of the value schema that should be used by the stream.

  4. Number of partitions: Choose the number of partitions in which the stream must be split. If unsure, leave it empty. This values defines the maximum number of consumer instances that can read from the stream.

The number of partitions can not be changed using Self Service after a stream configuration has been saved, due to technical limitations. However, there is a workaround: delete the stream configuration and re-configure the stream in a particular environment with the correct number of partitions.

  1. Click on Save button. The stream is now configured for the specific environment. Repeat this process for any other environment.

Supported Kafka Properties

When you need fine-tuning your stream config you may want to use some kafka properties, find below the full list of kafka properties that we are currently supporting.

  • max.compaction.lag.ms

  • message.timestamp.difference.max.ms

  • message.timestamp.type

  • min.compaction.lag.ms

  • retention.bytes

  • segment.ms

This list of properties will always be updated with every release and is also present in the API documentation

Inherited Kafka Properties

These kafka properties might be defined in different resources in the Self-Service so the StreamConfig just inherits from a different resource.

  • cleanup.policy

  • min.insync.replicas

  • replication.factor

The replication.factor property defined the number of brokers on which a kafka message will be replicated, this property depends on the number of brokers available for the Tenant so the value used by the StreamConfig comes from the Tenant resource. Same for min.insync.replicas that is defined in the Tenant resource.

The cleanup.policy defines if the kafka topic is a compact or a delete topic, this property is defined in the Stream and all StreamConfig inherits from it.

Unsupported Kafka Properties

These kafka properties can not be changed, we used the broker default to have consistency in the platform.

  • compression.type

  • delete.retention.ms

  • file.delete.delay.ms

  • flush.messages

  • flush.ms

  • follower.replication.throttled.replicas

  • index.interval.bytes

  • leader.replication.throttled.replicas

  • message.downconversion.enable

  • message.format.version

  • min.cleanable.dirty.ratio

  • preallocate

  • segment.bytes

  • segment.index.bytes

  • segment.jitter.ms

  • unclean.leader.election.enable

Deleting Stream Configuration

Stream configuration can be deleted from the configure stream modal. This can be done if there are no active producer/consumer application connections in the chosen environment.

Make sure you don’t have active stream connections in the environment for which you are deleting the stream configuration

  1. Visit the Stream Detail page and click on Configure button inside the stream box. The stream configuration modal opens as below:

Stream configuration modal

  1. Click on the Delete Stream Configuration button on the bottom left of the modal. If all constraints are met and deletion is possible, a confirmation modal opens, as below:

Delete stream configuration confirm

  1. Click "Confirm" to delete the stream configuration.

When you delete a stream config, recreating it may be blocked for a period that is configured in the Instance properties. You can configure per instance how long you want to wait before recreating a stream config, use the property create-stream.disable-time (the value is in minutes), see Instance Properties.

If not set in the Instance, the default value comes from Management API that is 0; that means you can recreate a stream-config immediately.

If not all constraints are met, the Delete button is disabled and hovering on it shows a tooltip with a reason as shown below:

Inform stream configuration delete modal

Viewing and Downloading Schema

To view and download avro schemas used on a stream see *Viewing and Downloading Schemas used on a topic

Deleting A Stream

A stream can be deleted providing that the stream does not have any active stream configurations currently active on the stream itself.

  1. Navigate to the edit a stream page and then click the 'Delete Stream' Button

Note: The delete a stream button will only be visible to those who have the correct access which is typically a stream owner or administrator.

  1. Click Edit Stream

  2. Click Delete

Stream Delete Button

  1. Providing there are no active stream configuration present the following modal will be displayed:

Stream Delete Modal

Once you’ve confirmed that you would like to delete the stream it will be removed and no longer accessible by any application.