Managing streams

Contents

Creating A Stream

Visit the Streams page and click on New Stream button. You will see the below page:

New stream form
  1. Start filling the form with required information. If you are unsure about certain values, please contact the team supporting the Axual Platform within your organization.

Make sure to be very descriptive when creating your stream, because that helps your colleagues finding your stream and implementing their data driven use case faster.
  • Name: The name of the stream. This must be in the format string-string (Needs to contain exactly one dash). The stream name is usually discussed and finalized as part of the Intake session or a follow up.

  • Description: A text describing the purpose of the stream.

  • Owner: The group which will own this stream. All members of that group will be able to modify the stream. This group should be created before creating the stream. See also: Creating A Group.

  • Key Type: The type of data that will be stored in the KEY part of the stream. This is usually AVRO.

  • Key Schema: If Key type is selected as AVRO, a new input field Key Schema will appear. This will have a list of available AVRO schemas. Choose the correct schema. As part of Schema design and development, these values would be known. If the required schema is not available, please contact support.

  • Value Type: The type of data that will be stored in the VALUE part of the stream. This is usually AVRO.

  • Value Schema: If Value type is selected as AVRO, a new input field Value Schema will appear. This will have a list of available AVRO schemas. Choose the correct schema. As part of Schema design and development, these values would be known. If the required schema is not available, please contact support.

  • Retention Policy: This field determines if the stream will retain messages forever or the messages will be deleted after a certain period. This would have been discussed in the Intake session.

    • Remove all messages when the retention period has passed: Choose this option if the messages should not be stored forever. The actual retention time will be configured later per environment.

    • Keep latest value for each key forever: Choose this option if messages should be retained forever. This is also called COMPACTION topic.

  • Properties: This is a set of key:value pair for setting various properties of a stream. For now this can be kept empty.

  1. Click on Create Stream. Once the stream is created, you will end up on Stream detail page.
    At this point, you have created a Stream in Self Service Portal. The stream is not ready for use yet. It needs to be configured for an environment.

Configuring A Stream For An Environment

Once the Stream has been created, the next step to actually configure the stream for any environment is to configure the stream.

  1. Visit the Stream page.

  2. Select the environment from the dropdown for which the stream needs to be configured

  3. If the stream has not yet been configured for the environment, a big bold + Configure Stream button will appear in the stream box. Click on it.

  4. A pop up will show up with some information to be filled:

Configure stream modal
  1. Retention time (milliseconds): Determine how long the messages should be available on a stream. There should be an agreed value most likely discussed in Intake session with the team supporting Axual Platform. In most cases, it is 7 days.

  2. Key Schema Version: If the Stream Key Type is AVRO, then this field will show up. Choose the version of the key schema that should be used by the stream.

  3. Value Schema Version: If the Stream Value Type is AVRO, then this field will show up. Choose the version of the value schema that should be used by the stream.

  4. Number of partitions: Choose the number of partitions in which the stream must be split. If unsure, leave it empty. This values defines the maximum number of consumer instances that can read from the stream.

The number of partitions can not be changed using Self Service after a stream configuration has been saved, due to technical limitations.
  1. Click on Save button. The stream is now configured for the specific environment. Repeat this process for any other environment.

Deleting Stream Configuration

Stream configuration can be deleted from the configure stream modal. This can be done if there are no active producer/consumer application connections in the chosen environment.

make sure you don’t have active stream connections in the environment for which you are deleting the stream configuration
  1. Visit the Stream Detail page and click on Configure button inside the stream box. The stream configuration modal opens as below:

Stream configuration modal
  1. Click on the Delete Stream Configuration button on the bottom left of the modal. If all constraints are met and deletion is possible, a confirmation modal opens, as below:

Delete stream configuration confirm
  1. Click "Confirm" to delete the stream configuration.

Inform stream configuration delete modal

Deleting A Stream

A stream can be deleted providing that the stream does not have any active stream configurations currently active on the stream itself.

  1. Navigate to the edit a stream page and then clickthe 'Delete Stream' Button

Note: The delete a stream button will only be visible to those who have the correct access which is typically a stream owner or administrator.

  1. Click "Edit a Stream"

  2. Click "Delete Stream"

Stream Delete Button
  1. Providing there are no active stream configuration present the following modal will be displayed:

Stream Delete Modal

Once you’ve confirmed that you would like to delete the stream it will be removed and no longer accessible by any application.

Authorizing Stream Access

Authorizing a stream access request can be done both through the dashboard and the stream detail page:

  1. Visit the dashboard page.

  2. Switch to the "Action needed" status

  3. Find the application that has requested access to the stream. Also verify the environment where request needs to be approved.

Approve request
  1. Click the Approve request button. A confirmation popup appears verifying that the request was approved.

  2. The application ACL will be authorized to allow access to your stream.

Authorizing an application to access a stream can also be done from the stream’s detail page:

  1. Search for your stream and visit the stream detail page.

  2. Scroll down to the section with a tab titled Pending streams.

Approve request
  1. Find the application that has requested access to the stream. Also verify the environment where request needs to be approved.

  2. Click the Approve request button. A confirmation popup appears verifying that the request was approved.

  3. The application ACL will be authorized in Axual Cloud cluster to allow access to your stream.

  4. The application request entry should move to the next tab titled History.

  5. The application color in the visualization also changes to green.

Deny Authorization To An Application To Access A Stream

Please refer to the above section Authorizing Stream Access. Follow the same steps up to step 4 and then instead of Approve request button click on Deny button, and provide a reason for the requestor.

Revoking An Application Access From Your Stream

In order to revoke access for a certain application in a certain environment, use the garbage icon next to the application that is connected to the stream.

  1. Click it to open the "Revoke stream access" modal.

Revoking stream access
  1. Select the environments for which you want to revoke access and provide a reason, followed by clicking "Save" to revoke access to the selected environments.

Adding a reason and selecting an environment when revoking access
  1. Once the action has completed, you can close the modal.