Getting Started

Where to start

Now that you understand the purpose of Self Service and you know where to find where, you want to get started. Depending on your situation, you either want to start from scratch with a new stream as indicated in the image below, or you want to consume the data on an existing stream.

I want to produce to or consume from an existing stream

You are: an Application Owner / DevOps team member.
Situation: You are currently developing a producer or consumer application that will use an existing stream.

  1. Create an application in Self Service that represents your actual producer/consumer application (which is deployed somewhere)

  2. Configure application principals for your application, so it can be authorized to produce to or consume from the stream. Note: you need to do this for every environment the application is running.

  3. Request stream access, so your application can actually produce or consume.

I want to produce to a new stream

You are: a Producer Application Owner
Situation: You are currently developing, or you have already developed, a producer application that will produce to a new stream.

OR

You are: a Stream Owner / Data Owner
Situation: You want to create a new stream (topic) for future producers and consumers.

Before creating a new stream (topic) most likely a schema must be discussed and finalized. This is currently outside the scope of Self Service. Once a schema has been developed and available for use, you can proceed with below steps to create a new stream in Self-Service.

  1. Create a Stream in Self Service, with all the required information

  2. Request produce access to the stream, as mentioned above.

I Want To Test My Application/Stream In My Own Environment

You are: an Application Owner / DevOps Team Member
Situation: You want to have a private environment where you can access a stream with exclusive access to your application, for testing purposes.

For this purpose, you need to Create a private environment

I Want To Authorize Stream Access To An Application

You are: a Stream Owner / Data Owner
Situation: You want to authorize an application to produce / consume your stream (topic).

Refer to Authorizing stream access

I Want To Revoke Stream Access For An Application

You are: a Stream Owner / Data Owner
Situation: You want to revoke an application’s access to produce / consume from your stream.

See here how to do that.

I already have set up an application principal, why do I need a keystore/truststore with my application?

You are: an Application Owner / DevOps Team Member
Situation: You want to set up a connection with a Broker

This is required due to the way mTLS is set up.

  • The keystore/truststore is used for the “authentication” phase, the mTLS layer (provided by the JVM running Kafka Broker), which checks key validity period and whether the “Signed by” entry is in the truststore on the Broker side, and vice versa of course.

  • What you upload in Self Service will be used by the Broker itself for the “authorisation” phase, and uses a Quorum that holds ACLs to check whether the provided certificate in the SSL context, combined with the topic it requests access too, is valid or not.

Is there a maximum to how many messages per second / hour / day can be processed?

You are: Anyone
Situation: You want to scale your Kafka clusters

Short answer: YES

Long answer: Yes, but it takes a while to reach. There are various criteria to the use case, but the most important ones are these:

  • Cost

  • Available bandwidth / Networking speed for both Broker and producer/consumer

  • Available CPU and memory to both Broker and producer/consumer

  • Average message-size and/or batch-size

  • Choice between throughput or latency

  • Producer client acknowledgement. Does a client want to know:

    1. a message has been sent to Broker;

    2. a message has been received by Broker;

    3. a message has been successfully submitted by Broker and its replicas.

  • Writing/Reading speeds of the disk(s) in the Broker

If you have no value for retention, you might be able to mount a RAMdisk on the Broker and send millions of messages of 10 bytes each second without any confirmation by Broker and a retention of 4 seconds. But in general you will want at least some form of confirmation that Broker has received your message, as well as some hours of retention to be able to mitigate downtime from any component. This means that HDD/SSD disks come into play, lowering transmission speeds by definition. Or it becomes REALLY expensive, real quick.