Getting Started

Where to start

Now that you understand the purpose of Self Service and you know where to find where, you want to get started. Depending on your situation, you either want to start from scratch with a new stream as indicated in the image below, or you want to consume the data on an existing stream.

I want to produce to or consume from an existing stream

You are: an Application Owner / DevOps team member.
Situation: You are currently developing a producer or consumer application that will use an existing stream.

  1. Create an application in Self Service that represents your actual producer/consumer application (which is deployed somewhere)

  2. Configure application principals for your application, so it can be authorized to produce to or consume from the stream. Note: you need to do this for every environment the application is running.

  3. Request stream access, so your application can actually produce or consume.

I want to produce to a new stream

You are: a Producer Application Owner
Situation: You are currently developing, or you have already developed, a producer application that will produce to a new stream.

OR

You are: a Stream Owner / Data Owner
Situation: You want to create a new stream (topic) for future producers and consumers.

Before creating a new stream (topic) most likely a schema must be discussed and finalized. This is currently outside the scope of Self Service. Once a schema has been developed and available for use, you can proceed with below steps to create a new stream in Self-Service.

  1. Create a Stream in Self Service, with all the required information

  2. Request produce access to the stream, as mentioned above.

I Want To Test My Application/Stream In My Own Environment

You are: an Application Owner / DevOps Team Member
Situation: You want to have a private environment where you can access a stream with exclusive access to your application, for testing purposes.

For this purpose, you need to Create a private environment

I Want To Authorize Stream Access To An Application

You are: a Stream Owner / Data Owner
Situation: You want to authorize an application to produce / consume your stream (topic).

I Want To Revoke Stream Access For An Application

You are: a Stream Owner / Data Owner
Situation: You want to revoke an application’s access to produce / consume from your stream.

See here how to do that.

I already have set up an application principal, why do I need a keystore/truststore with my application?

You are: an Application Owner / DevOps Team Member
Situation: You want to set up a connection with a Broker

This is required due to the way mTLS is set up.

  • The keystore/truststore is used for the “authentication” phase, the mTLS layer (provided by the JVM running Kafka Broker), which checks key validity period and whether the “Signed by” entry is in the truststore on the Broker side, and vice versa of course.

  • What you upload in Self Service will be used by the Broker itself for the “authorisation” phase, and uses a Quorum that holds ACLs to check whether the provided certificate in the SSL context, combined with the topic it requests access too, is valid or not.

How do I convert my java application that uses JKS to use in memory PEM instead?

You are: a Java Application Developer
Situation: You want to supply application keystore and truststore as PEM strings instead of using JKS files

In your installation you might want to avoid storing certificate files on a server, you can instead provide your certificate in string format which you can then plug into your SSL Configuration.

You would have to extract the entire chain from your keystore (similarly to how it was done at the time the application was registered on the Self Service UI).

The following is assuming that you are using a bash terminal and not ZSH or another custom terminal to run the commands.
  1. Export the application .jks keystore into the PKCS12 format using keytool, which generally comes with every JRE installation.

    keytool -importkeystore \
      -srckeystore srcKeystore.jks \
      -srcstoretype jks \
      -srcstorepass srcKeystorePassword \
      -destkeystore destKeystore.p12 \
      -deststoretype PKCS12 \
      -deststorepass destpwd
  2. Do the same for the application truststore

    keytool -importkeystore \
      -srckeystore srcTruststore.jks \
      -srcstoretype jks \
      -srcstorepass srcTruststorePassword \
      -destkeystore destTruststore.p12 \
      -deststoretype PKCS12 \
      -deststorepass destpwd

    You have now converted both the keystore and truststore in PKCS12 format, as an intermediate step.

  3. Retrieve the application certificate and print as a PEM string, using the openssl command

    openssl pkcs12 \
      -in destKeystore.p12 \
      -nokeys \
      -passin pass:destpwd \
      -passout pass:destpwd \
      | grep -v -e '^\s' | grep -v '^\(Bag\|subject\|issuer\)' | sed -e 's/$/\\n/' | tr -d '\r\n'

    This will print the keystore in PEM format on the console, in a single line with newline characters, so you can use it in your client configuration right away.

  4. Do exactly the same, but now for the application truststore

    openssl pkcs12 \
      -in destTruststore.p12 \
      -nokeys \
      -passin pass:destpwd \
      -passout pass:destpwd \
      | grep -v -e '^\s' | grep -v '^\(Bag\|subject\|issuer\)' | sed -e 's/$/\\n/' | tr -d '\r\n'
  5. Next, Export your private key as a PEM string

    openssl pkcs12 \
      -in destKeystore.p12 \
      -nocerts \
      -nodep \
      -passin pass:destpwd \
      -passout pass:destpwd \
      | grep -v -e '^\s' | grep -v '^\(Bag\|Key\)' | sed -e 's/$/\\n/' | tr -d '\r\n'

    This will print the private key in PEM format on the console, in a single line with newline characters

    You have now prepared all PEM strings which you can use in the SslConfig of your application.

  6. Use the provided builder, to set keystoreCertificateChain, keystoreKey and truststoreCertificates in SslConfig and copy/paste the strings as generated with the commands above

    SslConfig.newBuilder()
    	.setKeystoreType(SslConfig.KeystoreType.PEM)
    	.setKeystoreCertificateChain("----BEGIN CERTIFICATE----\n... ")
    	.setKeystoreKey("----BEGIN PRIVATE KEY----\n... ")
    	.setTruststoreType(SslConfig.TruststoreType.PEM)
    	.setTruststoreCertificates("----BEGIN CERTIFICATE----\n... ")
    	.build()

Is there a maximum to how many messages per second / hour / day can be processed?

Short answer: YES

Long answer: Yes, but it takes a while to reach. There are various criteria to the use case, but the most important ones are these:

  • Cost

  • Available bandwidth / Networking speed for both Broker and producer/consumer

  • Available CPU and memory to both Broker and producer/consumer

  • Average message-size and/or batch-size

  • Choice between throughput or latency

  • Producer client acknowledgement. Does a client want to know:

    1. a message has been sent to Broker;

    2. a message has been received by Broker;

    3. a message has been successfully submitted by Broker and its replicas.

  • Writing/Reading speeds of the disk(s) in the Broker

If you have no value for retention, you might be able to mount a RAMdisk on the Broker and send millions of messages of 10 bytes each second without any confirmation by Broker and a retention of 4 seconds. But in general you will want at least some form of confirmation that Broker has received your message, as well as some hours of retention to be able to mitigate downtime from any component. This means that HDD/SSD disks come into play, lowering transmission speeds by definition. Or it becomes REALLY expensive, real quick.

UI is not showing anything

In case your UI is not showing anything, check the console error of browser. There should be an error like:

CSP invalid.

In this case, copy the SHA value from the console and add it to /clusters/local/mgmt-ui.sh inside MGMT_UI_SMALLCHAT_SHA variable.