Getting Started
Where to start
Now that you understand the purpose of Self Service and you know where to find what, you want to get started. Depending on your situation, you either want to start from scratch with a new stream as indicated in the image below, or you want to consume the data on an existing stream.
How do I produce to or consume from an existing stream?
-
Create an application in Self Service that represents your actual producer/consumer application (which is deployed somewhere)
-
Configure application principals for your application, so it can be authorized to produce to or consume from the stream. Note: you need to do this for every environment the application is running.
-
Request stream access, so your application can actually produce or consume.
How do I produce to a new stream?
OR
Before creating a new stream (topic) most likely a schema must be discussed and finalized. This is currently outside the scope of Self Service. Once a schema has been developed and available for use, you can proceed with below steps to create a new stream in Self-Service.
-
Create a Stream in Self Service, with all the required information
-
Request produce access to the stream, as mentioned above.
How do I test my streaming application in my own environment?
For this purpose, you need to Create a private environment
How do I authorize an application to access my stream?
Refer to Authorizing stream access
How do I revoke application authorizations for a stream?
See here how to do that.
I already have set up an application principal, why do I need a keystore/truststore with my application?
This is required due to the way mTLS is set up.
-
The keystore/truststore is used for the “authentication” phase, the mTLS layer (provided by the JVM running Kafka Broker), which checks key validity period and whether the “Signed by” entry is in the truststore on the Broker side, and vice versa of course.
-
What you upload in Self Service will be used by the Broker itself for the “authorisation” phase, and uses a Quorum that holds ACLs to check whether the provided certificate in the SSL context, combined with the topic it requests access too, is valid or not.
How do I convert my java application that uses JKS to use in memory PEM instead?
In your installation you might want to avoid storing certificate files on a server, you can instead provide your certificate in string format which you can then plug into your SSL Configuration.
You would have to extract the entire chain from your keystore (similarly to how it was done at the time the application was registered on the Self Service UI).
The following is assuming that you are using a bash terminal and not ZSH or another custom terminal to run the commands. |
-
Export the application .jks keystore into the PKCS12 format using
keytool
, which generally comes with every JRE installation.keytool -importkeystore \ -srckeystore srcKeystore.jks \ -srcstoretype jks \ -srcstorepass srcKeystorePassword \ -destkeystore destKeystore.p12 \ -deststoretype PKCS12 \ -deststorepass destpwd
-
Do the same for the application truststore
keytool -importkeystore \ -srckeystore srcTruststore.jks \ -srcstoretype jks \ -srcstorepass srcTruststorePassword \ -destkeystore destTruststore.p12 \ -deststoretype PKCS12 \ -deststorepass destpwd
You have now converted both the keystore and truststore in
PKCS12
format, as an intermediate step. -
Retrieve the application certificate and print as a PEM string, using the openssl command
openssl pkcs12 \ -in destKeystore.p12 \ -nokeys \ -passin pass:destpwd \ -passout pass:destpwd \ | grep -v -e '^\s' | grep -v '^\(Bag\|subject\|issuer\)' | sed -e 's/$/\\n/' | tr -d '\r\n'
This will print the keystore in PEM format on the console, in a single line with newline characters, so you can use it in your client configuration right away.
-
Do exactly the same, but now for the application truststore
openssl pkcs12 \ -in destTruststore.p12 \ -nokeys \ -passin pass:destpwd \ -passout pass:destpwd \ | grep -v -e '^\s' | grep -v '^\(Bag\|subject\|issuer\)' | sed -e 's/$/\\n/' | tr -d '\r\n'
-
Next, Export your private key as a PEM string
openssl pkcs12 \ -in destKeystore.p12 \ -nocerts \ -nodep \ -passin pass:destpwd \ -passout pass:destpwd \ | grep -v -e '^\s' | grep -v '^\(Bag\|Key\)' | sed -e 's/$/\\n/' | tr -d '\r\n'
This will print the private key in PEM format on the console, in a single line with newline characters
You have now prepared all PEM strings which you can use in the
SslConfig
of your application. -
Use the provided builder, to set
keystoreCertificateChain
,keystoreKey
andtruststoreCertificates
inSslConfig
and copy/paste the strings as generated with the commands aboveSslConfig.newBuilder() .setKeystoreType(SslConfig.KeystoreType.PEM) .setKeystoreCertificateChain("----BEGIN CERTIFICATE----\n... ") .setKeystoreKey("----BEGIN PRIVATE KEY----\n... ") .setTruststoreType(SslConfig.TruststoreType.PEM) .setTruststoreCertificates("----BEGIN CERTIFICATE----\n... ") .build()
Is there a maximum to how many messages per second / hour / day can be processed?
Short answer: YES
Long answer: Yes, but it takes a while to reach. There are various criteria to the use case, but the most important ones are these:
-
Cost
-
Available bandwidth / Networking speed for both Broker and producer/consumer
-
Available CPU and memory to both Broker and producer/consumer
-
Average message-size and/or batch-size
-
Choice between throughput or latency
-
Producer client acknowledgement. Does a client want to know:
-
a message has been sent to Broker;
-
a message has been received by Broker;
-
a message has been successfully submitted by Broker and its replicas.
-
-
Writing/Reading speeds of the disk(s) in the Broker
If you have no value for retention, you might be able to mount a RAMdisk on the Broker and send millions of messages of 10 bytes each second without any confirmation by Broker and a retention of 4 seconds. But in general you will want at least some form of confirmation that Broker has received your message, as well as some hours of retention to be able to mitigate downtime from any component. This means that HDD/SSD disks come into play, lowering transmission speeds by definition. Or it becomes REALLY expensive, real quick.