Starting Connectors

Creating Connect-Applications

  1. Open the "Applications" page in the Self-Service portal

    Axual "Self-Service" UI, "Applications" page

  2. Click the New Application button. This window will open

    Axual "Self-Service" UI, "Add Application" page

  3. Fill the required information in the form:

    • ID: The Application ID of your Connect-Application.
      This is a string that uniquely identifies your Connect-Application.
      The maximum length is 255. It can contain numbers, letters and dots.
      We recommend using a fully qualified package/class name, which is sufficiently unique within the organization.
      E.g. com.company.division.AppName.

    • Name: Full name of the Connect-Application.
      This value is displayed when opening this Connect-Application’s page.
      The maximum length is 50. Allowed characters are displayed in the UI.

    • Short Name: A shorter name for the Connect-Application. This name is displayed in the applications list page, making it a little more important than the full name. The maximum length is 60. It can contain numbers, letters and underscores. It has to be unique within the company (tenant).

    • Owner: All members of the selected group will be authorized to edit this Connect-Application.
      This dropdown lists all the groups you belong to. Choose an adequate group.
      If no group is listed, ask a team member or platform-operator to add you to a group.

    • Type: choose "Connector"

    • Plugin Name: choose the plugin corresponding to the system you wish to integrate with Kafka

    • Visibility: Controls read access. Public applications are readable to the entire organisation.
      Private applications are only visible to its owners.

    • Description: A short summary describing the purpose of this application.
      Limit of 200 characters.

  4. Click the Add Application button in the bottom-right of the screen.
    Upon creating the Connector-Application in the Self-Service portal, you will be directed to the application detail page.
    Here you can "instantiate" this Connect-Application onto multiple environments.

    Axual "Self-Service" UI, "Application" page, showing an example sink connector application

Configuring and starting Connector-Applications

A Connector-Application needs 2 sets of configuration for every environment in which it’s deployed:

  1. Security: The certificate and private key files used for authentication and authorization against the Kafka cluster.

  2. Plugin Configuration: plugin-specific configuration, depending on the connector implementation (e.g. connectivity information for the integrated system).

Axual "Self-Service" UI, section of the "Application" page, showing the security and configuration modal of an unconfigured connect-application, with the blue circle indicators on both configuration buttons A blue indicator is shown if the Security or Plugin Configuration is not yet provided.

Security configuration

The certificate and private key is used to authorize the Connector-Application against kafka when accessing the streams. At least one certificate and private key must be configured for every environment this application intends to use.

  • The certificate must be X.509 formatted and contained within a PEM file. Ideally, it should contain the whole certification chain.
    The uploaded file must have one of the following file-extensions: .crt, .cert or .pem.

  • The private key must be pkcs8 formatted and contained within a PEM file.
    The uploaded file must have the .pkcs8 file extension.

If your key material is currently in JKS format, you can extract the certificate and private key using a tool such as Keystore Explorer. You can also install it using multiple package managers (see repology), including brew.

Considering you have both these files available, let’s configure the Connector-Application’s security:

  1. From the environment dropdown, select the environment for which you want to configure the certificate and private key.

  2. Click on the Padlock button in the application box (hovering on the button will show the text 'Authorization'). The following window will show up:

    Axual "Self-Service" UI, section of the "Application" page, showing the security-configuration pop-up modal of an unconfigured connect-application

  3. Upload the Certificate file your connector will use in this environment.
    The window will indicate the certificate was successfully uploaded, and request you follow-up with a private key:

    Axual "Self-Service" UI, section of the "Application" page, showing the security-configuration pop-up modal with only the certificate configured for a connect-application

  4. Upload the private-key file that matches the certificate.
    The private key will not be displayed on the screen. You will only see whether it was successfully uploaded or not. The window will indicate the private key was successfully uploaded.

    Axual "Self-Service" UI, section of the "Application" page, showing the security-configuration pop-up modal with correct configuration for a connect-application

  5. Click Apply

Plugin configuration

  1. Click on the Gear button in the application box (hovering on the button will show the text 'Configuration'). A window similar to this will show up:

    Axual "Self-Service" UI, section of the "Application" page, showing the <strong>Connector</strong>-configuration pop-up modal of an unconfigured connect-application

  2. If you are following a Connect-Plugin configuration guide, this is the step where you will provide the Connector configuration as instructed. Click the save button once you are done.

  3. If your Connector-Application was successfully configured in the selected environment, you will see an extra box at the bottom of the application box and a status label at the top:

    Axual "Self-Service" UI, section of the "Application" page, showing the <strong>Connector</strong>-configuration status box of a configured connector, in "ready" state (not started yet)

Requesting Stream Access

Your Connector-Application needs to be authorised to produce/consume a kafka stream before it can be started.
Follow the Requesting stream-access docs to grant it access.

Starting and stopping Connector-Applications

You can start the Connector by clicking the 'START' button. Within seconds, the status should change to 'RUNNING'.

Axual "Self-Service" UI, section of the "Application" page, showing the <strong>Connector</strong>-configuration status box of a configured connector, in "running" state

If the status changed to FAILED, you can click View tasks to open up a modal showing you a stack trace that should give you more information. Alternatively, you can take a look at the connector’s logging (see below).

Viewing Connector logging

As soon as you have started your connector application and something goes wrong, e.g. your connector goes in a FAILED state, you can use connector logging to find pointers for the cause of the error.

Prerequisite: connector logging is enabled by the platform administrator. See also Enable connector logging

  1. On the Connector page, click on the "Logs" tab. The following screen shows

    Axual "Self-Service" UI, section of the "Application" page, showing the connector logs tab

  2. Click "Search" to search immediately. This will return the logging of the last hour. If you are looking for specific log messages, you can choose to configure the following parameters:

    • Search string: here, enter the string you would like to search for in logging. Use ' ERROR ' or ' WARN ' (including spaces) to find logging on the ERROR or WARN log level respectively.

    • Environment: from which environment do you want to see the logging of your connector

    • Relative time range: if you want to limit the time interval to last 5 minutes or something else, use this dropdown

    • Absolute time range: if you know the specific time range you want to see logging from, define the start and end of the time range here

If your search returned logging for your connector, you will see a screen like this:

Axual "Self-Service" UI, section of the "Application" page, showing the connector logs tab after log messages have been returned
Please note that search results are returned in chronological order, with the oldest (log) message first. An improvement to allow control of the search result order is pending implementation.

Deleting A Connector Application

  1. Stop all the Connector-Applications on all environments.

  2. Follow the Deleting applications docs to remove the application.

Available Connector Application Actions

The following actions are available for Connector-Applications:

Connector action

Effect

START

start the connector application

STOP

stop the connector application

RESET

remove any existing active connector and its configuration (including offsets)

The following actions are available for Connector-Tasks:

Task action

Effect

VIEW

view the task(s) status and trace (in case the status is FAILED)

RESTART

restart a task in a FAILED state

Configuring Connector Plugin Defaults

Connector Defaults are no longer supported since Axual 2021.3