Topic, group and application patterns

Enabling multi environment and multi tenant support

Axual Platform supports having multiple environments on the same physical Kafka cluster as well as multiple tenants. Especially in situations where you want to support multiple environments but don’t want to be faced with costs and deployment complexity of multiple Kafka clusters, this works well.

To support this scenario of "multi-environment clusters", we introduced the concept of an Instance.

Axual Governance - environments, instances and clusters

The image above shows a typical setup, where two physical clusters are created ("Non-Prod" and "Production") and for each cluster an instance is created.

Lastly, to each instance, environments are mapped.

In this example, environment "Dev", "Test" and "Acc" are mapped to the instance "DTA", whereas environment "Prod" is mapped to instance "PROD".

To avoid clashes, the name of the instance and environment are used in the topic name, consumer group ID as well as the transactional ID.

When creating a cluster, you need to specify the following patterns:

  • Topic pattern: the pattern used to create topic names

  • Consumer Group pattern: the pattern used to create consumer group IDs

  • Transactional ID pattern: the pattern used for the transactional ID

    The Transactional ID pattern is used to create a unique Transactional ID for an application.

    In a multi-tenant environment, Axual Platform creates a Prefixed Transactional ID ACL on {tenant}-{instance}-{environment}-{transactional.id}.

    To enable transactional delivery, the producer application must specify the transactional.id Kafka property to a value similar to mytenant-myinstance-myenv-myappid-myproducertxn.

    In this case, myproducertxn can be any random string.

Which pattern should I use?

Which pattern to use depends on the scenario you want to support.

Scenario: single cluster, single instance, multiple environments (single tenant)

This is the scenario depicted above. In this situation, the following patterns would be advised:

  • Topic pattern: {environment}-{topic}

  • Consumer Group pattern: {environment}-{group}

  • Transactional ID pattern: {environment}-{transactional.id}

Example topic name: dev-foo

Scenario: single cluster, multiple instances, multiple environments (single tenant)

In this situation, the following patterns need to be used.

  • Topic pattern: {instance}-{environment}-{topic}

  • Consumer Group pattern: {instance}-{environment}-{group}

  • Transactional ID pattern: {instance}-{environment}-{transactional.id}

Example topic name: dta-dev-foo

Scenario: single cluster, single instance per tenant, multiple environments (multiple tenants)

Axual Governance also supports using multiple tenants using a single installation (thus a single cluster). This means that in addition to {instance} and {environment}, {tenant} needs to be added to the patterns.

  • Topic pattern: {tenant}-{instance}-{environment}-{topic}

  • Consumer Group pattern: {tenant}-{instance}-{environment}-{group}

  • Transactional ID pattern: {tenant}-{instance}-{environment}-{transactional.id}

Example topic name: `company-dta-dev-foo

What if I don’t set any pattern?

If you don’t set any pattern, the following defaults are used:

  • Topic pattern: {topic}

  • Consumer Group pattern: {group}

  • Transactional ID pattern: {transactional.id}