Topic, group and application patterns

Enabling multi environment and multi tenant support

Axual Platform supports having multiple environments on the same physical Kafka cluster as well as multiple tenants. Especially in situations where you want to support multiple environments but don’t want to be faced with costs and deployment complexity of multiple Kafka clusters, this works well.

To support this scenario of "multi environment clusters", we introduced the concept of an Instance.

Axual Governance - environments, instances and clusters

The image above shows a typical setup, where 2 physical clusters are created ("Non-Prod" and "Production") and for each cluster an instance is created. Lastly, to each instance, environments are mapped. In this example environment "Dev", "Test" and "Acc" are mapped to the instance "DTA", whereas environment "Prod" is mapped to instance "PROD".

To avoid clashes, the name of the instance and environment are used in the topic name, consumer group ID as well as the transactional ID.

When creating a cluster you need to specify the following patterns:

  • Topic pattern: the pattern used to create topic names

  • Group pattern: the pattern used to create consumer group IDs

  • Transactional pattern: the pattern used for the transactional ID

Which pattern should I use?

Which pattern to use depends on the scenario you want to support.

Scenario: single cluster, single instance, multiple environments (single tenant)

This is the scenario depicted above. In this situation, the following patterns would be advised:

  • Topic pattern: {environment}-{topic}

  • Group pattern: {environment}-{app.id}

  • Transactional pattern: {environment}-{app.id}-{transactional.id}

Example topic name: dev-foo

Scenario: single cluster, multiple instances, multiple environments (single tenant)

In this situation, the following patterns need to be used.

  • Topic pattern: {instance}-{environment}-{topic}

  • Group pattern: {instance}-{environment}-{app.id}

  • Transactional pattern: {instance}-{environment}-{app.id}-{transactional.id}

Example topic name: dta-dev-foo

Scenario: single cluster, single instance per tenant, multiple environments (multiple tenants)

Axual Governance also supports using multiple tenants using a single installation (thus a single cluster). This means that in addition to {instance} and {environment}, {tenant} needs to be added to the patterns.

  • Topic pattern: {tenant}-{instance}-{environment}-{topic}

  • Group pattern: {tenant}-{instance}-{environment}-{app.id}

  • Transactional pattern: {tenant}-{instance}-{environment}-{app.id}-{transactional.id}

Example topic name: `company-dta-dev-foo

What if I don’t set any pattern?

If you don’t set any pattern, the following defaults are used:

  • Topic pattern: {topic}

  • Group pattern: {app.id}

  • Transactional pattern: {transactional.id}