Monitoring & logging

Overview

In this segment we discuss how to monitor Connect. For dashboards and visualizations we use Grafana.
The way in which this is achieved is twofold, first there is Connect itself which is the docker container on which connectors are ran but there is also the connectors themselves.

The assumption is that you have provided all the necessary configurations per the deployment documentation such that the container running Connect exposes the metrics that are monitored here.

There is an abundance of metrics offered by Connect which can be used to generate dashboards catered to specific needs. Here we present the dashboards that come out of the box as a generic solution to monitoring connect. More dashboards can be added depending on the setup at hand.

You can find the link to Prometheus as the source of the Data that is queried from Grafana alongside documentation per exposed metric, to create custom visualizations.

Monitoring Connect

Connect Dashboard provides a general overview of Connect service of a given instance.

Connect Dashboard Overview

The views of this dashboard can be filtered by Instance / Connect Server / Connector / Task

Connect Dashboard Filters

Information found here:

  • Connect Overview (per instance):

    • Important - Connect Status per instance - UP / DOWN / N/A

    • Connector Startup Health - OK / N/A

    • Tasks Startup Health - OK / N/A

    • Number of Paused/Failed Connector/Tasks per instance

    • Number of Defined Connector per instance

    • Number of Assigned/Failing/Paused connectors/tasks per instance

    • Incoming/Outgoing byte rate per instance

    • Record Poll/Write rates

  • List of Connectors Overview - A Generic overview of connectors running on the instance, as they appear in Connector Monitoring.

When connect is not configured for the specific instance the value will be N/A

Monitoring Connectors

Connector Dashboard exposes metrics that concern individual connectors.

Connector Dashboard Filters

The views of this dashboard can be filtered by Instance/Connector/Task

Connector Dashboard Filters

Information found here includes (per connector):

  • Connector Overview

    • Task Assignments - Shows the task assignments per Axual Connect node

    • Important - Connector status per task - Running/Unassigned/Paused/Failed/Destroyed

    • Connector type - source-connector/sink-connector

    • Number of messages polled but not written

    • Transform dropped record rate

    • Offset commit success rates

    • Important - Tasks status - Running/Unassigned/Paused/Failed/Destroyed

    • Connect Record poll rates

    • Record drop rates

    • Processed batch sizes

When the connector is not configured for the specific instance the value will be N/A

Connect logging

The Connect cluster

Connect to the Instance node where Connect is running

  • Option 1 - docker command

docker logs -f {tenant}-[instance]-axual-connect
  • Option 2 - Open the log file system.log in the following path

[Log Path]/{tenant}-[instance]-axual-connect/
Collect your logfiles centrally so you don’t have to log in to individual nodes to gather information about your connect nodes and connectors.

Connector logging

Each connector instance coordinates a set of Tasks that actually copy the data.
To see logs for a specific connector you will first need to find which nodes running the Tasks for our Connector, use Monitoring Connectors dashboard to get the node name.

For each node, navigate to the following path, where you can find the connector log file - [CONNECTOR NAME].log

[Log Path]/{tenant}-[instance]-axual-connect/connectors

The Log Path value is define in LOG_DIR configuration located in platform-config/clusters/{cluster-name}/node.sh file
Default location for the logs file is /appl/logs/