Changing Log Levels

Kubernetes

In kubernetes each Axual micro-service can be configured with a log level. The log levels for each service except for the broker can be configured on the fly by changing the setting in the correct values.yml file and redeploying the chart.

The Kafka Broker and Kafka Connect are configured seperately

Settings Configuration

Download the Axual Helm charts. These can be maintained in a tool such as ArgosCD.

Each micro-service contains a values.yml file. For example <Install Path>/charts/mgmt/charts/api/values.yaml contains the settings for the Management API.

Each service contains a logging setup for example:

logging:
    audit:
        pattern: "%date{\"yyyy-MM-dd'T'HH:mm:ss,SSSXXX\", UTC} [audit] [%t] %-5level %logger{36} - %msg%n"
    loggers:
        io.axual.auditing.logging: info
        pattern: "%date{\"yyyy-MM-dd'T'HH:mm:ss,SSSXXX\", UTC} [system] [%t] %-5level %logger{36} - %msg%n"
        rootLoglevel: info
    loggers:
        io.axual: info

The Pattern must not be changed. Each service must log messages in the same format, or else searching in Elastic will become very messy. Under loggers you can change the logging settings. In the Spring Admnistration we changed the org.apache.catalina.core.StandaloneService settings. (That is module that will never cause an issue - thus fine for examples.) We will make the same change to the loggers:

logging:
    audit:
        pattern: "%date{\"yyyy-MM-dd'T'HH:mm:ss,SSSXXX\", UTC} [audit] [%t] %-5level %logger{36} - %msg%n"
    loggers:
        io.axual.auditing.logging: info
        pattern: "%date{\"yyyy-MM-dd'T'HH:mm:ss,SSSXXX\", UTC} [system] [%t] %-5level %logger{36} - %msg%n"
        rootLoglevel: info
    loggers:
        io.axual: info
        org.apache.catalina.core.StandaloneService: DEBUG

Deploying Changes

Deploy your changes with the command. Note that your company may use ArgosCD.

helm upgrade --install platform axual-stable/platform -f values.yaml -n kafka

It is possbile to create a new values.yaml with only changed entries. Your company may have a company best practices for updating helm files.

Axual Connect

You will need to use JMX to update the Axual Connect.

The standard Axual Connector port is 11002.

JMX

Most of our applications have exposed JMX ports. We generally use the VuisualVM JMX client. You can download the client here: VisualVM: Home

Setup

For the Micro-service that you are debugging you will need to know the JMX port. Axual connect has JMX installed with the setting. The default setting is port 11002. You can check this with the pod/container’s startup command.

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.port=11002
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.rmi.port=11002
-Djava.rmi.server.hostname=192.168.99.100

Implement these settings on the java startup to enable JMX. You also need to expose the port you can only do that on the docker image startup with the -p <docker internal port from>:<docker external port to> syntax. For example

docker run -d -p 5801:5801 -p 5802:5802

It is not easy to change our Platform Deploy startup scripts. As an alternate to changing our platform startup script you can create a docker run command from a running container. (I tried this and it works.) The repository GitHub - lavie/runlike: Given an existing docker container, prints the command line necessary to run a copy of it. contains a script that will generate a run command from a running container. Once that command has run, add the port mappings. (No, this is not something that you would do in production. I run this to make my local testing easier.)

As this point you will have a container with an exposed JMX port.

Changing Log Settings

Start Visual VM. At the top of the screen click Add Remote Connection. Enter a name and the connection Host and Port.

Next, click MBeans, Java.util.logging → Logging

In the screen on the right click the Attributes tab and then double click Attribute Names

Here you will see a list of logger names. Choose one - java.awt.Cursor.

Open the operations tab next to setLoggerLevel type in the value - java.awt.Cursor and the log level. For the log level you can choose from FINEST = debug level, INFO= info level and SEVERE = no logging.

For Connect this may be the only way that we can change the logging levels. The connect JMX port is by default enabled. This is not very nice.

Axual Broker

In the tools you will notice that we have mentioned the Kafka Admin Client. Implementing this client will assist in being able to view the Broker configurations and change Broker log levels.

It is in general not a good idea to make the Axual Broker log more information. All extra broker logging will slow down the system.