Kubernetes

On this page we list commands which are useful in monitoring/debugging problems in Kubernetes.

Please view the Kubernetes Documentation for further information on how to administer a kubernetes environment.

The Axual environment is built from containerized microservices. The containers are managed by Kubernetes. Kubernetes will list the status of each container and restart containers if health checks fail.

To list the status of each Axual module run the command:

The Axual Kubernetes Ingress (reverse proxy and security filter) runs inside of the nginx namespace. Your company may have its own Ingress configuration. Please check with your administrators if the nginx namespace is used.

In the output of the above command the status of each pod should be Running or Completed. If you have just started the system it will take some time for each pod to reach the status of Running. Some pods are used to initialize something in the system. After the initialization is complete the pod will stop with status Completed.

If a pod does not start, then querying the pod status is generally the first step in debugging.

From the get pods command, copy the name of the pod and use that in the command below.

In the output from this command check the pod status and the Events. The events sometimes show a reason for the pod failure which does not show up in the logs. It is good to always check a failing pod startup with the describe command.

If there are general error coming from the pod please check the pod logs. Your company should have its own logging components setup. At Axual we use the Kubernetes Elastic/Fluentd/Kibana stack.

We also have a general Logging page that describes how to view the Kubernetes pod logs. There is a kubernetes command that you can use to tail a Kubernetes pod log:

Please use this command if you have a reproducable error that should be easily viewed in logging.

If you need more detailed logs please alter the pod logging levels.

It is possible that a service may become unresponsive, even though kubernetes reports the service as running. If you are certain that the service is not responding then please identify and restart the appropriate pod.

Step 1: Find the pod. Run the command:

Step 2: Attempt to debug the cause of the issue.

Step 3: As a last resort restart the pod. (For the Broker and Zookeeper please read the Strimzi Documentation)

Each service is a part of a replica set and will be automatically restarted when removed.