How to Alert on a Log Pattern in Kibana

This guide shows you how to turn a log filter into a Watcher threshold alert in Kibana, set how often it runs, send it to Slack or email, and check that the watch is running.

Type

How-to guide

Goal

Get told the next time a known log pattern appears, instead of finding it by searching.

Audience

A Kibana user with permission to manage Watcher, which read-only accounts do not have.

When to use

Use this guide once a search has identified the log lines that signal a problem.

The screens below are Kibana, the query front end of the Elasticsearch, Fluentd and Kibana (EFK) stack the Axual cloud runs. Watcher is an Elasticsearch feature, so a central stack built on something else raises its alerts through its own tooling.

Contents

The sections below cover each task in this guide:

Prerequisites

Confirm the following before you begin.

Access and permissions required

You need the following access and permissions:

  • Access to the Kibana instance, with read access to the log indices.

  • Permission to manage Watcher. Creating an alert is a Management action, which read-only accounts do not have.

Resources that must exist before starting

The following must already exist:

  • The name of the index your logs land in, and the timestamp field it carries. Index names are chosen per deployment, so ask the team that runs the stack.

  • A filter that matches the lines worth being told about. How to Search Logs in Kibana covers building one.

  • A Slack or email destination available in Watcher, for the alert to send to.

Create the threshold alert

A filter that found a real problem is worth turning into an alert, so the next occurrence reaches someone instead of waiting to be searched for. An alert combining log_level: ERROR with a specific message is usually more useful than one on the level alone.

  1. Click the gear icon in the left toolbar to open Management.

  2. Click Watcher in the Elasticsearch list.

  3. Click Create, then Create threshold alert.

  4. Fill in the alert:

    1. Enter a name.

    2. Select the index in Indices to query.

    3. Select the timestamp field in the Time field area.

    4. Set how often the watch runs in Run watch every.

    5. Build the matching condition with the embedded query generator.

  5. Click Add Actions and choose Slack or Email as the destination.

Confirm the watch is running

The Watcher list is where the new alert shows whether it is active, so check it rather than waiting for the pattern to recur. Open Management, then Watcher, and find the watch under the name you gave it.