Viewing Application Logs

The sections below cover each task in this guide:

Complete the following before you start.

Access and permissions:

A connector is no different: its own owner group governs its logs, whichever Kafka Connect cluster it runs on. Being in the cluster’s Owner Group or an Authorized Group lets you deploy connectors there, and does not by itself let you read the logs of a connector another group owns. See Group authorization for Kafka Connect clusters.

Resources that must exist first:

The log console shows the lines an application writes while it runs, for one environment at a time.

The console opens and starts streaming. The header shows Live while lines are arriving, and Stopped once the stream ends.

An empty console does not mean logs are unavailable. The message in it names the case:

A single line reading Application is starting…​ means the container is not up yet. The console reconnects every five seconds while that stays the newest line, so it fills in on its own once the container starts.

The console follows the log while it streams, so new lines push older ones up. Use these controls when you need to read something that keeps scrolling away:

Two limits apply to the console:

The toolbar keeps a running count of the lines on screen, such as 1248 lines, and the bin icon beside it empties the console.

Clear the console before you restart a connector, so the lines arriving next are the restart and nothing else.

Search runs over the lines the console currently holds, and highlights every match.

A connector runs on a Kafka Connect cluster whose workers log every connector deployed to them. The This connector only toggle chooses which of the two you read. It appears for connector applications only, because a KSML application has a runtime of its own and no wider scope to switch to.

With This connector only off, lines from every worker of the cluster are merged into a single chronological order rather than shown one worker at a time, so interleaved output from two workers reads the same way it would from one machine.

Turn This connector only off when a connector never reached a running state. A connector that failed to start has written nothing under its own name, so its filtered log is empty. The reason it never started is in the worker log.

A running connector can also have lines missing from the filtered scope, for a different reason. See Why the connector filter can miss lines.

Switching the toggle reopens the stream and clears the console, because the two scopes are different content and interleaving them would attribute lines to the wrong connector.

This connector only is best effort. It shows the lines the platform can attribute to this connector, which is nearly all of them, and it cannot show a line that arrives with nothing naming the connector on it.

Almost every line a connector writes passes through the worker’s logging framework, which stamps each line with the connector that produced it. The filter matches on that stamp. A connector that writes straight to standard output instead, for example with a raw System.out.println call, goes around the logging framework, so its lines reach the console unstamped. The filter has nothing to match on and drops them, even though that connector wrote them.

This is why turning the toggle off can look inverted, as though the wider scope were the one showing this connector’s output. The lines were always this connector’s. They were never labelled as such, so only the unfiltered scope keeps them.

Few connectors write this way, and whether one does is a property of the connector’s own code rather than of the platform. Kafka’s FileStreamSinkConnector is the example most people meet.

Two scopes are complete, and neither depends on the stamp:

Use one of those before concluding that a connector logged nothing.

The Log level dropdown drops every line below the severity you pick. It appears for connector applications only, alongside the This connector only toggle.

A line that carries no severity of its own, such as a frame of a stack trace, stays with the line above it. An error therefore arrives with its whole trace under it, not as a bare first line.

Changing the level reopens the stream and clears the console, the same as the This connector only toggle. One console cannot hold two thresholds at once.

An empty console at Warnings and above or Errors and above usually means the workers logged nothing at that severity. The console says so directly, for example No logs for Warnings and above. Choose All levels to see everything. Pick All levels to confirm the application is logging at all.

The History dropdown sets how many earlier lines the stream sends before it starts following the log. It appears for connector applications only, and starts at Last 100 lines.

Changing the history reopens the stream and clears the console. Reopening is the only way to ask for more history, because the lines already on screen cannot be extended backwards.

The Worker dropdown narrows the console to the lines a single worker pod wrote. It appears for connector applications only, because only a Kafka Connect cluster runs several workers.

Filtering by worker acts on the lines the console already holds, so it neither reopens the stream nor discards anything. Switching back brings the hidden lines straight back.

Changing the Log level or the History reopens the stream, which empties the console and rebuilds the worker list from the lines that arrive next. The worker you selected stays selected while that happens, so a worker that has not logged again yet shows an empty console with a message naming it.

The Fields dropdown lists the parts of a structured log line as checkboxes. The console shows each one you tick on every line that carries it.

The fields render in the order above, whichever ones you tick. A line the console cannot parse into fields, such as the shell output a worker writes before its logging starts, shows its message alone.

Your choice is saved in the browser, so it applies to every log window you open next, for every application. Copy to clipboard and Download Visible Logs follow it too, which is worth checking before you attach a log to a ticket.

Take the log with you when you want to attach it to a ticket or read it in another tool. On a connector application the Download button opens a menu offering two scopes. A KSML application has a plain Download button instead, which saves what the console holds.

Copying and Download Visible Logs act on the console’s current contents, so pause the stream first if you want a stable snapshot.