# Logs & metrics

Every Smeltr process keeps two accounts of itself: logs, which are timed, structured events, and metrics, which are counters and gauges. A worker's heartbeat carries both to the leader, and a bundle can route either through a pipeline to any destination.

## Where logs and metrics go

| Goes to | Logs | Metrics |
| --- | --- | --- |
| The console | Every process writes one line per event to stderr. | — |
| The leader | Each worker sends its logs with its heartbeat. The leader keeps the newest 100,000 rows for the fleet. | Each worker sends its metrics with its heartbeat. The leader keeps them for 30 days, in coarser buckets as they age. |
| A destination | Through a [Smeltr logs source](/docs/sources/logs). | Through a [Smeltr metrics source](/docs/sources/metrics). |

The leader keeps its own logs on its own console. Where one process runs both the leader and a worker, the leader's logs reach the leader with that worker's heartbeat.

The leader's copy is its operational view. Send logs and metrics you need to keep or search elsewhere to a destination.

## Read them in the GUI

-   **Status → Logs** shows the fleet's logs, newest first. Choose severities, a worker, a group, a site, or text in the message, then choose **Search**. A severity is not a minimum: choose `error` and `warn` to see both. Select a row's fields to open them. The filters are in the page's address, so you can share a view as a link.
-   **Status** shows Rows in and out, Errors and Furthest from normal, from the metrics the heartbeat carries. Choose the window of Rows in and out: 15m, 60m, 24h, 7d or 30d. A gap in a line is missing history, not zero.

The leader cuts a log's site, message and each text field to 1,000 characters, and keeps its first 16 fields. A search cannot find what the leader cut. A worker out of contact with the leader holds the oldest 4,096 of its own events and the newest 2,048 from the process running its pipelines.

## Send them to a destination

1.  Select the worker group and work on a branch.
2.  Open Build → Sources, choose **\+ Add source**, choose **Local, each worker**, and then **smeltr logs** or **smeltr metrics**. Its settings are on the [Smeltr logs](/docs/sources/logs#settings) and [Smeltr metrics](/docs/sources/metrics#settings) pages.
3.  Connect it to a route, which sends its records through a pipeline to a [destination](/docs/destinations). [Follow the routing example](/docs/first-pipeline#route).
4.  [Review and deploy](/docs/review).

Each source gives the logs or metrics of the worker it runs on, covering both of that worker's processes: the `supervisor`, which talks to the leader, and the `runtime`, which runs its pipelines. Add one to every worker group you want to watch. Its records carry `__source.topic` `__smeltr_logs` or `__smeltr_metrics`.

## What metrics exist

Smeltr has 74 metrics, grouped by what they measure: sources, processing, pipeline steps, destinations, memory, the worker and its logs. Each is a counter or a gauge with labels naming its subject. The [Smeltr metrics source](/docs/sources/metrics#how-it-works) lists every one, with its labels and what it measures, and shows the record each one makes.

## Check for lost logs

Writing a log never waits. A process with no room for an event drops it and counts it. These metrics carry a `process` label naming the process that wrote them.

| Metric | Measures |
| --- | --- |
| `smeltr_logs_written_total` | Log events written since the process started, by the `site` that wrote them. |
| `smeltr_logs_dropped_total` | Log events dropped for want of room since the process started. |
| `smeltr_logs_unread_total` | Log events a full logs reader's queue refused since the process started, one count per reader. |
| `smeltr_logs_undelivered_total` | Log events refused while the leader was out of contact, since the process started. |
| `smeltr_logs_unwritable_total` | Log events the console would not take since the process started. |

## Change log verbosity

A process collects logs at or above its log level, `info` unless you set it. The console, the leader and every Smeltr logs source get only what the process collects. To change it, see [Change log verbosity](/docs/operations#logs).