Operate and recover the fleet

Read the running state

  1. Open Status and check Online, Silent, Never heartbeated and Stopped workers, plus active warnings. A Stopped worker said it was stopping and raises no warning. An ephemeral worker leaves the list once it is gone.
  2. Select an affected worker and compare its reported Commit with the deployed main commit. Check its active issues and last-seen time. A heartbeat shows the worker recently contacted the leader. Check the running commit to see whether deployment succeeded.
  3. Trace the affected source, pipeline and destination in the map. Inspect throughput and owned partitions before changing thread limits. A source warns when its rows in the last 30 minutes fall outside its expected range, learned from the same time of day over the last week. Set how wide that range is in Admin → System → General.
  4. Use Logs to narrow by severity, worker, group, message and time. Preserve timestamps, commit IDs and error text for an incident report; omit secrets and sensitive sample records.

How to compare the deployed and running commits →

Delivery and recovery

For a stalled destination, distinguish connection failures from refused records. Check the destination's errors, its action for refused records, and any destinations chosen to keep those records. Check how many records are waiting and how long the source will keep them. Fix the cause before replaying or changing policy.

Delivery covers holding and retrying delivery, failures while saving refused records, and recovering and replaying records.

Change log verbosity

Set the top-level log_level before any TOML section header, or set SMELTR_LOG_LEVEL in the process environment, and restart the affected leader or worker:

log_level = "debug"

[leader]
enabled = true

Levels: error, warn, info (default), debug and trace. Use debug or trace briefly, then restore info. Do not share logs containing credentials or sensitive records.

Deploy and roll back configuration

Save, test, review, then deploy. Deployment publishes configuration to main.

  1. For rollback, open Branches → History and identify the earlier known-good commit.
  2. Review its affected files and secret-version requirements. Rollback can affect the whole fleet, not only the group selected in the UI.
  3. Select Roll back to here and confirm Deploy. The fleet stays on that commit until Release on the same page.
  4. Check data flow in Status.

Do not rewrite the worker's local configuration manually. A configuration rollback is different from downgrading the executable.

Leader outages and revoked workers

These outage rules apply to workers running separately from the leader. In standalone mode, stopping or restarting the process affects both roles. New workers need a successful initial check-in before they start following main. Once initialized, workers keep running while the leader is unavailable. A worker that restarts runs nothing until the leader answers its first check-in. The default unreachable policy has no automatic stop; an explicit worker.heartbeat.stop_after_unreachable_secs changes this behavior.

Ephemeral workers run on through a leader outage and stay enrolled: the leader counts a worker's silence only while it is up. One that stops during the outage keeps trying to say so while it stops. If it never gets through, the leader reports it silent, then forgets it once the leader has been back for longer than the Ephemeral workers setting.

Revoking a worker blocks leader access but does not stop its pipeline. To stop it immediately, stop or isolate the host and revoke its external credentials.