Destinations

A destination writes the records a route sends it to a system. Each kind of destination has a page of its own.

Kinds

Log analytics

KindtypeWhat it does
ElasticsearchelasticsearchIndex documents through the Bulk API.
Log Analytics workspacelogAnalyticsSend records to a Log Analytics table through a data collection rule.
Splunk HECsplunkHecSend events to HTTP Event Collector.

Object storage

KindtypeWhat it does
Amazon S3s3Write batches as objects in a bucket.
Azure Blob StorageazureBlobWrite batches as blobs in a container.
Google Cloud StoragegcsWrite batches as objects in a bucket.
SFTPsftpWrite batches as files in a directory on an SFTP server.

Observability

KindtypeWhat it does
Prometheus remote_writeprometheusSend samples over remote_write to Prometheus or any store that takes it.

Streams

KindtypeWhat it does
Azure Event HubseventHubsWrite records to event hubs through their Kafka endpoint.
KafkakafkaWrite processed records to topics.

Tables

KindtypeWhat it does
Apache IcebergicebergCommit Parquet files to a table in S3 through its REST catalog.
Delta LakedeltaCommit Parquet files to a table in S3 through its log.

Web

KindtypeWhat it does
HTTPhttpPOST records to any HTTP service, batched or one per request.

How it works

From a source to its destinations, and back to the commitThe reader polls a batch from the source and tracks its position. Transform routes the batch and runs its pipelines. Egress, one per destination, writes each destination's records and retries until they land. Each destination acknowledges its records, and once every destination the batch reached has acknowledged it, the reader commits the batch's position to the source.acknowledgedcommitSourceKafka, metrics, logsReaderPolls a batch,tracks its positionTransformRoutes, runs pipelines,applies deliver rulesEgressOne per destination,retries until it landsDestinationAcknowledgesor refusesCommitWaits for everydestination's ackFrom a source to its destinations, and back to the commitThe reader polls a batch from the source and tracks its position. Transform routes the batch and runs its pipelines. Egress, one per destination, writes each destination's records and retries until they land. Each destination acknowledges its records, and once every destination the batch reached has acknowledged it, the reader commits the batch's position to the source.ackcommitSourceKafka, metrics, logsReaderPolls a batch,tracks its positionTransformRoutes, runspipelinesEgressOne per destination,retriesDestinationAcknowledgesor refusesCommitWaits for everydestination's ack
A position is committed only after every destination the batch reached has acknowledged it, oldest batch first. After a restart or a retry a record can arrive twice, but it is not lost. Smeltr metrics and logs sources keep no position.

What happens to a record decides when its batch's position commits. Until it commits, a restart reads the batch again.

The recordThe position commits
Is acknowledged by every destination it was sent toOnce every older batch has committed too.
Is dropped by a step, or picked by no deliver ruleWith its batch.
Reaches a destination that does not answerOnly once it lands. Egress retries, and a destination that stays down pauses the sources feeding it.
Carries an exceptionBy the answer to it, as below. A pipeline's exceptions are answered by its route's exceptions over the group's; Unmatched and Unprocessable by the group's; Refused by the destination's over the group's.
AnswerThe position commits
ignore, tagOnce the destinations the record goes on to acknowledge it.
dropWith its batch.
deliverOnce every destination named acknowledges the record.
blockNot until the record goes through. A refused record is sent again until the destination takes it; any other stops its source until a new config is deployed.

Unmatched, Unprocessable and Refused block unless an answer says otherwise, and take no ignore or tag. Exceptions covers each answer.

Settings

A destination is one entry of the bundle's Destinations array. Every kind takes these settings:

id string required

The destination's name, unique among destinations. Routes and other destinations' answers name it.

Examples
events-outsiem-archive

exceptions answers default the group's answer, block unless it gives another

What happens to a record this destination refuses: block keeps it and sends it again until the destination takes it, drop lets it go, and deliver sends it to other destinations. See Handle refused records.

Examples
{"kinds": {"Refused": "drop"}}{"kinds": {"Refused": "block"}}

Formats

A destination's format decides what it writes of each record. Routes and exception answers name only destinations, so one record can reach two destinations in two forms.

formatWhat it writes
json, ndjson, parquet, avro, csvThe record as the pipeline wrote it.
textThe text of the field field names, _raw by default. The destination refuses a record without that field.
bytesOne field's bytes, once per source record. The destination's field names the field, and defaults to __source.bytes: each record's bytes as the source received them, before any pipeline ran. Those bytes let you replay records: a source that reads them back gets what the original source got.

Each kind writes only some formats:

Destinationformat
HTTPjson (default), text, bytes (base64 in a batch: bodies)
Kafkajson (default), text, bytes, avro
Elasticsearchjson
Splunk HEC, event endpointjson (default), text
Splunk HEC, raw endpointjson (default), text, bytes
S3, Google Cloud Storage, Azure Blob, SFTPndjson (default), json, parquet, avro, csv, text, bytes
Apache Iceberg, Delta Lakeparquet, as the table's data files

Fields

Smeltr never delivers a record field whose name starts with __. A pipeline sets __destination fields to steer a record, and each kind reads only its own:

FieldTypeHolds
__destination.topicstrThe Kafka topic or Event Hub a destination writes the record to. Null means the default value for that destination.
__destination.indexstrThe index an Elasticsearch or Splunk HEC destination files the record under. Null means the destination's default index. A Splunk HEC destination on the raw endpoint refuses a record naming an index other than its own.
__destination.idstrThe Elasticsearch document ID, used when the destination's document ID setting is dedupe or overwrite. When it is null, the destination's id field names the field that gives it, and with neither the document is sent with no ID.
__destination.prefixstrWhat an S3, Google Cloud Storage or Azure Blob Storage destination's object key starts with, taken as it is. Null means the destination's own prefix, with its time placeholders filled in.

Add a destination

  1. Select the worker group and work on a branch.
  2. Open Build → Destinations and choose + Add destination.
  3. Choose its type, give it a unique id and fill in its settings, then choose Add. Save the page.
  4. Connect it to a route's delivery path, whose condition selects the records it receives from a source. Follow the routing example.
  5. Review and deploy, then verify it.

Verify

  1. Check that the intended workers are online and run the deployed commit.
  2. Send a record you can recognize through a source and a route to the destination.
  3. Check Status and Logs for source, routing or destination errors.
  4. Find the record at the destination and check its fields.

A preview of a saved sample checks the pipeline, not the live connection.

Troubleshooting

No records arrive
Follow No records arriving to check input, worker configuration, routing, filtering and delivery in order.
TLS verification fails
Check the server name and the authority in the connection identity. If the server uses a private CA, the connector must trust it.

See also