Sources

A source reads records from a system and hands them to the routes that name it. Each kind of source has a page of its own.

Kinds

Pull, all workers

Every worker of the group reads it, and the queue or stream splits the work between them. More workers read more.

KindtypeWhat it does
Amazon S3 via SQSs3SqsRead the objects S3 announces on an SQS queue.
Azure Blob via Storage queueazureBlobQueueRead the blobs Event Grid announces on a Storage queue.
Azure Event HubseventHubsRead event hubs through their Kafka endpoint.
Google Cloud Storage via Pub/SubgcsPubSubRead the objects Cloud Storage announces on a Pub/Sub subscription.
KafkakafkaRead topic streams or a range of them.

Pull, one worker

One worker of the group reads it at a time, and the leader keeps its progress. If that worker goes silent another takes over. More workers do not read it faster.

KindtypeWhat it does
Amazon S3s3ListRead the objects under a prefix of a bucket that cannot notify.
Azure Blob StorageazureBlobListRead the blobs under a prefix of a container that cannot notify.
Google Cloud StoragegcsListRead the objects under a prefix of a bucket that cannot notify.
HTTP APIhttpApiPoll an address that answers a GET with JSON or lines of text.
Okta System LogoktaLogsRead an Okta org’s System Log, as an API token or a service app.
PostgreSQLpostgresRun a query against a PostgreSQL database every poll and read its rows.
MySQL and MariaDBmysqlRun a query against a MySQL or MariaDB database every poll and read its rows.
SFTPsftpRead the files under a directory on an SFTP server.

Push, all workers

Senders connect to it, and every worker of the group listens. More workers take more.

KindtypeWhat it does
HTTP receiverhttpReceiverTake what senders POST to a path on a port every worker listens on.
HEC receiverhecReceiverTake what Splunk HTTP Event Collector clients send to a port every worker listens on.
S2S receivers2sReceiverTake what Splunk forwarders send over S2S to a port every worker listens on.
OTLP receiverotlpReceiverTake the logs OpenTelemetry exporters send over OTLP/HTTP to a port every worker listens on.
Syslog receiversyslogReceiverTake what syslog senders send over TCP or TLS to a port every worker listens on.
Prometheus receiverprometheusReceiverTake the samples Prometheus sends over remote_write to a port every worker listens on.
Elasticsearch receiverelasticReceiverTake what Beats, Fluent Bit and other Elasticsearch clients send to a port every worker listens on.

Local, each worker

Each worker reads what is on its own host.

KindtypeWhat it does
Smeltr logslogsCollect each worker’s own log records.
Smeltr metricsmetricsCollect each worker’s own metrics.

How it works

From a source to its destinations, and back to the commitThe reader polls a batch from the source and tracks its position. Transform routes the batch and runs its pipelines. Egress, one per destination, writes each destination's records and retries until they land. Each destination acknowledges its records, and once every destination the batch reached has acknowledged it, the reader commits the batch's position to the source.acknowledgedcommitSourceKafka, metrics, logsReaderPolls a batch,tracks its positionTransformRoutes, runs pipelines,applies deliver rulesEgressOne per destination,retries until it landsDestinationAcknowledgesor refusesCommitWaits for everydestination's ackFrom a source to its destinations, and back to the commitThe reader polls a batch from the source and tracks its position. Transform routes the batch and runs its pipelines. Egress, one per destination, writes each destination's records and retries until they land. Each destination acknowledges its records, and once every destination the batch reached has acknowledged it, the reader commits the batch's position to the source.ackcommitSourceKafka, metrics, logsReaderPolls a batch,tracks its positionTransformRoutes, runspipelinesEgressOne per destination,retriesDestinationAcknowledgesor refusesCommitWaits for everydestination's ack
A position is committed only after every destination the batch reached has acknowledged it, oldest batch first. After a restart or a retry a record can arrive twice, but it is not lost. Smeltr metrics and logs sources keep no position.

What happens to a record decides when its batch's position commits. Until it commits, a restart reads the batch again.

The recordThe position commits
Is acknowledged by every destination it was sent toOnce every older batch has committed too.
Is dropped by a step, or picked by no deliver ruleWith its batch.
Reaches a destination that does not answerOnly once it lands. Egress retries, and a destination that stays down pauses the sources feeding it.
Carries an exceptionBy the answer to it, as below. A pipeline's exceptions are answered by its route's exceptions over the group's; Unmatched and Unprocessable by the group's; Refused by the destination's over the group's.
AnswerThe position commits
ignore, tagOnce the destinations the record goes on to acknowledge it.
dropWith its batch.
deliverOnce every destination named acknowledges the record.
blockNot until the record goes through. A refused record is sent again until the destination takes it; any other stops its source until a new config is deployed.

Unmatched, Unprocessable and Refused block unless an answer says otherwise, and take no ignore or tag. Exceptions covers each answer.

Settings

A source is one entry of the bundle's Sources array. Every kind takes these settings:

id string required

The source's name, unique among sources. Routes name it, and its records carry it in __source.name.

Examples
events-infirewall-syslog

Fields

Every record carries its text in _raw, unless its source's encoding is none. An Avro or SQL source gives its declared fields instead. Each kind of source adds fields of its own.

FieldTypeHolds
_rawstrThe text the source received, decoded with the source's encoding. A parser reads it. The destination delivers it with the record unless a step sets it to null. A source whose encoding is none gives no _raw.

Add a source

  1. Select the worker group and work on a branch.
  2. Open Build → Sources and choose + Add source.
  3. Choose how the group reads it (Pull, all workers, Pull, one worker, Push, all workers or Local, each worker), then its type. Give it a unique id and fill in its settings, then choose Add. Save the page.
  4. Connect it to a route, which sends its records through a pipeline to a destination. Follow the routing example.
  5. Review and deploy, then verify it.

Verify

  1. Check that the intended workers are online and run the deployed commit.
  2. Send a record you can recognize into the source's system.
  3. Check Status and Logs for source, routing or destination errors.
  4. Find the record at the destination and check its fields.

A preview of a saved sample checks the pipeline, not the live connection.

Troubleshooting

No records arrive
Follow No records arriving to check input, worker configuration, routing, filtering and delivery in order.
TLS verification fails
Check the server name and the authority in the connection identity. If the server uses a private CA, the connector must trust it.

See also