Amazon S3 via SQS

Reads each object S3 announces on an SQS queue, splits it into records, and deletes the message once Smeltr delivers every record.

The source receives a bucket's object-created notifications from an SQS queue and reads the objects they name, from whichever bucket each one names. It reads a notification that reaches the queue directly from the bucket, through an SNS topic, or through an EventBridge rule.

Example

Reads each announced object one line per record, with the worker host's AWS credentials.

{
  "id": "bucket-logs",
  "type": "s3Sqs",
  "queueUrl": "https://sqs.us-east-2.amazonaws.com/111122223333/smeltr-logs",
  "framing": "lines"
}

The S3 via SQS form on the Sources page: id bucket-logs, its queue URL, framing lines. — The same source on the Sources page.

Before you start

Create an SQS queue in the bucket's region, and have the bucket's object-created events reach it by one of these:
- Directly: AWS describes it in Configuring a bucket for notifications.
- Through SNS: send the bucket's events to an SNS topic and subscribe the queue to the topic. Raw message delivery may be on or off.
- Through EventBridge: turn on EventBridge for the bucket, and add a rule that sends its Object Created events to the queue.
Give the queue a dead-letter queue, so a message the source cannot use stops returning: one whose object it cannot read, or one that is not about S3 at all, such as another message on the same SNS topic.
Allow the credentials Smeltr uses sqs:ReceiveMessage, sqs:ChangeMessageVisibility and sqs:DeleteMessage on the queue, and s3:GetObject on the buckets. A versioned bucket also needs s3:GetObjectVersion. Allow sqs:GetQueueAttributes too for the source to report the queue's depth as its lag; without it the source reads the same, and reports no lag.

Settings

It also takes the settings every source takes.

queueUrl string required

The queue's URL, which names its region and account.

Examples

https://sqs.us-east-2.amazonaws.com/111122223333/smeltr-logshttps://sqs.us-east-2.amazonaws.com/444455556666/cloudtrail-eventshttp://localhost:4566/000000000000/smeltr-logs

accessKeyId credential default the host's credentials

The credential holding the access key's id. Set both accessKeyId and secretAccessKey, or neither.

Examples

AWS_ACCESS_KEY_ID{"secret": "aws-access-key-id","version": 1}

secretAccessKey credential default the host's credentials

The credential holding the access key's secret.

Examples

AWS_SECRET_ACCESS_KEY{"secret": "aws-secret-access-key","version": 1}

format string default each payload is text, in _raw

How each payload becomes the record.

avro: Reads Avro files. Fields you don't declare are kept.

fields list of fields default none

The Avro fields to read, each with a name and a type such as i64 or str. Name a nested field with dots, such as user.id. Top-level fields you don't list are kept. Only with format avro.

Example

[{"name": "user_id","type": "i64"},{"name": "user.name","type": "str"}]

framing string default lines

How an object becomes records.

lines: Gives one record per line, or per piece lineBreaker cuts.
object: Gives the whole object as one record, for a JSON document such as CloudTrail's that a pipeline unrolls.
escapedLines: Gives one record per line, unescaped as a bytes object destination wrote it.

lineBreaker string default a record per line

With lines framing, and only then: where a record ends. Its first capture group is the break and is dropped: the bytes before it end a record, line ends and all, and the bytes after it start the next. What follows the group is kept, so the pattern can say how a record starts: ([\r\n]+)\d{4}- keeps a stack trace with the line that logged it, and </event>(\s*) cuts XML events with no line ends between them. It needs a capture group and must not match the empty string.

Examples

([\r\n]+)\d{4}-</event>(\s*)

identity identity default none

The connection identity it connects under, for a private CA or a client certificate.

Examples

aws-proxycorp-proxy

Advanced

region string default the queue URL's, else the host's

The region of the queue and its buckets. S3 notifies only a queue in its bucket's region.

Examples

us-east-2eu-west-1

endpoint string default the host's, else AWS

An S3-compatible service's address. Without an access key, the worker host's AWS settings, such as AWS_ENDPOINT_URL, come first.

Examples

https://minio.example.com:9000http://localhost:9000https://objects.example.com

roleArn string default none

A role to assume, for a queue or bucket in another account.

Examples

arn:aws:iam::111122223333:role/smeltr-readerarn:aws-us-gov:iam::444455556666:role/log-archive-reader

externalId string default none

The external id the role requires. Only with roleArn.

Examples

smeltr7f3c1a9e-2b4d-4e8f-9a61-0c5d3e2b1f47

authEndpoint string default the region's own

Where the role in roleArn is assumed. Only with roleArn.

Examples

https://sts.us-east-2.amazonaws.comhttps://sts.us-gov-west-1.amazonaws.comhttps://sts.amazonaws.com

encoding string default auto

How a record's bytes become _raw's text: auto, none, or a WHATWG encoding label other than the replacement ones, such as utf-8, utf-16le or shift_jis.

auto: Detects each record's encoding.
utf-8: Decodes every record as UTF-8, as any other label decodes as its own encoding. Bytes that do not decode become U+FFFD.
none: Decodes nothing: the record has no _raw, and its payload is __source.bytes alone. A pipeline reads it as text with decode(__source.bytes, encoding).

visibilitySecs integer default 60

Seconds a received message stays hidden from other readers. The source extends it while it reads the message's objects. 1 to 43200.

Examples

3003043200

Fields it gives

Field	Type	Holds
`__source.name`	`str`	The name of the source that read the record.
`__source.type`	`str`	The kind of source that read it, its `type`, such as `kafka`, `s3List`, `httpApi` or `hecReceiver`.
`__source.bucket`	`str`	The bucket, or Azure container, the record's object is in.
`__source.key`	`str`	The key, or Azure blob name, of the object the record was read from.
`__source.offset`	`i64`	The record's offset in its Kafka or Event Hubs partition, or its record number in its object, from 0.
`__source.bytes`	`bytes`	The record's payload as the source read it, before any decoding. The source decodes `_raw` from it. A `bytes` destination writes it unless the destination names another field. Text functions refuse it; `decode(__source.bytes, encoding)` reads it as text.

How it works

The source reads the object version the notification names, and decodes a URL-encoded key. The source deletes a message that announces anything other than a created object without reading anything. A message that is not about S3 at all stays on the queue, like one whose object the source cannot read.

The source decompresses a gzip, zstd, framed snappy or framed lz4 object first and reads any other as it is.

The source keeps a message hidden while it reads the message's objects and Smeltr delivers them, and deletes it once Smeltr commits every record of every object it names. It never deletes a message whose object it cannot read.

Verify

Upload an object with a line you can recognize to the bucket, then find that line at the destination. The queue's message count returns to zero once Smeltr delivers it.

Troubleshooting

Nothing arrives: Check that the bucket's object-created events reach this queue, and that the queue's access policy lets the sender in: S3 when the bucket sends directly, SNS when it goes through a topic, whose own access policy must also let S3 publish to it, EventBridge when it goes through a rule. For EventBridge, also check that the bucket has EventBridge turned on.
The same object is read again and again: The source cannot read the object: check s3:GetObject on its bucket and the worker log for the reason. A dead-letter queue stops the repeats.

Amazon S3 via SQS

Example

Before you start

Settings

Advanced

Fields it gives

How it works

Verify

Troubleshooting

See also