# Amazon S3 via SQS

Reads each object S3 announces on an SQS queue, splits it into records, and deletes the message once Smeltr delivers every record.

The source receives a bucket's object-created notifications from an SQS queue and reads the objects they name, from whichever bucket each one names. It reads a notification that reaches the queue directly from the bucket, through an SNS topic, or through an EventBridge rule.

## Example

Reads each announced object one line per record, with the worker host's AWS credentials.

```
{
  "id": "bucket-logs",
  "type": "s3Sqs",
  "queueUrl": "https://sqs.us-east-2.amazonaws.com/111122223333/smeltr-logs",
  "framing": "lines"
}
```

![The S3 via SQS form on the Sources page: id bucket-logs, its queue URL, framing lines.](/s3-sqs-source-form-light.png)

The same source on the Sources page.

## Before you start

-   Create an SQS queue in the bucket's region, and have the bucket's object-created events reach it by one of these:
    -   **Directly**: AWS describes it in [Configuring a bucket for notifications](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ways-to-add-notification-config-to-bucket.html).
    -   **Through SNS**: send the bucket's events to an SNS topic and [subscribe the queue to the topic](https://docs.aws.amazon.com/sns/latest/dg/subscribe-sqs-queue-to-sns-topic.html). Raw message delivery may be on or off.
    -   **Through EventBridge**: [turn on EventBridge](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-event-notifications-eventbridge.html) for the bucket, and add a rule that sends its `Object Created` events to the queue.
-   Give the queue a [dead-letter queue](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html), so a message the source cannot use stops returning: one whose object it cannot read, or one that is not about S3 at all, such as another message on the same SNS topic.
-   Allow the credentials Smeltr uses `sqs:ReceiveMessage`, `sqs:ChangeMessageVisibility` and `sqs:DeleteMessage` on the queue, and `s3:GetObject` on the buckets. A versioned bucket also needs `s3:GetObjectVersion`. Allow `sqs:GetQueueAttributes` too for the source to report the queue's depth as its lag; without it the source reads the same, and reports no lag.

## Settings

It also takes the settings [every source](/docs/sources#settings) takes.

`queueUrl`  string required

The queue's URL, which names its region and account.

Examples:

- `https://sqs.us-east-2.amazonaws.com/111122223333/smeltr-logs`
- `https://sqs.us-east-2.amazonaws.com/444455556666/cloudtrail-events`
- `http://localhost:4566/000000000000/smeltr-logs`

`accessKeyId`  credential default the host's credentials

The [credential](/docs/secrets) holding the access key's id. Set both `accessKeyId` and `secretAccessKey`, or neither.

Examples:

- `AWS_ACCESS_KEY_ID`
- `{"secret": "aws-access-key-id","version": 1}`

`secretAccessKey`  credential default the host's credentials

The [credential](/docs/secrets) holding the access key's secret.

Examples:

- `AWS_SECRET_ACCESS_KEY`
- `{"secret": "aws-secret-access-key","version": 1}`

`format`  string default each payload is text, in `_raw`

How each payload becomes the record.

`avro`

Reads Avro files. Fields you don't declare are kept.

`fields`  list of fields default none

The Avro fields to read, each with a `name` and a `type` such as `i64` or `str`. Name a nested field with dots, such as `user.id`. Top-level fields you don't list are kept. Only with format `avro`.

Example:

- `[{"name": "user_id","type": "i64"},{"name": "user.name","type": "str"}]`

`framing`  string default `lines`

How an object becomes records.

`lines`

Gives one record per line, or per piece `lineBreaker` cuts.

`object`

Gives the whole object as one record, for a JSON document such as CloudTrail's that a pipeline unrolls.

`escapedLines`

Gives one record per line, unescaped as a `bytes` object destination wrote it.

`lineBreaker`  string default a record per line

With `lines` framing, and only then: where a record ends. Its first capture group is the break and is dropped: the bytes before it end a record, line ends and all, and the bytes after it start the next. What follows the group is kept, so the pattern can say how a record starts: `([\r\n]+)\d{4}-` keeps a stack trace with the line that logged it, and `</event>(\s*)` cuts XML events with no line ends between them. It needs a capture group and must not match the empty string.

Examples:

- `([\r\n]+)\d{4}-`
- `</event>(\s*)`

`identity`  identity default none

The [connection identity](/docs/secrets#identities) it connects under, for a private CA or a client certificate.

Examples:

- `aws-proxy`
- `corp-proxy`

### Advanced

`region`  string default the queue URL's, else the host's

The region of the queue and its buckets. S3 notifies only a queue in its bucket's region.

Examples:

- `us-east-2`
- `eu-west-1`

`endpoint`  string default the host's, else AWS

An S3-compatible service's address. Without an access key, the worker host's AWS settings, such as `AWS_ENDPOINT_URL`, come first.

Examples:

- `https://minio.example.com:9000`
- `http://localhost:9000`
- `https://objects.example.com`

`roleArn`  string default none

A role to assume, for a queue or bucket in another account.

Examples:

- `arn:aws:iam::111122223333:role/smeltr-reader`
- `arn:aws-us-gov:iam::444455556666:role/log-archive-reader`

`externalId`  string default none

The external id the role requires. Only with `roleArn`.

Examples:

- `smeltr`
- `7f3c1a9e-2b4d-4e8f-9a61-0c5d3e2b1f47`

`authEndpoint`  string default the region's own

Where the role in `roleArn` is assumed. Only with `roleArn`.

Examples:

- `https://sts.us-east-2.amazonaws.com`
- `https://sts.us-gov-west-1.amazonaws.com`
- `https://sts.amazonaws.com`

`encoding`  string default `auto`

How a record's bytes become `_raw`'s text: `auto`, `none`, or a WHATWG encoding label other than the replacement ones, such as `utf-8`, `utf-16le` or `shift_jis`.

`auto`

Detects each record's encoding.

`utf-8`

Decodes every record as UTF-8, as any other label decodes as its own encoding. Bytes that do not decode become U+FFFD.

`none`

Decodes nothing: the record has no `_raw`, and its payload is `__source.bytes` alone. A pipeline reads it as text with `decode(__source.bytes, encoding)`.

`visibilitySecs`  integer default `60`

Seconds a received message stays hidden from other readers. The source extends it while it reads the message's objects. 1 to 43200.

Examples:

- `300`
- `30`
- `43200`

## Fields it gives

| Field | Type | Holds |
| --- | --- | --- |
| `__source.name` | `str` | The name of the source that read the record. |
| `__source.type` | `str` | The kind of source that read it, its `type`, such as `kafka`, `s3List`, `httpApi` or `hecReceiver`. |
| `__source.bucket` | `str` | The bucket, or Azure container, the record's object is in. |
| `__source.key` | `str` | The key, or Azure blob name, of the object the record was read from. |
| `__source.offset` | `i64` | The record's offset in its Kafka or Event Hubs partition, or its record number in its object, from 0. |
| `__source.bytes` | `bytes` | The record's payload as the source read it, before any decoding. The source decodes `_raw` from it. A `bytes` destination writes it unless the destination names another field. Text functions refuse it; `decode(__source.bytes, encoding)` reads it as text. |

## How it works

The source reads the object version the notification names, and decodes a URL-encoded key. The source deletes a message that announces anything other than a created object without reading anything. A message that is not about S3 at all stays on the queue, like one whose object the source cannot read.

The source decompresses a gzip, zstd, framed snappy or framed lz4 object first and reads any other as it is.

The source keeps a message hidden while it reads the message's objects and Smeltr delivers them, and deletes it once Smeltr commits every record of every object it names. It never deletes a message whose object it cannot read.

## Verify

Upload an object with a line you can recognize to the bucket, then find that line at the destination. The queue's message count returns to zero once Smeltr delivers it.

## Troubleshooting

**Nothing arrives**

Check that the bucket's object-created events reach this queue, and that the queue's access policy lets the sender in: S3 when the bucket sends directly, SNS when it goes through a topic, whose own access policy must also let S3 publish to it, EventBridge when it goes through a rule. For EventBridge, also check that the bucket has EventBridge turned on.

**The same object is read again and again**

The source cannot read the object: check `s3:GetObject` on its bucket and the worker log for the reason. A dead-letter queue stops the repeats.

## See also

-   [Sources](/docs/sources)
-   [Amazon S3 destination](/docs/destinations/s3)
-   [Recover and replay](/docs/delivery#recovery)