# Azure Blob Storage

Lists a prefix of one container every poll interval, reads each blob it finds new or changed, and splits it into records. Only one worker of each group reads this source at a time.

Use it where the storage account cannot notify. Where it can, [Azure Blob via Storage queue](/docs/sources/azure-blob-queue) reads each blob as the storage account announces it, with no listing, and any number of workers share the work.

## Example

Reads each blob under `app/` one line per record, as the worker host's managed identity, and moves it under `done/` once Smeltr delivers it.

```
{
  "id": "container-export",
  "type": "azureBlobList",
  "account": "acmelogs",
  "container": "logs",
  "prefix": "app/",
  "framing": "lines",
  "afterCommit": "move",
  "movePrefix": "done/"
}
```

![The Azure Blob listing form on the Sources page: id container-export, account acmelogs, container logs, prefix app/, after commit move to done/, framing lines.](/azure-blob-list-source-form-light.png)

The same source in the Sources page's add dialog, as it is added.

## Before you start

-   Choose what it does with a blob once Smeltr delivers it, `afterCommit`. That sets the permission it needs: `keep` only lists and reads; `delete` also deletes; `move` also writes the blob under `movePrefix`, then deletes it.
-   Choose how it signs in, and give that identity that permission on the container. A key, signature or client secret goes in a [credential](/docs/secrets).
    -   the worker host's [managed identity](https://learn.microsoft.com/entra/identity/managed-identities-azure-resources/overview), or a [service principal](https://learn.microsoft.com/entra/identity-platform/howto-create-service-principal-portal), given Storage Blob Data Reader on the container for `keep`, or Storage Blob Data Contributor for `delete` and `move` (see [assign a role for data access](https://learn.microsoft.com/azure/storage/blobs/assign-azure-role-data-access));
    -   an [account key](https://learn.microsoft.com/azure/storage/common/storage-account-keys-manage); or
    -   a [shared access signature](https://learn.microsoft.com/azure/storage/common/storage-sas-overview) for the blob service, allowed to list and read, and also to delete for `delete`, or to create, write and delete for `move`.

## Settings

It also takes the settings [every source](/docs/sources#settings) takes.

`container`  string required

The container's name, alone.

Examples:

- `logs`
- `insights-logs-auditevent`

`account`  string required

The storage account holding the container.

Examples:

- `acmelogs`
- `acmearchive01`

`prefix`  string default

Only blobs whose name starts with it. It does not start with `/`. Empty is the whole container.

Examples:

- `app/`
- `resourceId=/SUBSCRIPTIONS/`
- `logs/2026/`

`accessKey`  credential default none

The [credential](/docs/secrets) holding the account's access key. Set at most one of `accessKey`, `sasToken`, `clientSecret` and `managedIdentityClientId`; with none, the host's managed or workload identity is used.

Examples:

- `AZURE_ACCESS_KEY`
- `{"secret": "azure-access-key","version": 1}`

`sasToken`  credential default none

The [credential](/docs/secrets) holding a shared access signature that covers the blob service.

Examples:

- `AZURE_SAS_TOKEN`
- `{"secret": "azure-sas-token","version": 1}`

`tenantId`  string default none

A service principal's tenant. Set `tenantId`, `clientId` and `clientSecret` together, or none of them.

Example:

- `00000000-0000-0000-0000-000000000000`

`clientId`  string default none

A service principal's client id.

Example:

- `11111111-1111-1111-1111-111111111111`

`clientSecret`  credential default none

The [credential](/docs/secrets) holding a service principal's client secret.

Examples:

- `AZURE_CLIENT_SECRET`
- `{"secret": "azure-client-secret","version": 1}`

`managedIdentityClientId`  string default the host's managed or workload identity

The client id of the user-assigned managed identity to sign in as. Set it when the worker's host has more than one managed identity. Left out, it signs in as the host's managed identity, or an AKS pod's workload identity.

Example:

- `22222222-2222-2222-2222-222222222222`

`format`  string default each payload is text, in `_raw`

How each payload becomes the record.

`avro`

Reads Avro files. Fields you don't declare are kept.

`fields`  list of fields default none

The Avro fields to read, each with a `name` and a `type` such as `i64` or `str`. Name a nested field with dots, such as `user.id`. Top-level fields you don't list are kept. Only with format `avro`.

Example:

- `[{"name": "user_id","type": "i64"},{"name": "user.name","type": "str"}]`

`framing`  string default `lines`

How an object becomes records.

`lines`

Gives one record per line, or per piece `lineBreaker` cuts.

`object`

Gives the whole object as one record, for a JSON document such as CloudTrail's that a pipeline unrolls.

`escapedLines`

Gives one record per line, unescaped as a `bytes` object destination wrote it.

`lineBreaker`  string default a record per line

With `lines` framing, and only then: where a record ends. Its first capture group is the break and is dropped: the bytes before it end a record, line ends and all, and the bytes after it start the next. What follows the group is kept, so the pattern can say how a record starts: `([\r\n]+)\d{4}-` keeps a stack trace with the line that logged it, and `</event>(\s*)` cuts XML events with no line ends between them. It needs a capture group and must not match the empty string.

Examples:

- `([\r\n]+)\d{4}-`
- `</event>(\s*)`

`identity`  identity default none

The [connection identity](/docs/secrets#identities) it connects under, for a private CA or a client certificate.

Examples:

- `azure-proxy`
- `corp-proxy`

`afterCommit`  string default `keep`

What it does with a blob once every record is delivered.

`keep`

Leaves it and records it as read. Needs only read access, and costs a full listing per poll.

`delete`

Deletes it.

`move`

Copies it under `movePrefix` in place of `prefix`, then deletes it.

`movePrefix`  string default none

Where `move` puts a blob, in place of `prefix`. Only with `move`, and outside `prefix`. It does not start with `/`, and ends in `/` when `prefix` does or is empty, and only then.

Examples:

- `done/`
- `archive/2026/`
- `processed/app/`

### Advanced

`start`  string default `everything`

What it reads first.

`everything`

Reads everything its first listing finds.

`new`

Reads only what arrives after its first listing. Needs `afterCommit` `keep`.

`pollIntervalSecs`  integer default `60`

Seconds from the start of one listing to the start of the next. At least 1.

Examples:

- `300`
- `30`
- `3600`

`endpoint`  string default the account's, in Azure's public cloud

The blob service's address, such as Azurite's or a sovereign cloud's.

Examples:

- `http://localhost:10000/acmelogs`
- `https://acmelogs.blob.core.usgovcloudapi.net`

`authEndpoint`  string default its cloud's own

Where a client secret is exchanged for a token. Only with `clientSecret`.

Examples:

- `https://login.microsoftonline.us`
- `https://login.chinacloudapi.cn`

`encoding`  string default `auto`

How a record's bytes become `_raw`'s text: `auto`, `none`, or a WHATWG encoding label other than the replacement ones, such as `utf-8`, `utf-16le` or `shift_jis`.

`auto`

Detects each record's encoding.

`utf-8`

Decodes every record as UTF-8, as any other label decodes as its own encoding. Bytes that do not decode become U+FFFD.

`none`

Decodes nothing: the record has no `_raw`, and its payload is `__source.bytes` alone. A pipeline reads it as text with `decode(__source.bytes, encoding)`.

## Fields it gives

`__source.bucket` holds the container, and `__source.key` the blob's name.

| Field | Type | Holds |
| --- | --- | --- |
| `__source.name` | `str` | The name of the source that read the record. |
| `__source.type` | `str` | The kind of source that read it, its `type`, such as `kafka`, `s3List`, `httpApi` or `hecReceiver`. |
| `__source.bucket` | `str` | The bucket, or Azure container, the record's object is in. |
| `__source.key` | `str` | The key, or Azure blob name, of the object the record was read from. |
| `__source.offset` | `i64` | The record's offset in its Kafka or Event Hubs partition, or its record number in its object, from 0. |
| `__source.bytes` | `bytes` | The record's payload as the source read it, before any decoding. The source decodes `_raw` from it. A `bytes` destination writes it unless the destination names another field. Text functions refuse it; `decode(__source.bytes, encoding)` reads it as text. |

## How it works

Every `pollIntervalSecs` it lists the blobs whose names start with `prefix`. The source reads a blob once two listings in a row find it unchanged, since an append blob grows while something writes it. It reads the blob whole: it decompresses a gzip, zstd, framed snappy or framed lz4 blob first and reads any other as it is. An empty blob holds no record, and the source does not read it.

One worker of the group reads it at a time, the one the leader holds it for; the others read nothing. The leader gives the hold to another worker after three heartbeat intervals without hearing from the holder, the holder's interval or the leader's, whichever is longer. With `keep`, its progress is a **ledger**: the blobs it has read, each with the version it read. The holder reports it on its heartbeat and the leader keeps it; the next holder fetches the ledger and reads only what is not in it. The next holder delivers again what the previous holder delivered after the last progress the leader kept. With `delete` or `move` the source keeps no progress: the next holder reads what the container still holds, so it delivers again a blob delivered but not yet deleted or moved.

`afterCommit` says what it does with a blob once Smeltr delivers every record.

-   `keep` leaves the blob and records it in the ledger. When a blob changes after the source reads it, the source reads it again whole. A blob that is gone from the listing leaves the ledger. Every listing lists the whole prefix, so the cost of each poll grows with what the prefix holds.
-   `delete` deletes the blob, and `move` copies it under `movePrefix` in place of `prefix` and deletes it. The container is then its own record of what is left to read, and the source keeps no ledger. A blob that changed after the source read it stays in place, and the source reads it again. It deletes or moves an empty blob unread.

A blob handed out five times and never delivered is no longer read until it changes. With `keep` it is recorded in the ledger as read; with `delete` or `move` it stays in the container. The worker log says which blob. With `delete` or `move` only the holder remembers it, so a restart or a new holder hands it out again.

`start` says what the first listing reads. `everything` reads every blob it finds. `new` records every blob it finds as read and reads only what arrives or changes after it; it needs `afterCommit` `keep`, since only the ledger remembers what was skipped.

The source's progress belongs to its name and to its storage account, container and prefix. Renamed, or pointed at another container or prefix, it starts afresh.

Once you remove the source and deploy, the leader forgets its progress, so a rollback that brings the source back starts it afresh.

## Verify

Upload a blob with a line you can recognize under the prefix, then find that line at the destination after two listings. On the Sources page, the source's **worker** column names the worker that holds it, and with `keep` how many blobs its ledger holds. With `delete` or `move`, the blob leaves the prefix once it is delivered.

## Troubleshooting

**Nothing arrives**

The source reads a blob only on the second listing that finds it unchanged, so wait two `pollIntervalSecs`. Check that its name starts with `prefix`, which does not start with `/`, and the worker log for a listing that failed.

**The worker column is empty**

No worker of the group runs the source, or none has reported it to the leader. Check that the group's workers are enrolled and running the deployment.

**A blob is read again**

It changed after the source read it, or the source moved to another worker before its last progress reached the leader. Both deliver its records again.

**Blobs stay in the container with delete or move**

The identity cannot delete or write there: check its role or signature, and the worker log for the reason.

## See also

-   [Sources](/docs/sources)
-   [Azure Blob via Storage queue](/docs/sources/azure-blob-queue)
-   [Amazon S3](/docs/sources/s3-list)
-   [Google Cloud Storage](/docs/sources/gcs-list)
-   [SFTP](/docs/sources/sftp)
-   [Azure Blob Storage destination](/docs/destinations/azure-blob)
-   [Recover and replay](/docs/delivery#recovery)