# Google Cloud Storage

Lists a prefix of one Cloud Storage bucket every poll interval, reads each new or changed object it finds, and splits it into records. Only one worker of each group reads this source at a time.

Use it for a bucket that cannot send Pub/Sub notifications, or one you may not configure. When the bucket can notify, use [Google Cloud Storage via Pub/Sub](/docs/sources/gcs-pubsub): every worker of the group reads, and nothing lists the bucket.

## Example

Reads each object under `app/` one line per record, signed in with the worker host's Application Default Credentials, and moves each object under `done/` once Smeltr delivers every record.

```
{
  "id": "bucket-export",
  "type": "gcsList",
  "bucket": "acme-logs",
  "prefix": "app/",
  "framing": "lines",
  "afterCommit": "move",
  "movePrefix": "done/"
}
```

![The GCS listing form on the Sources page: id bucket-export, bucket acme-logs, prefix app/, after commit move to done/, framing lines.](/gcs-list-source-form-light.png)

The same source in the Sources page's add dialog, as it is added.

## Before you start

-   Choose what happens to an object once Smeltr delivers it, `afterCommit`. `keep` leaves the bucket as it is. `delete` and `move` drain it, so each listing finds only what is still to read.
-   Give the service account it reads as a role on the bucket. `keep` needs to list and read objects: Storage Object Viewer. `delete` also needs to delete them, and `move` to create them as well, since a move copies the object and then deletes it: Storage Object User covers both. See [Cloud Storage IAM roles](https://cloud.google.com/storage/docs/access-control/iam-roles).
-   Choose how it signs in. With no key, it uses the worker host's [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials). Otherwise put a [service account key](https://cloud.google.com/iam/docs/keys-create-delete), the whole JSON file, in a [credential](/docs/secrets).

## Settings

It also takes the settings [every source](/docs/sources#settings) takes.

`bucket`  string required

The bucket's name, alone.

Examples:

- `acme-logs`
- `acme-audit-logs-prod`
- `logs.acme.example`

`prefix`  string default

Only objects whose name starts with it. It does not start with `/`. Empty is the whole bucket.

Examples:

- `app/`
- `cloudaudit.googleapis.com/activity/`
- `logs/2026/`

`serviceAccountKey`  credential default the host's Application Default Credentials

The [credential](/docs/secrets) holding a service account's JSON key. The account needs to list and read the bucket, and to delete or create objects in it for `delete` and `move`.

Examples:

- `GCS_KEY`
- `{"secret": "gcs-key","version": 1}`

`format`  string default each payload is text, in `_raw`

How each payload becomes the record.

`avro`

Reads Avro files. Fields you don't declare are kept.

`fields`  list of fields default none

The Avro fields to read, each with a `name` and a `type` such as `i64` or `str`. Name a nested field with dots, such as `user.id`. Top-level fields you don't list are kept. Only with format `avro`.

Example:

- `[{"name": "user_id","type": "i64"},{"name": "user.name","type": "str"}]`

`framing`  string default `lines`

How an object becomes records.

`lines`

Gives one record per line, or per piece `lineBreaker` cuts.

`object`

Gives the whole object as one record, for a JSON document such as CloudTrail's that a pipeline unrolls.

`escapedLines`

Gives one record per line, unescaped as a `bytes` object destination wrote it.

`lineBreaker`  string default a record per line

With `lines` framing, and only then: where a record ends. Its first capture group is the break and is dropped: the bytes before it end a record, line ends and all, and the bytes after it start the next. What follows the group is kept, so the pattern can say how a record starts: `([\r\n]+)\d{4}-` keeps a stack trace with the line that logged it, and `</event>(\s*)` cuts XML events with no line ends between them. It needs a capture group and must not match the empty string.

Examples:

- `([\r\n]+)\d{4}-`
- `</event>(\s*)`

`identity`  identity default none

The [connection identity](/docs/secrets#identities) it connects under, for a private CA or a client certificate.

Examples:

- `gcp-proxy`
- `corp-proxy`

`afterCommit`  string default `keep`

What it does with an object once every record is delivered.

`keep`

Leaves it and records it as read. Needs only read access, and costs a full listing per poll.

`delete`

Deletes it.

`move`

Copies it under `movePrefix` in place of `prefix`, then deletes it.

`movePrefix`  string default none

Where `move` puts an object, in place of `prefix`. Only with `move`, and outside `prefix`. It does not start with `/`, and ends in `/` when `prefix` does or is empty, and only then.

Examples:

- `done/`
- `archive/2026/`
- `processed/app/`

### Advanced

`start`  string default `everything`

What it reads first.

`everything`

Reads everything its first listing finds.

`new`

Reads only what arrives after its first listing. Needs `afterCommit` `keep`.

`pollIntervalSecs`  integer default `60`

Seconds from the start of one listing to the start of the next. At least 1.

Examples:

- `300`
- `30`
- `3600`

`endpoint`  string default the key's own cloud

Cloud Storage's address, such as an emulator's.

Examples:

- `http://localhost:4443`
- `https://storage-acme.p.googleapis.com`

`encoding`  string default `auto`

How a record's bytes become `_raw`'s text: `auto`, `none`, or a WHATWG encoding label other than the replacement ones, such as `utf-8`, `utf-16le` or `shift_jis`.

`auto`

Detects each record's encoding.

`utf-8`

Decodes every record as UTF-8, as any other label decodes as its own encoding. Bytes that do not decode become U+FFFD.

`none`

Decodes nothing: the record has no `_raw`, and its payload is `__source.bytes` alone. A pipeline reads it as text with `decode(__source.bytes, encoding)`.

## Fields it gives

| Field | Type | Holds |
| --- | --- | --- |
| `__source.name` | `str` | The name of the source that read the record. |
| `__source.type` | `str` | The kind of source that read it, its `type`, such as `kafka`, `s3List`, `httpApi` or `hecReceiver`. |
| `__source.bucket` | `str` | The bucket, or Azure container, the record's object is in. |
| `__source.key` | `str` | The key, or Azure blob name, of the object the record was read from. |
| `__source.offset` | `i64` | The record's offset in its Kafka or Event Hubs partition, or its record number in its object, from 0. |
| `__source.bytes` | `bytes` | The record's payload as the source read it, before any decoding. The source decodes `_raw` from it. A `bytes` destination writes it unless the destination names another field. Text functions refuse it; `decode(__source.bytes, encoding)` reads it as text. |

## How it works

Every `pollIntervalSecs`, counted from the start of the last listing, the source lists the objects whose names start with `prefix`. It reads each object the listing finds new, or changed since it last read it, and reads a changed object again whole. An empty object holds no record, and the source does not read it. The source decompresses a gzip, zstd, framed snappy or framed lz4 object first and reads any other as it is.

One worker of the group holds the source and reads it; every other worker running it reads nothing. The leader chooses the holder from the workers' heartbeats. The leader gives the hold to another worker after three heartbeat intervals without hearing from the holder, the holder's interval or the leader's, whichever is longer, and that worker resumes from the last progress the leader kept. That worker delivers again what the previous holder delivered after that progress.

With `afterCommit` `keep`, the progress is a **ledger**: each object read, by name and version. The holder records an object in it once Smeltr delivers every record of the object and reports it to the leader on its heartbeats, so the next holder reads only what is not in it. An object that leaves the bucket leaves the ledger. Every listing walks the whole prefix, so a large prefix costs a full listing each poll.

With `delete` or `move`, the bucket itself says what is still to read and there is no ledger. Once Smeltr delivers every record of an object, the source deletes it, or copies it to `movePrefix` in place of `prefix` and deletes the original. An object overwritten since the source read it stays in place, and the source reads it again. It deletes or moves an empty object unread.

`start` `everything` reads every object the first listing finds. `new` records the first listing as read and reads only what arrives or changes after it, so it needs `keep`.

The source's progress belongs to its name and to its bucket and prefix. Renamed, or pointed at another bucket or prefix, it starts afresh.

Once you remove the source and deploy, the leader forgets its progress, so a rollback that brings the source back starts it afresh.

An object handed out five times and never delivered is no longer offered until it changes. With `keep` it is recorded in the ledger as read; with `delete` or `move` it stays in the bucket. The worker log names it. With `delete` or `move` only the holder remembers it, so a restart or a new holder hands it out again.

## Verify

Upload an object with a line you can recognize under the prefix, then find that line at the destination after the next listing. On the **Sources** page, the source's **worker** column names the worker holding it, and with `keep` how many objects its ledger holds. With `delete` or `move`, the object leaves the prefix once it is delivered.

## Troubleshooting

**Nothing arrives**

Check that the object's name starts with `prefix`, and that the poll interval has passed. With `start` `new`, the source does not read what was there at the first listing until it changes. The worker log of the holder says why a listing failed, such as a missing role.

**The worker column is empty**

No worker holds the source: check that a worker of its group runs the deployment that has it.

**The holder was last heard some time ago**

The leader has not heard from that worker. It gives the hold to another worker after three heartbeat intervals without hearing from the holder, the holder's interval or the leader's, whichever is longer.

**An object stays under the prefix**

The source could not read it or could not delete it: check the role on the bucket and the worker log for the reason.

## See also

-   [Sources](/docs/sources)
-   [Google Cloud Storage via Pub/Sub](/docs/sources/gcs-pubsub)
-   [Amazon S3](/docs/sources/s3-list)
-   [Azure Blob Storage](/docs/sources/azure-blob-list)
-   [SFTP](/docs/sources/sftp)
-   [Google Cloud Storage destination](/docs/destinations/gcs)
-   [Recover and replay](/docs/delivery#recovery)