# Google Cloud Storage

Writes batches of records as objects in a Cloud Storage bucket.

## Example

Reads the service account key from the worker-host variable `GCS_SERVICE_ACCOUNT_KEY`.

```
{
  "id": "archive",
  "type": "gcs",
  "bucket": "acme-logs",
  "serviceAccountKey": "GCS_SERVICE_ACCOUNT_KEY"
}
```

![The Google Cloud Storage form on the Destinations page: id archive, bucket acme-logs, format ndjson.](/gcs-destination-form-light.png)

A Google Cloud Storage destination on the Destinations page, signed in with the host's Application Default Credentials.

## Before you start

-   [Create the bucket](https://cloud.google.com/storage/docs/creating-buckets).
-   Give the service account the destination writes as permission to create objects in the bucket, such as the Storage Object Creator role. See [Cloud Storage IAM roles](https://cloud.google.com/storage/docs/access-control/iam-roles).
-   Choose how it signs in. With no key, it uses the worker host's [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials). Otherwise put a [service account key](https://cloud.google.com/iam/docs/keys-create-delete), the whole JSON file, in a [credential](/docs/secrets).

## Settings

It also takes the settings [every destination](/docs/destinations#settings) takes.

`bucket`  string required

The bucket.

Examples:

- `acme-logs`
- `acme-logs-eu`

`serviceAccountKey`  credential default the host's Application Default Credentials

The [credential](/docs/secrets) holding a service account's JSON key.

Examples:

- `GCS_KEY`
- `{"secret": "gcs-key","version": 1}`

`prefix`  string default `%Y/%m/%d/%H/`

What each object's key starts with, for a record with no `__destination.prefix`. `%Y`, `%m`, `%d`, `%H` and `%M` are the write time, UTC, and no other `%` is allowed. A key never starts with `/`.

Examples:

- `events/%Y/%m/%d/`
- `firewall/dt=%Y-%m-%d/hour=%H/`
- `acme/audit/`

`format`  string default `ndjson`

What each object holds.

`ndjson`

Writes one JSON document per line.

`json`

Writes one JSON array of every record.

`parquet`

Writes one Parquet file.

`avro`

Writes one Avro file.

`csv`

Writes one CSV file.

`text`

Writes one line of text per record, from `field`.

`bytes`

Writes each source record's bytes as read, before pipeline processing, one per line: `__source.bytes` unless `field` names another `bytes` field.

`field`  string default `_raw` for `text`, `__source.bytes` for `bytes`

The field each line of a `text` or `bytes` object holds. Only with format `text` or `bytes`.

Example:

- `message`

`newlines`  string default `escape`

What a `bytes` object does with a record holding a line break. Only with format `bytes`.

`escape`

Writes `\`, newline and carriage return as `\\`, `\n` and `\r`, which a source with framing `escapedLines` reverses.

`refuse`

Refuses the record.

`identity`  identity default none

The [connection identity](/docs/secrets#identities) it connects under, for a private CA or a client certificate.

Examples:

- `store-client`
- `corp-proxy`

### Advanced: Endpoint and delivery

`endpoint`  string default the key's own cloud

Another address, such as an emulator's.

Examples:

- `http://localhost:4443`
- `https://storage.example.com`

`compression`  string default `gzip`

How it compresses each object. A `parquet` object compresses each column inside the file instead, and its key ends with `.parquet` whatever the compression. An `avro` file compresses inside the file, and its key ends with `.avro`. Its `gzip` is Avro's `deflate`. Avro has no `lz4`.

`none`

Writes it uncompressed. The key ends with the format's own extension, such as `.ndjson`.

`gzip`

Compresses it with gzip and appends `.gz` to the key, such as `.ndjson.gz`.

`snappy`

Compresses it with framed Snappy and appends `.sz` to the key, such as `.ndjson.sz`.

`lz4`

Compresses it with framed LZ4 and appends `.lz4` to the key, such as `.ndjson.lz4`.

`zstd`

Compresses it with Zstandard and appends `.zst` to the key, such as `.ndjson.zst`.

`storageClass`  string default the bucket's own

The class each object is stored in, one of Cloud Storage's own: `STANDARD`, `NEARLINE`, `COLDLINE` or `ARCHIVE`.

Examples:

- `NEARLINE`
- `COLDLINE`

`inFlight`  integer default `4`

How many parts of one file upload at once. 1 to 64.

Examples:

- `8`
- `1`
- `2`

`maxObjectBytes`  size default `64 MiB`

The largest file it puts in the bucket, compressed. Set it to what the tools that read the bucket handle best. 64 KiB to 64 GiB.

Examples:

- `128 MiB`
- `2 GiB`
- `32 MB`

`bufferBytes`  size default 256 MiB

Memory it may use on each worker for records not yet in the bucket. Files are finished early when it fills, so files reach `maxObjectBytes` only when it holds that many records before compression. 4 MiB to 32 TiB.

Examples:

- `1 GiB`
- `4 GiB`

`stallAfterSecs`  integer default `30`

Seconds one call to the store, or one finishing of the files, may take before it counts as stalled. 1 to 120, and at least 15 on SFTP.

Examples:

- `60`
- `120`

`fileIntervalSecs`  integer default `60`

The longest records wait, in seconds, before they land in the bucket. A longer wait makes larger files; records are acknowledged only once they land, so a source is held open up to this long, and after a crash reads that much again. 5 to 3600.

Examples:

- `30`
- `240`

## Fields it reads

| Field | Type | Holds |
| --- | --- | --- |
| `__destination.prefix` | `str` | What an S3, Google Cloud Storage or Azure Blob Storage destination's object key starts with, taken as it is. Null means the destination's own prefix, with its time placeholders filled in. |

## How it works

The destination keeps one object open per prefix, and for Parquet, Avro and CSV per record type of the routes feeding it, across batches, named `<prefix><random id>.<extension>`, and finishes it when it reaches `maxObjectBytes` on disk, when `fileIntervalSecs` passes, when `bufferBytes` fills on the worker, or when a 65th object opens and this one was written least recently. A batch is acknowledged once every object holding its records has landed. Two objects never share a key, so records written again after a failure make new objects.

## Verify

Send a record with a unique marker through the source, then list the bucket under the prefix for a new object and look for the marker in it.

## Troubleshooting

**Permission denied**

Check that the service account may create objects in the bucket.

## See also

-   [Destinations](/docs/destinations)
-   [Amazon S3 destination](/docs/destinations/s3)
-   [Azure Blob Storage destination](/docs/destinations/azure-blob)
-   [Secrets & credentials](/docs/secrets)