# Elasticsearch

Writes each record as a JSON document through Elasticsearch's Bulk API, many documents to a request.

## Example

Authorizes with the worker-host variable `ELASTIC_AUTH`, which holds the complete Authorization header. Replace the URL and index.

```
{
  "id": "search-out",
  "type": "elasticsearch",
  "url": "https://elasticsearch.example.com:9200",
  "defaultIndex": "smeltr-events",
  "auth": "ELASTIC_AUTH"
}
```

## Before you start

-   Prepare the target index or data stream and its mappings, or let the account create them.
-   Give an account permission to write there, and put its complete Authorization header, such as `ApiKey <encoded-key>`, in a [credential](/docs/secrets).
-   Select a [connection identity](/docs/secrets#identities) for a private CA or a client certificate.

## Settings

It also takes the settings [every destination](/docs/destinations#settings) takes.

`url`  string required

The cluster's base address.

Examples:

- `https://elasticsearch.example.com:9200`
- `https://10.20.0.5:9200`

`defaultIndex`  string required

The index for a record with no `__destination.index`.

Examples:

- `orders`
- `logs-firewall-default`

`identity`  identity default none

The [connection identity](/docs/secrets#identities) it connects under, for a private CA or a client certificate.

Examples:

- `elastic-client`
- `elastic-mtls`

`auth`  credential default no Authorization header

The [credential](/docs/secrets) holding the whole `Authorization` header value, such as `ApiKey <encoded-key>`.

Examples:

- `ELASTIC_AUTH`
- `{"secret": "elastic-auth","version": 1}`

`docId`  string default `auto`

Who chooses each document's id. See [Duplicate documents](/docs/destinations/elasticsearch#duplicate-documents).

`auto`

Lets Elasticsearch choose each document's id.

`dedupe`

Keeps the first document written under an id.

`overwrite`

Replaces the document written under an id.

`idField`  string default `__destination.id`

The field holding a document's id under `dedupe` or `overwrite`.

Examples:

- `event.id`
- `request_id`

### Advanced

`inFlight`  integer default `4`

How many bulk requests it sends at once to start with. 1 to 64.

Examples:

- `8`
- `1`
- `16`

`maxBulkBytes`  size default `5 MiB`

The most one bulk request carries. A record larger than this still goes as a single record. 64 KiB to 100 MiB.

Examples:

- `10 MiB`
- `512 KiB`
- `15MB`

`bufferBytes`  size default 64 MiB, or one bulk request if that is larger

How much undelivered output it may hold. 4 MiB to 1 GiB, and at least one bulk request.

Examples:

- `256 MiB`
- `128 MiB`
- `1 GiB`

`stallAfterSecs`  integer default `30`

Seconds one batch may take to deliver, every retry included, before it counts as stalled. 1 to 120.

Examples:

- `60`
- `120`

`zoneUrls`  map default none

An address per zone. A worker in a listed [zone](/docs/reference#settings) sends there instead of `url`.

Examples:

- `us-east-1a = https://es-a.example.com:9200`
- `us-east-1a = https://es-a.example.com:9200`
- `us-east-1b = https://es-b.example.com:9200`

`sniff`  bool default `false`

Learns the cluster's nodes from `url` and spreads requests across them, preferring the worker's [zone](/docs/reference#settings). Every node must be reachable from the workers, and the credential needs the `monitor` cluster privilege.

`zoneAttribute`  string default `zone`

The node attribute that names a node's zone, set on each node as `node.attr.<name>`. Requires `sniff` enabled.

Examples:

- `rack`
- `availability_zone`

## Fields it reads

| Field | Type | Holds |
| --- | --- | --- |
| `__destination.index` | `str` | The index an Elasticsearch or Splunk HEC destination files the record under. Null means the destination's default index. A Splunk HEC destination on the `raw` endpoint refuses a record naming an index other than its own. |
| `__destination.id` | `str` | The Elasticsearch document ID, used when the destination's document ID setting is `dedupe` or `overwrite`. When it is null, the destination's id field names the field that gives it, and with neither the document is sent with no ID. |

## How it works

### Duplicate documents

| Policy | Behavior |
| --- | --- |
| auto (default) | Sends no document ID. A record resent after a crash can create a second document. |
| dedupe | Use a stable ID and keep the first document written with that ID. |
| overwrite | Replace the document with that ID. Concurrent requests can arrive out of order. This policy cannot write to a data stream. |

For dedupe or overwrite, set `idField` to a stable field such as `event.id`, or have the pipeline set `__destination.id`. Keep auto when records have no stable ID.

## Verify

Send a record with a unique marker through the source, then search the index for it after a refresh. Check the document's fields and the worker's delivery errors.

## Troubleshooting

**401 or 403**

Check the full Authorization header, that each worker has the secret, and permissions on the target index.

**Documents are refused**

Read the per-record error. Check mappings, field types, document ids and the refused-record answer before retrying.

## See also

-   [Destinations](/docs/destinations)
-   [Elasticsearch receiver source](/docs/sources/elastic-receiver)
-   [Handle refused records](/docs/delivery#refused)
-   [Secrets & credentials](/docs/secrets)