Elasticsearch

Writes each record as a JSON document through Elasticsearch's Bulk API, many documents to a request.

Example

Authorizes with the worker-host variable ELASTIC_AUTH, which holds the complete Authorization header. Replace the URL and index.

{
  "id": "search-out",
  "type": "elasticsearch",
  "url": "https://elasticsearch.example.com:9200",
  "defaultIndex": "smeltr-events",
  "auth": "ELASTIC_AUTH"
}

Before you start

  • Prepare the target index or data stream and its mappings, or let the account create them.
  • Give an account permission to write there, and put its complete Authorization header, such as ApiKey <encoded-key>, in a credential.
  • Select a connection identity for a private CA or a client certificate.

Settings

It also takes the settings every destination takes.

url string required

The cluster's base address.

Examples
https://elasticsearch.example.com:9200https://10.20.0.5:9200

defaultIndex string required

The index for a record with no __destination.index.

Examples
orderslogs-firewall-default

identity identity default none

The connection identity it connects under, for a private CA or a client certificate.

Examples
elastic-clientelastic-mtls

auth credential default no Authorization header

The credential holding the whole Authorization header value, such as ApiKey <encoded-key>.

Examples
ELASTIC_AUTH{"secret": "elastic-auth","version": 1}

docId string default auto

Who chooses each document's id. See Duplicate documents.

auto
Lets Elasticsearch choose each document's id.
dedupe
Keeps the first document written under an id.
overwrite
Replaces the document written under an id.

idField string default __destination.id

The field holding a document's id under dedupe or overwrite.

Examples
event.idrequest_id

Advanced

inFlight integer default 4

How many bulk requests it sends at once to start with. 1 to 64.

Examples
8116

maxBulkBytes size default 5 MiB

The most one bulk request carries. A record larger than this still goes as a single record. 64 KiB to 100 MiB.

Examples
10 MiB512 KiB15MB

bufferBytes size default 64 MiB, or one bulk request if that is larger

How much undelivered output it may hold. 4 MiB to 1 GiB, and at least one bulk request.

Examples
256 MiB128 MiB1 GiB

stallAfterSecs integer default 30

Seconds one batch may take to deliver, every retry included, before it counts as stalled. 1 to 120.

Examples
60120

zoneUrls map default none

An address per zone. A worker in a listed zone sends there instead of url.

Examples
us-east-1a = https://es-a.example.com:9200us-east-1a = https://es-a.example.com:9200us-east-1b = https://es-b.example.com:9200

sniff bool default false

Learns the cluster's nodes from url and spreads requests across them, preferring the worker's zone. Every node must be reachable from the workers, and the credential needs the monitor cluster privilege.

zoneAttribute string default zone

The node attribute that names a node's zone, set on each node as node.attr.<name>. Requires sniff enabled.

Examples
rackavailability_zone

Fields it reads

FieldTypeHolds
__destination.indexstrThe index an Elasticsearch or Splunk HEC destination files the record under. Null means the destination's default index. A Splunk HEC destination on the raw endpoint refuses a record naming an index other than its own.
__destination.idstrThe Elasticsearch document ID, used when the destination's document ID setting is dedupe or overwrite. When it is null, the destination's id field names the field that gives it, and with neither the document is sent with no ID.

How it works

Duplicate documents

PolicyBehavior
auto (default)Sends no document ID. A record resent after a crash can create a second document.
dedupeUse a stable ID and keep the first document written with that ID.
overwriteReplace the document with that ID. Concurrent requests can arrive out of order. This policy cannot write to a data stream.

For dedupe or overwrite, set idField to a stable field such as event.id, or have the pipeline set __destination.id. Keep auto when records have no stable ID.

Verify

Send a record with a unique marker through the source, then search the index for it after a refresh. Check the document's fields and the worker's delivery errors.

Troubleshooting

401 or 403
Check the full Authorization header, that each worker has the secret, and permissions on the target index.
Documents are refused
Read the per-record error. Check mappings, field types, document ids and the refused-record answer before retrying.

See also