# HEC receiver Takes what Splunk HTTP Event Collector (HEC) clients send to its port, at Splunk's paths. It answers each request once every destination has its events. OpenTelemetry Collectors, Fluent Bit, Vector, logging libraries and Splunk forwarders' `httpout` send to it. Every worker in the group listens on its port, and the sender, or a balancer in front of the workers, picks a worker for each request. Add workers to take more. ## Example Takes HEC events on port 8088, over TLS with the [identity](/docs/secrets#identities) `receiver-tls`. Senders present the token held in the worker-host variable `HEC_TOKEN`. ``` { "id": "otel-logs", "type": "hecReceiver", "port": 8088, "tokens": { "otel": { "token": "HEC_TOKEN" } }, "identity": "receiver-tls" } ``` ![The HEC receiver form on the Sources page: id otel-logs, port 8088, no tokens yet.](/hec-receiver-source-form-light.png) The same source in the Sources page's add dialog, before you add its token and identity. A sender posts an event like this: ``` curl -H "Authorization: Splunk $HEC_TOKEN" \ -d '{"event":"hello","host":"web-1","sourcetype":"demo"}' \ https://workers.example.com:8088/services/collector/event ``` ## Before you start - Pick a port no other receiver in the group uses. - Decide how senders prove who they are: a token, a client certificate, or both. See [Tokens and certificates](#senders). - Put each token in a [credential](/docs/secrets). - Open the port on every worker, and give senders one address that reaches them all. See [Open its port](#port). ## Settings It also takes the settings [every source](/docs/sources#settings) takes. `port` integer required The port it listens on. Every worker of the group binds it, so no two receivers share one. Examples: - `8088` - `8089` `tokens` map default none Each token's name to its `token`, the [credential](/docs/secrets) holding it, and `acknowledgement`. A sender presents a token as `Authorization: Splunk `, and any other request is refused with code `2`, `3` or `4`. With `acknowledgement` true, as Splunk's `useACK`, its senders name a channel and are answered at once with an `ackId`, and one that names none is refused code `10`; without it, a request is held until delivered and `/services/collector/ack` answers code `14`. A receiver with no token needs an `identity` with authorities, and holds every request. Examples: - `{"forwarders": {"token": "HEC_TOKEN"}}` - `{"otel": {"token": "OTEL_HEC_TOKEN"},"vector": {"token": "VECTOR_HEC_TOKEN","acknowledgement": true}}` ### Advanced `identity` identity default plain HTTP The [connection identity](/docs/secrets#identities) it serves TLS under: its certificate and key. With authorities, a sender must present a certificate that chains to one of them. Example: - `receiver-tls` `sourcetypes` map default a record per line for every sourcetype Maps each sourcetype to a lineBreaker, which splits text into multiple events. Its first capture group is the break and is dropped: the bytes before it end an event, line ends and all, and the bytes after it start the next. It needs a capture group and must not match the empty string. A raw request of a sourcetype not named, or named with no `lineBreaker` or with `([\r\n]+)`, is a record per line. The event endpoint's envelopes are not broken. Example: - `{"app:xml": {"lineBreaker": "(\\s*)"}}` `maxRequestBytes` integer default `10485760` The largest request body it reads, between 1024 and 67108864. A larger one is refused `413`. A body sent as gzip is held to it as it inflates. Examples: - `1048576` - `67108864` ## Paths it serves **POST** `/services/collector`, `/services/collector/event`, `/services/collector/event/1.0` The body holds HEC envelopes back to back, such as `{"event":"hello","host":"web-1"}`, with or without whitespace between them. Each envelope becomes one record: a string `event` becomes the record's text, and any other `event` its JSON. The envelope's `host`, `source`, `sourcetype`, `index`, `time` and `fields` become [`__source` fields](#fields-it-gives). **POST** `/services/collector/raw`, `/services/collector/raw/1.0` The body is text, one record per line. The query's `host`, `source`, `sourcetype`, `index` and `time` apply to every line, as in `/services/collector/raw?sourcetype=syslog`. Matching sourcetypes apply their linebreaker to replace the default of `([\r\n]+)`. **POST** `/services/collector/s2s` What a Splunk forwarder's `httpout` posts: S2S, as its `tcpout` sends to an [S2S receiver](/docs/sources/s2s-receiver), compressed. Each request is whole: its channels' events are broken by sourcetype as on the raw path, and the request ends each channel's last event. A record carries what the forwarder parsed, as an S2S receiver's does. See [Senders it takes](#clients). **GET** `/services/collector/health`, `/services/collector/health/1.0` Says whether the receiver takes requests. See [Health check](#health). **POST** `/services/collector/ack`, `/services/collector/ack/1.0` For a sender whose token has `acknowledgement`: takes `{"acks":[0,1]}` on a channel and answers `{"acks":{"0":true,"1":false}}`. See [Indexer acknowledgement](#acks). `time` is Unix seconds, as a number or as text, and `fields` is a JSON object. The receiver inflates a body sent with `Content-Encoding: gzip`. ## Fields it gives A field the sender did not set is null. `__source.remote` holds the IP address the request came from: the balancer's, when one is in front of the workers. | Field | Type | Holds | | --- | --- | --- | | `__source.name` | `str` | The name of the source that read the record. | | `__source.type` | `str` | The kind of source that read it, its `type`, such as `kafka`, `s3List`, `httpApi` or `hecReceiver`. | | `__source.remote` | `str` | The address of the sender that delivered the record to a receiver. | | `__source.host` | `str` | The host a HEC sender filed the event under, or null where it named none. | | `__source.source` | `str` | The source a HEC sender filed the event under, or null where it named none. | | `__source.sourcetype` | `str` | The sourcetype a HEC sender filed the event under, or null where it named none. | | `__source.index` | `str` | The index a HEC sender filed the event under, or an Elasticsearch client's bulk action named for its document; null where it named none. | | `__source.time` | `datetime` | The time a sender gave the event, null otherwise. | | `__source.fields` | `variant` | The indexed fields a HEC sender gave the event, or null where it gave none. Read one as text with `__source.fields.env as str`. | | `__source.bytes` | `bytes` | The record's payload as the source read it, before any decoding. The source decodes `_raw` from it. A `bytes` destination writes it unless the destination names another field. Text functions refuse it; `decode(__source.bytes, encoding)` reads it as text. | ## Tokens and certificates Anyone a receiver admits can put records into the group's pipelines, so a receiver needs `tokens`, an `identity` with authorities, or both. The leader refuses a receiver with neither. - **Tokens:** a sender presents one as `Authorization: Splunk `, as HEC clients do. The receiver refuses any other request before reading its body. Without an identity, the token crosses the network in the clear, so use a token alone only behind a balancer that terminates TLS. - **Client certificates:** with an [identity](/docs/secrets#identities), the receiver serves TLS with the identity's certificate and key. If the identity has authorities, every connection, health checks included, must present a certificate that chains to one of them. With both, a sender presents a certificate and a token. ## What it answers The receiver answers in HEC's JSON, `{"text": ..., "code": ...}`. The receiver answers success only once every destination has the request's events, so a sender that retries an unanswered request loses nothing. **200**, code `0` `{"text":"Success","code":0}`: every destination has the request's events. **503**, code `9`, with `Retry-After` The receiver is full, [blocked](/docs/delivery#hold) or stopping. It refuses a new request at once, and a stopping receiver answers every request it holds. A sender may send it again. **401**, code `2` or `3` The request has no token (`2`), or gives one other than as `Splunk ` (`3`). **403**, code `4` The receiver does not hold the token. **400**, code `5`, `6`, `12`, `13` or `15` The body is empty (`5`), is not HEC envelopes, or at `/services/collector/s2s` not whole S2S (`6`), has an envelope without an `event` (`12`) or with an empty one (`13`), or has `fields` that are not an object (`15`). `invalid-event-number` names the faulty envelope, counting from 0. The receiver refuses the whole body and takes none of its events. **400**, code `10`, `11` or `14` A sender whose token has `acknowledgement` named no channel (`10`), a channel is not 1 to 128 letters, digits, `-` and `_` (`11`), or a sender whose token lacks `acknowledgement` asked after an `ackId` (`14`, ACK is disabled). **413** The body, or what it inflates to, is larger than `maxRequestBytes`, or a forwarder's `httpout` request holds more than the receiver holds at once. **408** The sender took more than 30 seconds to send the whole body. **415** The body uses a `Content-Encoding` other than `gzip` or `identity`. ## Indexer acknowledgement Each token sets its own acknowledgement. Set `acknowledgement: true` on the token of a sender that has indexer acknowledgement turned on: ``` { "vector": { "token": "VECTOR_HEC_TOKEN", "acknowledgement": true }, "fluentbit": { "token": "FLUENTBIT_HEC_TOKEN" } } ``` That sender names a channel, in the `X-Splunk-Request-Channel` header or the `channel` query. The receiver answers at once with an ID, `{"text":"Success","code":0,"ackId":7}`, and the sender asks after it at `/services/collector/ack`. The ID reads `true` once every destination has the request's events, and the receiver forgets it once the sender has heard that. An ID reads `false` while its events are on their way, and also when the worker asked does not hold it, because another worker gave it or the worker restarted. The sender then sends the request again: a duplicate, never a loss. A balancer in front of the workers must therefore keep each sender's connections on one worker, so a channel's requests and its ack queries reach the same worker. The receiver holds the request of a sender whose token lacks `acknowledgement` until its events are delivered, whether or not it names a channel. A sender that names a channel but never asks after an `ackId` needs such a token. ## Health check `GET /services/collector/health` answers 200 with code `17`, `HEC is healthy`, while the receiver takes requests, and 503 with code `18` while it is full, blocked or stopping. It needs no token. Point a balancer's health check or a readiness probe at it, so a busy worker gets no requests until it can take them. With an identity that has authorities, it needs a client certificate like any other request. ## Senders it takes Any sender that speaks HEC over HTTP works: - The OpenTelemetry Collector's `splunk_hec` exporter. - Fluent Bit's `splunk` output. - Vector's `splunk_hec_logs` sink. With indexer acknowledgements on, set [`acknowledgement`](#acks) on its token. - Logging libraries that send to HEC. A Splunk universal or heavy forwarder's `httpout` posts S2S to `/services/collector/s2s`, presenting one of the receiver's tokens. The receiver holds each request until delivered, whatever its token's `acknowledgement`. A forwarder sending with `tcpout` needs an [S2S receiver](/docs/sources/s2s-receiver) instead. ``` # outputs.conf on the forwarder [httpout] httpEventCollectorToken = uri = https://workers.example.com:8088 ``` This OpenTelemetry Collector configuration sends logs to the receiver, with `HEC_TOKEN` set in the collector's environment: ``` exporters: splunk_hec: token: ${env:HEC_TOKEN} endpoint: https://workers.example.com:8088/services/collector source: otel sourcetype: otel service: pipelines: logs: receivers: [otlp] exporters: [splunk_hec] ``` ## Open its port Every worker in the group listens on the receiver's `port`. Open that port to senders on each worker, and give senders one address that reaches them all, such as a balancer's. On a Unix host, a request sent while a worker deploys a new configuration waits, and the new configuration answers it. - On Kubernetes, add the port to the worker pods and put a Service in front of them: [Expose a receiver](/docs/enroll-worker/kubernetes#receiver). - On a host or in a container, open the host's firewall or publish the container's port: [Enroll a worker on a host](/docs/enroll-worker/host). ## Verify POST an event you can recognize, as in the [example](#example). The receiver answers `{"text":"Success","code":0}` once every destination has it. Then find it at the destination. ## Troubleshooting **The sender cannot connect** Check that the port is open on every worker and that the sender's address reaches them. An identity with authorities refuses a connection without a certificate that chains to one of them. **Every request gets code 2, 3 or 4** Check that the sender presents `Authorization: Splunk `, with a token one of the receiver's credentials holds. **Every request gets code 9** Check the source's status. A blocked receiver takes nothing until its destinations take records again. A full one takes more with more workers. **Acks never read true, and the sender sends everything again** The ack queries reach a different worker from the one that took the request. Keep each sender's connections on one worker at the balancer, or turn indexer acknowledgement off in both the sender and its token. **The sender reports code 10 or 14** The sender and its token disagree on indexer acknowledgement. Set `acknowledgement: true` on the token when the sender has it on, and leave it out when not. **A Splunk forwarder cannot send to it** A forwarder sending with `tcpout` needs an [S2S receiver](/docs/sources/s2s-receiver). Point it at one, or switch it to `httpout`, which this receiver takes at `/services/collector/s2s`. **The same events arrive twice** A sender that hears no answer may send the request again. Give the sender a timeout longer than its destinations take to deliver. ## See also - [Sources](/docs/sources) - [HTTP receiver source](/docs/sources/http-receiver) - [Splunk destination](/docs/destinations/splunk) - [Credentials](/docs/secrets) - [Delivery](/docs/delivery)