HEC receiver

Takes what Splunk HTTP Event Collector (HEC) clients send to its port, at Splunk's paths. It answers each request once every destination has its events.

OpenTelemetry Collectors, Fluent Bit, Vector, logging libraries and Splunk forwarders' httpout send to it. Every worker in the group listens on its port, and the sender, or a balancer in front of the workers, picks a worker for each request. Add workers to take more.

Example

Takes HEC events on port 8088, over TLS with the identity receiver-tls. Senders present the token held in the worker-host variable HEC_TOKEN.

{
  "id": "otel-logs",
  "type": "hecReceiver",
  "port": 8088,
  "tokens": {
    "otel": {
      "token": "HEC_TOKEN"
    }
  },
  "identity": "receiver-tls"
}

The HEC receiver form on the Sources page: id otel-logs, port 8088, no tokens yet. — The same source in the Sources page's add dialog, before you add its token and identity.

A sender posts an event like this:

curl -H "Authorization: Splunk $HEC_TOKEN" \
  -d '{"event":"hello","host":"web-1","sourcetype":"demo"}' \
  https://workers.example.com:8088/services/collector/event

Before you start

Pick a port no other receiver in the group uses.
Decide how senders prove who they are: a token, a client certificate, or both. See Tokens and certificates.
Put each token in a credential.
Open the port on every worker, and give senders one address that reaches them all. See Open its port.

Settings

It also takes the settings every source takes.

port integer required

The port it listens on. Every worker of the group binds it, so no two receivers share one.

Examples

80888089

tokens map default none

Each token's name to its token, the credential holding it, and acknowledgement. A sender presents a token as Authorization: Splunk <token>, and any other request is refused with code 2, 3 or 4. With acknowledgement true, as Splunk's useACK, its senders name a channel and are answered at once with an ackId, and one that names none is refused code 10; without it, a request is held until delivered and /services/collector/ack answers code 14. A receiver with no token needs an identity with authorities, and holds every request.

Examples

{"forwarders": {"token": "HEC_TOKEN"}}{"otel": {"token": "OTEL_HEC_TOKEN"},"vector": {"token": "VECTOR_HEC_TOKEN","acknowledgement": true}}

Advanced

identity identity default plain HTTP

The connection identity it serves TLS under: its certificate and key. With authorities, a sender must present a certificate that chains to one of them.

Example

receiver-tls

sourcetypes map default a record per line for every sourcetype

Maps each sourcetype to a lineBreaker, which splits text into multiple events. Its first capture group is the break and is dropped: the bytes before it end an event, line ends and all, and the bytes after it start the next. It needs a capture group and must not match the empty string. A raw request of a sourcetype not named, or named with no lineBreaker or with ([\r\n]+), is a record per line. The event endpoint's envelopes are not broken.

Example

{"app:xml": {"lineBreaker": "</event>(\\s*)"}}

maxRequestBytes integer default 10485760

The largest request body it reads, between 1024 and 67108864. A larger one is refused 413. A body sent as gzip is held to it as it inflates.

Examples

104857667108864

Paths it serves

POST /services/collector, /services/collector/event, /services/collector/event/1.0: The body holds HEC envelopes back to back, such as {"event":"hello","host":"web-1"}, with or without whitespace between them. Each envelope becomes one record: a string event becomes the record's text, and any other event its JSON. The envelope's host, source, sourcetype, index, time and fields become __source fields.
POST /services/collector/raw, /services/collector/raw/1.0: The body is text, one record per line. The query's host, source, sourcetype, index and time apply to every line, as in /services/collector/raw?sourcetype=syslog. Matching sourcetypes apply their linebreaker to replace the default of ([\r\n]+).
POST /services/collector/s2s: What a Splunk forwarder's httpout posts: S2S, as its tcpout sends to an S2S receiver, compressed. Each request is whole: its channels' events are broken by sourcetype as on the raw path, and the request ends each channel's last event. A record carries what the forwarder parsed, as an S2S receiver's does. See Senders it takes.
GET /services/collector/health, /services/collector/health/1.0: Says whether the receiver takes requests. See Health check.
POST /services/collector/ack, /services/collector/ack/1.0: For a sender whose token has acknowledgement: takes {"acks":[0,1]} on a channel and answers {"acks":{"0":true,"1":false}}. See Indexer acknowledgement.

time is Unix seconds, as a number or as text, and fields is a JSON object. The receiver inflates a body sent with Content-Encoding: gzip.

Fields it gives

A field the sender did not set is null. __source.remote holds the IP address the request came from: the balancer's, when one is in front of the workers.

Field	Type	Holds
`__source.name`	`str`	The name of the source that read the record.
`__source.type`	`str`	The kind of source that read it, its `type`, such as `kafka`, `s3List`, `httpApi` or `hecReceiver`.
`__source.remote`	`str`	The address of the sender that delivered the record to a receiver.
`__source.host`	`str`	The host a HEC sender filed the event under, or null where it named none.
`__source.source`	`str`	The source a HEC sender filed the event under, or null where it named none.
`__source.sourcetype`	`str`	The sourcetype a HEC sender filed the event under, or null where it named none.
`__source.index`	`str`	The index a HEC sender filed the event under, or an Elasticsearch client's bulk action named for its document; null where it named none.
`__source.time`	`datetime`	The time a sender gave the event, null otherwise.
`__source.fields`	`variant`	The indexed fields a HEC sender gave the event, or null where it gave none. Read one as text with `__source.fields.env as str`.
`__source.bytes`	`bytes`	The record's payload as the source read it, before any decoding. The source decodes `_raw` from it. A `bytes` destination writes it unless the destination names another field. Text functions refuse it; `decode(__source.bytes, encoding)` reads it as text.

Tokens and certificates

Anyone a receiver admits can put records into the group's pipelines, so a receiver needs tokens, an identity with authorities, or both. The leader refuses a receiver with neither.

Tokens: a sender presents one as Authorization: Splunk <token>, as HEC clients do. The receiver refuses any other request before reading its body. Without an identity, the token crosses the network in the clear, so use a token alone only behind a balancer that terminates TLS.
Client certificates: with an identity, the receiver serves TLS with the identity's certificate and key. If the identity has authorities, every connection, health checks included, must present a certificate that chains to one of them.

With both, a sender presents a certificate and a token.

What it answers

The receiver answers in HEC's JSON, {"text": ..., "code": ...}. The receiver answers success only once every destination has the request's events, so a sender that retries an unanswered request loses nothing.

200, code 0: {"text":"Success","code":0}: every destination has the request's events.
503, code 9, with Retry-After: The receiver is full, blocked or stopping. It refuses a new request at once, and a stopping receiver answers every request it holds. A sender may send it again.
401, code 2 or 3: The request has no token (2), or gives one other than as Splunk <token> (3).
403, code 4: The receiver does not hold the token.
400, code 5, 6, 12, 13 or 15: The body is empty (5), is not HEC envelopes, or at /services/collector/s2s not whole S2S (6), has an envelope without an event (12) or with an empty one (13), or has fields that are not an object (15). invalid-event-number names the faulty envelope, counting from 0. The receiver refuses the whole body and takes none of its events.
400, code 10, 11 or 14: A sender whose token has acknowledgement named no channel (10), a channel is not 1 to 128 letters, digits, - and _ (11), or a sender whose token lacks acknowledgement asked after an ackId (14, ACK is disabled).
413: The body, or what it inflates to, is larger than maxRequestBytes, or a forwarder's httpout request holds more than the receiver holds at once.
408: The sender took more than 30 seconds to send the whole body.
415: The body uses a Content-Encoding other than gzip or identity.

Indexer acknowledgement

Each token sets its own acknowledgement. Set acknowledgement: true on the token of a sender that has indexer acknowledgement turned on:

{
  "vector": {
    "token": "VECTOR_HEC_TOKEN",
    "acknowledgement": true
  },
  "fluentbit": {
    "token": "FLUENTBIT_HEC_TOKEN"
  }
}

That sender names a channel, in the X-Splunk-Request-Channel header or the channel query. The receiver answers at once with an ID, {"text":"Success","code":0,"ackId":7}, and the sender asks after it at /services/collector/ack. The ID reads true once every destination has the request's events, and the receiver forgets it once the sender has heard that.

An ID reads false while its events are on their way, and also when the worker asked does not hold it, because another worker gave it or the worker restarted. The sender then sends the request again: a duplicate, never a loss. A balancer in front of the workers must therefore keep each sender's connections on one worker, so a channel's requests and its ack queries reach the same worker.

The receiver holds the request of a sender whose token lacks acknowledgement until its events are delivered, whether or not it names a channel. A sender that names a channel but never asks after an ackId needs such a token.

Health check

GET /services/collector/health answers 200 with code 17, HEC is healthy, while the receiver takes requests, and 503 with code 18 while it is full, blocked or stopping. It needs no token. Point a balancer's health check or a readiness probe at it, so a busy worker gets no requests until it can take them. With an identity that has authorities, it needs a client certificate like any other request.

Senders it takes

Any sender that speaks HEC over HTTP works:

The OpenTelemetry Collector's splunk_hec exporter.
Fluent Bit's splunk output.
Vector's splunk_hec_logs sink. With indexer acknowledgements on, set acknowledgement on its token.
Logging libraries that send to HEC.

A Splunk universal or heavy forwarder's httpout posts S2S to /services/collector/s2s, presenting one of the receiver's tokens. The receiver holds each request until delivered, whatever its token's acknowledgement. A forwarder sending with tcpout needs an S2S receiver instead.

# outputs.conf on the forwarder
[httpout]
httpEventCollectorToken = <a token of the receiver>
uri = https://workers.example.com:8088

This OpenTelemetry Collector configuration sends logs to the receiver, with HEC_TOKEN set in the collector's environment:

exporters:
  splunk_hec:
    token: ${env:HEC_TOKEN}
    endpoint: https://workers.example.com:8088/services/collector
    source: otel
    sourcetype: otel

service:
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [splunk_hec]

Open its port

Every worker in the group listens on the receiver's port. Open that port to senders on each worker, and give senders one address that reaches them all, such as a balancer's. On a Unix host, a request sent while a worker deploys a new configuration waits, and the new configuration answers it.

On Kubernetes, add the port to the worker pods and put a Service in front of them: Expose a receiver.
On a host or in a container, open the host's firewall or publish the container's port: Enroll a worker on a host.

Verify

POST an event you can recognize, as in the example. The receiver answers {"text":"Success","code":0} once every destination has it. Then find it at the destination.

Troubleshooting

The sender cannot connect: Check that the port is open on every worker and that the sender's address reaches them. An identity with authorities refuses a connection without a certificate that chains to one of them.
Every request gets code 2, 3 or 4: Check that the sender presents Authorization: Splunk <token>, with a token one of the receiver's credentials holds.
Every request gets code 9: Check the source's status. A blocked receiver takes nothing until its destinations take records again. A full one takes more with more workers.
Acks never read true, and the sender sends everything again: The ack queries reach a different worker from the one that took the request. Keep each sender's connections on one worker at the balancer, or turn indexer acknowledgement off in both the sender and its token.
The sender reports code 10 or 14: The sender and its token disagree on indexer acknowledgement. Set acknowledgement: true on the token when the sender has it on, and leave it out when not.
A Splunk forwarder cannot send to it: A forwarder sending with tcpout needs an S2S receiver. Point it at one, or switch it to httpout, which this receiver takes at /services/collector/s2s.
The same events arrive twice: A sender that hears no answer may send the request again. Give the sender a timeout longer than its destinations take to deliver.

HEC receiver

Example

Before you start

Settings

Advanced

Paths it serves

Fields it gives

Tokens and certificates

What it answers

Indexer acknowledgement

Health check

Senders it takes

Open its port

Verify

Troubleshooting

See also