S2S receiver

Takes what Splunk forwarders send over Splunk-to-Splunk (S2S).

Every worker in the group listens on its port. Add workers to take more.

Example

Takes forwarders on port 9997, over TLS with the identity s2s-tls. Forwarders present the token held in the worker-host variable S2S_TOKEN. An event of the sourcetype java:app starts at a line that starts with a date and holds the lines after it. The sourcetype access_combined breaks at every line end, as every sourcetype not named does, so its row is there only to show the default.

The S2S receiver form on the Sources page: id forwarders, port 9997, no tokens yet, the sourcetype java:app with its date line breaker, and access_combined with ([\r\n]+).
Open complete screenshot ↗
The S2S receiver form on the Sources page: id forwarders, port 9997, no tokens yet, the sourcetype java:app with its date line breaker, and access_combined with ([\r\n]+).
Open complete screenshot ↗
The same source in the Sources page's add dialog, with its sourcetypes, before you add its token and identity.

A forwarder sends to it with this outputs.conf:

[tcpout]
defaultGroup = smeltr

[tcpout:smeltr]
server = worker1:9997, worker2:9997
useACK = true
token = 0d2a5f6c-4b1e-4c3a-9f7d-2e8b6a1c5d40

Before you start

  • Pick a port no other receiver in the group uses.
  • Decide how forwarders prove who they are: a token, a client certificate, or both. See Tokens and certificates.
  • Put each token in a credential.
  • List only the sourcetypes whose events span several lines, such as stack traces, or break somewhere other than a line end. Every other sourcetype is an event per line. See How lines become events.
  • Open the port on every worker, and give forwarders the address of each. See Open its port.

Settings

It also takes the settings every source takes.

port integer required

The port it listens on. Every worker of the group binds it, so no two receivers share one.

Examples
99979998

tokens map default none

Each token's name to the credential holding it. A forwarder presents one as its outputs.conf token, and one that presents none or another is cut off. A receiver with no token needs an identity with authorities.

Examples
uf-fleet = S2S_TOKENdatacenter = DC_FORWARDER_TOKENbranches = BRANCH_FORWARDER_TOKEN

sourcetypes map default an event per line for every sourcetype

Each sourcetype's name to its lineBreaker, where an event ends. Its first capture group is the break and is dropped: the bytes before it end an event, line ends and all, and the bytes after it start the next. What follows the group is kept, so the pattern can say how an event starts: ([\r\n]+)\d{4}- keeps a stack trace with the line that logged it, and </event>(\s*) cuts a stream of XML events with no line ends between them. It needs a capture group and must not match the empty string. A sourcetype not named, or named with no lineBreaker or with ([\r\n]+), is an event per line, a line ending at \n, \r\n or a lone \r. An event that arrives already broken is one record as it came.

Examples
{"java:app": {"lineBreaker": "([\\r\\n]+)\\d{4}-"}}{"java:app": {"lineBreaker": "([\\r\\n]+)\\d{4}-\\d{2}-\\d{2}"},"app:xml": {"lineBreaker": "</event>(\\s*)"}}

Advanced

identity identity default plain TCP

The connection identity it serves TLS under: its certificate and key. With authorities, a forwarder must present a certificate that chains to one of them.

Example
s2s-tls

maxRequestBytes integer default 10485760

The largest frame it reads, and the most one record holds, between 1024 and 67108864.

Examples
104857667108864

Point a forwarder at it

In the forwarder's outputs.conf, a [tcpout] group's server names every worker on the receiver's port, as in the example. Splunk's Configure forwarding with outputs.conf covers the file.

  • useACK = true makes delivery at-least-once. See Acknowledgement.
  • token is the value one of the receiver's tokens holds. Leave it out when the receiver has no tokens.
  • compressed = true, when set, needs no change on the receiver: it takes compressed and uncompressed connections alike.

How lines become events

The receiver breaks what a forwarder sends into events by the rule sourcetypes names for its sourcetype:

  • A sourcetype not named in sourcetypes is an event per line, so most need no row.
  • A line ends at \n, \r\n or a lone \r.
  • A sourcetype named in sourcetypes ends an event where its lineBreaker's first capture group matches. What the group matched is dropped; what follows it starts the next event, so the pattern can say how an event starts. The pattern needs a capture group and must not match the empty string.
  • An event that arrives already broken is one record as it came.
  • A line still unfinished when its connection ends, or after 30 seconds with no more of it from a forwarder with useACK, is an event as it stands. A line longer than maxRequestBytes is cut at that length.

A sourcetype not named, or one whose line breaker is left blank, breaks as ([\r\n]+) does: at every line end, with no event for a blank line.

The line breaker ([\r\n]+)\d{4}-\d{2}-\d{2} keeps each Java stack trace with the log line above it: it breaks only at line ends followed by a date, and only the log line starts with one. The line breaker </event>(\s*) makes an event of each XML event in a stream that has no line ends between them.

Fields it gives

Each event is a record, its text in _raw. The host, source, sourcetype, index and time the forwarder files it under, and its indexed fields, become __source fields. __source.remote holds the IP address the forwarder's connection came from.

FieldTypeHolds
__source.namestrThe name of the source that read the record.
__source.typestrThe kind of source that read it, its type, such as kafka, s3List, httpApi or hecReceiver.
__source.remotestrThe address of the sender that delivered the record to a receiver.
__source.hoststrThe host a HEC sender filed the event under, or null where it named none.
__source.sourcestrThe source a HEC sender filed the event under, or null where it named none.
__source.sourcetypestrThe sourcetype a HEC sender filed the event under, or null where it named none.
__source.indexstrThe index a HEC sender filed the event under, or an Elasticsearch client's bulk action named for its document; null where it named none.
__source.timedatetimeThe time a sender gave the event, null otherwise.
__source.fieldsvariantThe indexed fields a HEC sender gave the event, or null where it gave none. Read one as text with __source.fields.env as str.
__source.bytesbytesThe record's payload as the source read it, before any decoding. The source decodes _raw from it. A bytes destination writes it unless the destination names another field. Text functions refuse it; decode(__source.bytes, encoding) reads it as text.

Tokens and certificates

Anyone a receiver admits can put records into the group's pipelines, so a receiver needs tokens, an identity with authorities, or both. The leader refuses a receiver with neither.

  • Tokens: a forwarder presents one as its outputs.conf token. The receiver closes the connection of a forwarder with no token or a wrong one before admitting any of its data. Without an identity, the token crosses the network in the clear.
  • Client certificates: with an identity, the receiver serves TLS with the identity's certificate and key. If the identity has authorities, every forwarder must present a certificate that chains to one of them.

With both, a forwarder presents a certificate and a token.

Acknowledgement

The receiver acknowledges an event once every record holding its bytes has reached every destination. Set useACK = true on every forwarder whose events must all arrive; an event can then arrive twice, never be lost. Splunk's Protect against loss of in-flight data covers the forwarder's side.

When it is full

A receiver that is full, or blocked, stops reading from its connections until its destinations take records again. Name every worker in the forwarder's server, so it has others to send to.

Forwarders it takes

  • A forwarder sending with tcpout and its default protocol, compressed or not.
  • A forwarder set to enableOldS2SProtocol = true is refused.
  • A forwarder's httpout posts S2S to /services/collector/s2s, which the HEC receiver serves.

Open its port

Every worker in the group listens on the receiver's port. Open that port to forwarders on each worker, and name every worker in the forwarder's server.

Verify

  1. Point a forwarder at the receiver, as in the outputs.conf above.
  2. Append a line you can recognize to a file the forwarder monitors.
  3. Find it at the destination, with its host, source and sourcetype.

Troubleshooting

The forwarder cannot connect
Check that the port is open on every worker and that the forwarder's addresses reach them. An identity with authorities refuses a connection without a certificate that chains to one of them.
The receiver closes the forwarder's connections at once
The forwarder has no token, or one no credential of the receiver's tokens holds. Set its outputs.conf token to one the receiver holds.
The forwarder is refused on every connection
Check its outputs.conf for enableOldS2SProtocol = true, and remove it. The receiver takes only the forwarder's default protocol.
Nothing arrives
Check the source's status. A blocked receiver reads nothing until its destinations take records again. A full one takes more with more workers.
A multi-line event arrives as one record per line
Name its sourcetype in sourcetypes with a lineBreaker that matches only line ends before an event's first line, such as ([\r\n]+)\d{4}-.
Many events arrive as one record
No line end separates them. Name their sourcetype in sourcetypes with a lineBreaker that matches between two events, such as </event>(\s*).
A line arrives cut in two, and the worker warns that a line was cut
No more of the line arrived for 30 seconds, as when its file stopped growing mid-line, or when the source's destinations are slow. A line longer than maxRequestBytes is cut at that length.
Events are missing after a worker stopped
Set useACK = true on the forwarder. See Acknowledgement.
The same events arrive twice
With useACK, an event not acknowledged when its connection was cut can arrive again. See Acknowledgement.

See also