# S2S receiver

Takes what Splunk forwarders send over Splunk-to-Splunk (S2S).

Every worker in the group listens on its port. Add workers to take more.

## Example

Takes forwarders on port 9997, over TLS with the [identity](/docs/secrets#identities) `s2s-tls`. Forwarders present the token held in the worker-host variable `S2S_TOKEN`. An event of the sourcetype `java:app` starts at a line that starts with a date and holds the lines after it. The sourcetype `access_combined` breaks at every line end, as every sourcetype not named does, so its row is there only to show the default.

![The S2S receiver form on the Sources page: id forwarders, port 9997, no tokens yet, the sourcetype java:app with its date line breaker, and access_combined with ([\r\n]+).](/s2s-receiver-source-form-light.png)

The same source in the Sources page's add dialog, with its sourcetypes, before you add its token and identity.

A forwarder sends to it with this outputs.conf:

```
[tcpout]
defaultGroup = smeltr

[tcpout:smeltr]
server = worker1:9997, worker2:9997
useACK = true
token = 0d2a5f6c-4b1e-4c3a-9f7d-2e8b6a1c5d40
```

## Before you start

-   Pick a port no other receiver in the group uses.
-   Decide how forwarders prove who they are: a token, a client certificate, or both. See [Tokens and certificates](#senders).
-   Put each token in a [credential](/docs/secrets).
-   List only the sourcetypes whose events span several lines, such as stack traces, or break somewhere other than a line end. Every other sourcetype is an event per line. See [How lines become events](#events).
-   Open the port on every worker, and give forwarders the address of each. See [Open its port](#port).

## Settings

It also takes the settings [every source](/docs/sources#settings) takes.

`port`  integer required

The port it listens on. Every worker of the group binds it, so no two receivers share one.

Examples:

- `9997`
- `9998`

`tokens`  map default none

Each token's name to the [credential](/docs/secrets) holding it. A forwarder presents one as its outputs.conf `token`, and one that presents none or another is cut off. A receiver with no token needs an `identity` with authorities.

Examples:

- `uf-fleet = S2S_TOKEN`
- `datacenter = DC_FORWARDER_TOKEN`
- `branches = BRANCH_FORWARDER_TOKEN`

`sourcetypes`  map default an event per line for every sourcetype

Each sourcetype's name to its `lineBreaker`, where an event ends. Its first capture group is the break and is dropped: the bytes before it end an event, line ends and all, and the bytes after it start the next. What follows the group is kept, so the pattern can say how an event starts: `([\r\n]+)\d{4}-` keeps a stack trace with the line that logged it, and `</event>(\s*)` cuts a stream of XML events with no line ends between them. It needs a capture group and must not match the empty string. A sourcetype not named, or named with no `lineBreaker` or with `([\r\n]+)`, is an event per line, a line ending at `\n`, `\r\n` or a lone `\r`. An event that arrives already broken is one record as it came.

Examples:

- `{"java:app": {"lineBreaker": "([\\r\\n]+)\\d{4}-"}}`
- `{"java:app": {"lineBreaker": "([\\r\\n]+)\\d{4}-\\d{2}-\\d{2}"},"app:xml": {"lineBreaker": "</event>(\\s*)"}}`

### Advanced

`identity`  identity default plain TCP

The [connection identity](/docs/secrets#identities) it serves TLS under: its certificate and key. With authorities, a forwarder must present a certificate that chains to one of them.

Example:

- `s2s-tls`

`maxRequestBytes`  integer default `10485760`

The largest frame it reads, and the most one record holds, between 1024 and 67108864.

Examples:

- `1048576`
- `67108864`

## Point a forwarder at it

In the forwarder's outputs.conf, a `[tcpout]` group's `server` names every worker on the receiver's port, as in the [example](#example). Splunk's [Configure forwarding with outputs.conf](https://help.splunk.com/en/splunk-enterprise/forward-and-process-data/universal-forwarder-manual/10.6/forward-data/configure-forwarding-with-outputs.conf) covers the file.

-   `useACK = true` makes delivery at-least-once. See [Acknowledgement](#acks).
-   `token` is the value one of the receiver's `tokens` holds. Leave it out when the receiver has no tokens.
-   `compressed = true`, when set, needs no change on the receiver: it takes compressed and uncompressed connections alike.

## How lines become events

The receiver breaks what a forwarder sends into events by the rule `sourcetypes` names for its sourcetype:

-   A sourcetype not named in `sourcetypes` is an event per line, so most need no row.
-   A line ends at `\n`, `\r\n` or a lone `\r`.
-   A sourcetype named in `sourcetypes` ends an event where its `lineBreaker`'s first capture group matches. What the group matched is dropped; what follows it starts the next event, so the pattern can say how an event starts. The pattern needs a capture group and must not match the empty string.
-   An event that arrives already broken is one record as it came.
-   A line still unfinished when its connection ends, or after 30 seconds with no more of it from a forwarder with `useACK`, is an event as it stands. A line longer than `maxRequestBytes` is cut at that length.

A sourcetype not named, or one whose line breaker is left blank, breaks as `([\r\n]+)` does: at every line end, with no event for a blank line.

The line breaker `([\r\n]+)\d{4}-\d{2}-\d{2}` keeps each Java stack trace with the log line above it: it breaks only at line ends followed by a date, and only the log line starts with one. The line breaker `</event>(\s*)` makes an event of each XML event in a stream that has no line ends between them.

## Fields it gives

Each event is a record, its text in `_raw`. The host, source, sourcetype, index and time the forwarder files it under, and its indexed fields, become `__source` fields. `__source.remote` holds the IP address the forwarder's connection came from.

| Field | Type | Holds |
| --- | --- | --- |
| `__source.name` | `str` | The name of the source that read the record. |
| `__source.type` | `str` | The kind of source that read it, its `type`, such as `kafka`, `s3List`, `httpApi` or `hecReceiver`. |
| `__source.remote` | `str` | The address of the sender that delivered the record to a receiver. |
| `__source.host` | `str` | The host a HEC sender filed the event under, or null where it named none. |
| `__source.source` | `str` | The source a HEC sender filed the event under, or null where it named none. |
| `__source.sourcetype` | `str` | The sourcetype a HEC sender filed the event under, or null where it named none. |
| `__source.index` | `str` | The index a HEC sender filed the event under, or an Elasticsearch client's bulk action named for its document; null where it named none. |
| `__source.time` | `datetime` | The time a sender gave the event, null otherwise. |
| `__source.fields` | `variant` | The indexed fields a HEC sender gave the event, or null where it gave none. Read one as text with `__source.fields.env as str`. |
| `__source.bytes` | `bytes` | The record's payload as the source read it, before any decoding. The source decodes `_raw` from it. A `bytes` destination writes it unless the destination names another field. Text functions refuse it; `decode(__source.bytes, encoding)` reads it as text. |

## Tokens and certificates

Anyone a receiver admits can put records into the group's pipelines, so a receiver needs `tokens`, an `identity` with authorities, or both. The leader refuses a receiver with neither.

-   **Tokens:** a forwarder presents one as its outputs.conf `token`. The receiver closes the connection of a forwarder with no token or a wrong one before admitting any of its data. Without an identity, the token crosses the network in the clear.
-   **Client certificates:** with an [identity](/docs/secrets#identities), the receiver serves TLS with the identity's certificate and key. If the identity has authorities, every forwarder must present a certificate that chains to one of them.

With both, a forwarder presents a certificate and a token.

## Acknowledgement

The receiver acknowledges an event once every record holding its bytes has reached every destination. Set `useACK = true` on every forwarder whose events must all arrive; an event can then arrive twice, never be lost. Splunk's [Protect against loss of in-flight data](https://help.splunk.com/en/splunk-enterprise/forward-and-process-data/forwarding-and-receiving-data/10.2/perform-advanced-configuration/protect-against-loss-of-in-flight-data) covers the forwarder's side.

## When it is full

A receiver that is full, or [blocked](/docs/delivery#hold), stops reading from its connections until its destinations take records again. Name every worker in the forwarder's `server`, so it has others to send to.

## Forwarders it takes

-   A forwarder sending with `tcpout` and its default protocol, compressed or not.
-   A forwarder set to `enableOldS2SProtocol = true` is refused.
-   A forwarder's `httpout` posts S2S to `/services/collector/s2s`, which the [HEC receiver](/docs/sources/hec-receiver) serves.

## Open its port

Every worker in the group listens on the receiver's `port`. Open that port to forwarders on each worker, and name every worker in the forwarder's `server`.

-   On Kubernetes, add the port to the worker pods: [Expose a receiver](/docs/enroll-worker/kubernetes#receiver).
-   On a host or in a container, open the host's firewall or publish the container's port: [Enroll a worker on a host](/docs/enroll-worker/host).

## Verify

1.  Point a forwarder at the receiver, as in the [outputs.conf above](#forwarder).
2.  Append a line you can recognize to a file the forwarder monitors.
3.  Find it at the destination, with its host, source and sourcetype.

## Troubleshooting

**The forwarder cannot connect**

Check that the port is open on every worker and that the forwarder's addresses reach them. An identity with authorities refuses a connection without a certificate that chains to one of them.

**The receiver closes the forwarder's connections at once**

The forwarder has no token, or one no credential of the receiver's `tokens` holds. Set its outputs.conf `token` to one the receiver holds.

**The forwarder is refused on every connection**

Check its outputs.conf for `enableOldS2SProtocol = true`, and remove it. The receiver takes only the forwarder's default protocol.

**Nothing arrives**

Check the source's status. A blocked receiver reads nothing until its destinations take records again. A full one takes more with more workers.

**A multi-line event arrives as one record per line**

Name its sourcetype in `sourcetypes` with a `lineBreaker` that matches only line ends before an event's first line, such as `([\r\n]+)\d{4}-`.

**Many events arrive as one record**

No line end separates them. Name their sourcetype in `sourcetypes` with a `lineBreaker` that matches between two events, such as `</event>(\s*)`.

**A line arrives cut in two, and the worker warns that a line was cut**

No more of the line arrived for 30 seconds, as when its file stopped growing mid-line, or when the source's destinations are slow. A line longer than `maxRequestBytes` is cut at that length.

**Events are missing after a worker stopped**

Set `useACK = true` on the forwarder. See [Acknowledgement](#acks).

**The same events arrive twice**

With `useACK`, an event not acknowledged when its connection was cut can arrive again. See [Acknowledgement](#acks).

## See also

-   [Sources](/docs/sources)
-   [HEC receiver source](/docs/sources/hec-receiver)
-   [Splunk destination](/docs/destinations/splunk)
-   [Credentials](/docs/secrets)
-   [Delivery](/docs/delivery)