# Prometheus receiver

Takes the samples Prometheus sends over remote\_write to its port. It answers each request once every destination has its samples.

Prometheus and any agent that speaks remote\_write 1.0 or 2.0 send to it. Every worker in the group listens on its port, and the sender, or a balancer in front of the workers, picks a worker for each request. Add workers to take more.

## Example

Takes remote\_write on port 9090, over TLS with the [identity](/docs/secrets#identities) `receiver-tls`. Senders present the token held in the worker-host variable `PROM_TOKEN`.

```
{
  "id": "prometheus",
  "type": "prometheusReceiver",
  "port": 9090,
  "tokens": {
    "prometheus": "PROM_TOKEN"
  },
  "identity": "receiver-tls"
}
```

![The Prometheus receiver form on the Sources page: id prometheus, port 9090, no tokens yet.](/prometheus-receiver-source-form-light.png)

The same source in the Sources page's add dialog, before you add its token and identity.

Prometheus sends to it with a `remote_write` block like this:

```
remote_write:
  - url: https://workers.example.com:9090/api/v1/write
    authorization:
      credentials_file: /etc/prometheus/smeltr-token
    tls_config:
      ca_file: /etc/prometheus/smeltr-ca.pem
```

## Before you start

-   Pick a port no other receiver in the group uses.
-   For native histograms, set `send_native_histograms: true` in the sender's [remote\_write](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write) settings.
-   Decide how senders prove who they are: a token, a client certificate, or both. See [Tokens and certificates](#senders).
-   Put each token in a [credential](/docs/secrets).
-   For exemplars, set `send_exemplars: true` in the sender's [remote\_write](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write) settings.
-   Open the port on every worker, and give senders one address that reaches them all. See [Open its port](#port).

## Settings

It also takes the settings [every source](/docs/sources#settings) takes.

`port`  integer required

The port it listens on. Every worker of the group binds it, so no two receivers share one.

Examples:

- `9090`
- `19291`

`tokens`  map default none

Each token's name to the [credential](/docs/secrets) holding it. A sender presents one as `Authorization: Bearer <token>`, and any other request is refused `401`. A receiver with no token needs an `identity` with authorities.

Examples:

- `prometheus = PROM_TOKEN`
- `{"agents": {"secret": "prom-agents","version": 1}}`

### Advanced

`identity`  identity default plain HTTP

The [connection identity](/docs/secrets#identities) it serves TLS under: its certificate and key. With authorities, a sender must present a certificate that chains to one of them.

Example:

- `receiver-tls`

`maxRequestBytes`  integer default `10485760`

The largest request body it reads, between 1024 and 67108864. A larger one is refused `413`, which a sender may drop.

Examples:

- `1048576`
- `67108864`

## Paths it serves

**POST** `/api/v1/write`

A remote\_write 1.0 or 2.0 request, as the sender sends it.

**GET** `/health`

Says whether the receiver takes requests. See [Health check](#health).

## Records it gives

Each sample of a request becomes one record: a float sample, or a native histogram. A series of three samples is three records, each with the series' labels. Each field has the name and type of an [OTLP metric point's](/docs/sources/otlp-receiver#records) field, so a pipeline reads a metric from either receiver alike. A field a sample does not have is null.

`time`

A datetime, from the sample's milliseconds.

`name`

The series' metric name, its `__name__` label.

`value`

A variant holding the sample's double, bit for bit, so a staleness marker stays one. Read it as a number with `value as f64`.

`attributes`

The series' other labels, an object of strings, or null when the series has no label but `__name__`. Read one with `attributes["job"] as str`.

`type`

A native histogram's: `exponentialHistogram`, or `histogram` for one with custom buckets. Null for a float sample.

`aggregationTemporality`

2, cumulative, for a counter's histogram; null for a gauge histogram.

`count`, `sum`

A histogram's count and sum, doubles.

`scale`, `zeroCount`, `zeroThreshold`, `positive`, `negative`

An exponential histogram's, as [OpenTelemetry](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#exponentialhistogram) lays it out. `positive` and `negative` each hold `offset` and `bucketCounts`, an array of doubles.

`resetHint`

`yes` or `no` where the sender said whether the histogram's counter reset, and null where it did not say.

`bucketCounts`, `explicitBounds`

A histogram of custom buckets: its bucket counts and their bounds, as [OpenTelemetry](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#histogram) lays them out.

`exemplars`

The series' exemplars, on its last record of the request, each shaped as an OTLP point's exemplar: `time`, `value`, `traceId`, `spanId`, and its other labels in `filteredAttributes`. A series sent with exemplars alone is one record, its `value` null.

`startTime`

When the series started counting, which remote\_write 2.0 may send with a sample or histogram. Null from 1.0, and where the sender did not say.

Each metric family's metadata is a record of its own. Its `time` and `value` are null, `name` is the family's, `description` its help and `unit` its unit. Its `type` is as an OTLP metric's: a counter is `sum` with `isMonotonic` true. A pipeline that wants samples alone drops records with no `time`.

Over remote\_write 2.0, a family's metadata record comes once a request, ahead of the family's first series.

Classic histograms arrive as samples of their `_bucket`, `_sum` and `_count` series, as a [classic histogram](https://prometheus.io/docs/concepts/metric_types/#histogram) is laid out.

This pipeline keeps one metric and reads its job as text:

```
if (name != "http_requests_total") drop();
job = attributes["job"] as str;
```

## Fields it gives

`__source.remote` holds the IP address the request came from: the balancer's, when one is in front of the workers.

| Field | Type | Holds |
| --- | --- | --- |
| `__source.name` | `str` | The name of the source that read the record. |
| `__source.type` | `str` | The kind of source that read it, its `type`, such as `kafka`, `s3List`, `httpApi` or `hecReceiver`. |
| `__source.remote` | `str` | The address of the sender that delivered the record to a receiver. |

## Tokens and certificates

Anyone a receiver admits can put records into the group's pipelines, so a receiver needs `tokens`, an `identity` with authorities, or both. The leader refuses a receiver with neither.

-   **Tokens:** a sender presents one as `Authorization: Bearer <token>`, Prometheus's `authorization` block. The receiver refuses any other request before reading its body. Without an identity, the token crosses the network in the clear, so use a token alone only behind a balancer that terminates TLS.
-   **Client certificates:** with an [identity](/docs/secrets#identities), the receiver serves TLS with the identity's certificate and key. If the identity has authorities, every connection, health checks included, must present a certificate that chains to one of them: Prometheus's `tls_config` with `cert_file` and `key_file`.

With both, a sender presents a certificate and a token.

## What it answers

The receiver answers success only once every destination has the request's samples, so a sender that retries an unanswered request loses nothing. The receiver takes a request whole or refuses it whole, and each refusal's body says why.

**204**

Every destination has the request's samples.

**503**, with `Retry-After`

The receiver is full, [blocked](/docs/delivery#hold) or stopping, or the sender took more than 30 seconds to send the whole body. A sender may send it again.

**401**

The request has no token, gives one other than as `Bearer <token>`, or gives one the receiver does not hold.

**400**

The body does not read as remote\_write, or it holds a series or histogram remote\_write does not allow, such as a label with an empty name or a histogram whose count is not what its buckets hold. The answer's body names what is wrong. A sender may drop it.

**413**

The body is larger than `maxRequestBytes`, or what it holds decompressed and decoded is more than a receiver holds at once. Send smaller requests, or raise `maxRequestBytes`.

**415**

The request is neither remote\_write 1.0 nor 2.0. The answer's body names what is wrong.

## Health check

`GET /health` on the receiver's port answers 200 while the receiver takes requests and 503 while it is full, blocked or stopping. It needs no token. Point a balancer's health check or a readiness probe at it, so a busy worker gets no requests until it can take them. Under an identity with authorities it needs a client certificate like any other request.

## Senders it takes

Any sender of remote\_write 1.0 or 2.0 works.

-   Prometheus's [remote\_write](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write), in server or agent mode. Set `protobuf_message: io.prometheus.write.v2.Request` to send 2.0, which carries each series' metadata and start time.
-   Grafana Alloy's `prometheus.remote_write`.
-   VictoriaMetrics' `vmagent`.
-   The OpenTelemetry Collector's `prometheusremotewrite` exporter, exponential histograms included.

Grafana Alloy sends to it what its scrape components forward to `prometheus.remote_write.smeltr.receiver`, with `PROM_TOKEN` set in its environment:

```
prometheus.remote_write "smeltr" {
  endpoint {
    url = "https://workers.example.com:9090/api/v1/write"
    bearer_token = sys.env("PROM_TOKEN")
    tls_config {
      ca_file = "/etc/alloy/smeltr-ca.pem"
    }
  }
}
```

## Open its port

Every worker in the group listens on the receiver's `port`. Open that port to senders on each worker, and give senders one address that reaches them all, such as a balancer's. On a Unix host, a request sent while a worker deploys a new configuration waits, and the new configuration answers it.

-   On Kubernetes, add the port to the worker pods and put a Service in front of them: [Expose a receiver](/docs/enroll-worker/kubernetes#receiver).
-   On a host or in a container, open the host's firewall or publish the container's port: [Enroll a worker on a host](/docs/enroll-worker/host).

## Verify

Point a sender at the receiver and check in its own logs or metrics that its requests succeed. Then find its samples at the destination.

## Troubleshooting

**The sender cannot connect**

Check that the port is open on every worker and that the sender's address reaches them. An identity with authorities refuses a connection without a certificate that chains to one of them.

**Every request gets 401**

Check that the sender presents `Authorization: Bearer <token>`, with a token one of the receiver's credentials holds.

**Every request gets 415**

The sender does not speak remote\_write 1.0 or 2.0. The answer's body names what is wrong.

**Every request gets 503**

Check the source's status. A blocked receiver takes nothing until its destinations take records again. A full one takes more with more workers.

**The same samples arrive twice**

A sender that hears no answer may send the request again. Give it a `remote_timeout` longer than its destinations take to deliver.

## See also

-   [Sources](/docs/sources)
-   [HTTP receiver source](/docs/sources/http-receiver)
-   [HEC receiver source](/docs/sources/hec-receiver)
-   [OTLP receiver source](/docs/sources/otlp-receiver)
-   [Credentials](/docs/secrets)
-   [Delivery](/docs/delivery)