Prometheus remote_write

Sends each record as a sample over remote_write 1.0 or 2.0 to any store that takes it.

Example

Sends to a Mimir tenant, and reads the Authorization header's value from the credential PROMETHEUS_TOKEN.

{
  "id": "metrics-out",
  "type": "prometheus",
  "url": "https://mimir.example.com/api/v1/push",
  "headers": {
    "X-Scope-OrgID": "tenant-1"
  },
  "credentials": {
    "Authorization": "PROMETHEUS_TOKEN"
  }
}

Before you start

  • Find the store's remote_write URL, path included:
  • To send native histograms, make sure the store takes them. See its docs.
  • To send remote_write 2.0, make sure the store takes it. See its docs. 1.0 is the default.
  • To keep exemplars, make sure the store keeps them. See its docs.
  • Allow workers to reach the store.
  • Put each secret header's whole value in a credential. For Authorization, the credential holds Bearer <token>, prefix included.

Settings

It also takes the settings every destination takes.

url string required

The store's remote_write endpoint, http:// or https://. It sends a credential only over https://, or to this host.

Examples
https://prometheus.example.com/api/v1/writehttps://mimir.example.com/api/v1/push

version string default 1.0

The version of remote_write it sends.

1.0
The default.
2.0
Each series carries its family's latest metadata and its start time. A store that does not take it stalls the destination until this is set back to 1.0.

headers map default none

Headers it sends on every request, each name to its value. Smeltr sets Content-Type, Content-Encoding, Content-Length, Transfer-Encoding, Host, Cookie and X-Prometheus-Remote-Write-Version itself.

Example
X-Scope-OrgID = tenant-1

credentials map default none

Headers whose values are secret, each name to the credential holding its whole value, such as Bearer <token>.

Example
Authorization = PROMETHEUS_TOKEN

Advanced

identity identity default none

The connection identity it connects under, for a private CA or a client certificate. Only on an https:// URL.

Example
prometheus-client

maxRequestBytes size default 1 MiB

The most one request carries, before compression. A record larger than this still goes as a single record. 4 KiB to 100 MiB.

Examples
4 MiB256 KiB10 MB

bufferBytes size default 64 MiB, or one request if that is larger

How much undelivered output it may hold. 4 MiB to 1 GiB, and at least one request.

Examples
256 MiB64 MiB1 GiB

stallAfterSecs integer default 30

Seconds one batch may take to deliver, every retry included, before the destination reports it stalled. 1 to 120.

Examples
60120

Records it sends

Each record is one sample, a float or a native histogram, in the shape the Prometheus receiver gives, so its records pass through unchanged. An OTLP receiver's metric points have the same shape. A float sample:

{
  "time": "2026-10-11T18:00:00.123Z",
  "name": "http_requests_total",
  "value": 1027,
  "attributes": {
    "job": "api",
    "instance": "api-1:9100",
    "code": "200"
  }
}
name
A non-empty string: the series' metric name, sent as its __name__ label.
value
A number.
time
A datetime, sent as its milliseconds.
attributes
An object of the series' other labels. A number or a boolean is sent as its text. A label that is null or empty is none, and a record with no labels but its name may leave attributes out or null.
type
Absent, null, gauge or sum for a float sample. exponentialHistogram for a native histogram, or histogram for a histogram of explicit bounds. Neither needs value. A summary is refused.

A histogram also needs these, as the Prometheus receiver gives them:

count, sum
Numbers.
aggregationTemporality
2 for a counter's histogram; absent or null for a gauge histogram. A delta, 1, is refused.
scale, zeroCount, zeroThreshold, positive, negative
An exponential histogram's.
bucketCounts, explicitBounds
A histogram's, sent as a classic histogram: its _bucket, _sum and _count series.
resetHint
yes or no, sent as an exponential histogram's reset hint. Left out, the hint sent is unknown.

Any record may also carry exemplars, as either receiver gives them:

exemplars
An array of objects, each a value, a time or none for the record's, traceId, spanId and filteredAttributes. An exemplar the store would not keep is left out, and its record still sent.

A record with no time and no value is a family's metadata, as the Prometheus receiver gives it: its description, unit and type go as the family's. A point's description or unit, as an OTLP point has them, goes as its family's too.

Over remote_write 2.0, each series carries its family's latest metadata. A record's startTime goes as its series' start time.

The destination refuses a record that is no sample, and names why. Choose what happens to it → A value that was NaN or an infinity arrives with none, because JSON holds neither, so such a record is refused. Staleness markers are not sent.

Requests

A request is remote_write 1.0, or 2.0 with version set to it. A series' records in a request go oldest first.

The destination sends one request at a time, and each lands before the next is sent, so a series' samples arrive in order.

Answers

The store answersThe destination
2xxcounts every record the request carried as delivered.
429, 5xxbacks off and sends the request again.
400, 409, 413, 422 to 2.0, saying it wrote some of the requestcounts the request as delivered, and what the store did not write in smeltr_destination_unwritten_total.
400, 409, 413, 422splits the request in halves and sends each. Halves the store takes land; a sample it still refuses on its own, the destination refuses alone.
415, or a success that stored nothing, to 2.0stalls, and reports that the store takes only remote_write 1.0.
401, 403stalls, and reports that the store refused its credentials.
anything else, a redirect includedstalls, and reports the status. It never follows a redirect.

A batch counts as delivered once every request carrying it gets a 2xx. A sample sent again after a failure may already be in the store.

Stores it sends to

A store may refuse a sample older than the last it holds for that series, such as one from a second worker or another writer. Each store's out-of-order docs:

Verify

Send a sample through the source, then query its series in the store. If the destination stalls, the Status page flags it, and the worker's log names the status the store answered.

Troubleshooting

Every record is refused
The refusal names the field at fault. A float sample needs name, value and time; a histogram needs count, sum and its buckets in place of value. A pipeline renames a record's own fields to them.
The store answers 404 or 405
Give the remote_write URL, path included, and check that the store's remote_write receiver is on.
The store answers 415, or a success that stored nothing
The store does not take remote_write 2.0. Set version to 1.0.
The store answers 401 or 403
Check that the credential holds the header's whole value, prefix included, such as Bearer <token>, and that the store needs no other header, such as a tenant's.
Samples are refused as out of order
A record arrived older than one of its series already sent, from the source itself, from another writer, or from a second worker sending the same series. Send each series from one worker, in time order, or turn on the store's out-of-order window.

See also