Pipeline reference

Parser step

A Parser step reads one text field of each record and writes the fields it finds there into the record.

The input is usually _raw, the text a source received. The declared fields are the ones later steps calculate with, each with its type. Suggest fields proposes declared fields from the sample and production. In each example, the input is the raw text of each record and the output is the parsed record, with _raw left out.

Example

This step reads JSON and declares service as str and status as i64 (an integer).

Step

Saved JSON

{
  "id": "parse",
  "type": "parser",
  "input": "_raw",
  "kind": { "type": "json" },
  "declared": [
    { "name": "service", "type": "str" },
    { "name": "status", "type": "i64" }
  ]
}

{"service":"checkout","status":200,"region":"eu"}

{
  "service": "checkout",
  "status": 200,
  "region": "eu"
}

Both declared fields arrive typed, and the undeclared region passes through.

Settings

It also takes the settings every step takes.

Formats

kind names the format as { "type": "json" }, with the format's own settings beside type. Every format also takes the Advanced setting splitDots.

JSON

{ "type": "json" } reads one JSON object with named fields. It has no settings of its own.

CSV

{ "type": "csv" } reads values in a fixed column order.

Key/value

{ "type": "keyValue" } reads named pairs such as status=200.

Regex/Grok

{ "type": "grok" } reads text that a regular expression or Grok pattern matches, such as a syslog line.

Advanced

Every format takes this setting. Split keys at dots shows what it does.

How it works

Text the format cannot read raises an Unparsable exception, and a value that does not convert to its type raises UnparsableField.

A Parser step can have a filter or sit inside an If step's branches. Records it skips keep their existing values and receive no errors or warnings from it.

JSON objects

A record must be one JSON object. Top-level arrays, strings, numbers and literals do not parse. The parser accepts a trailing comma in an object or array, such as {"status":200,}; it does not support other JSON5 extensions.

Fields you do not declare pass through to the output as they arrived. Nested fields use dots: event.dataset is the field dataset inside the object event. To declare an object, choose the type object and list its fields. A closed object keeps only the fields you list and drops its other fields; turn on open to keep them.

CSV columns and quoting

Each record is one data row, and the columns name its values by position. The parser does not read a header row. The parser drops a column typed skip, while the columns after it keep their positions. It ignores extra values beyond the columns; a missing value is null, and the output leaves out a null field.

Double quotes can enclose delimiters, and two quotes inside a quoted field represent one literal quote. An unclosed quote raises an Unparsable exception.

This step reads comma-separated values with these columns:

Step

Saved JSON

{
  "id": "parse",
  "type": "parser",
  "input": "_raw",
  "kind": { "type": "csv", "delimiter": "," },
  "declared": [
    { "name": "service", "type": "str" },
    { "name": "debug", "type": "skip" },
    { "name": "status", "type": "i64" }
  ]
}

checkout,trace-1,200

"check,out",trace-2

"say ""hi""",trace-3,404,extra

{
  "service": "checkout",
  "status": 200
}

{
  "service": "check,out"
}

{
  "service": "say \"hi\"",
  "status": 404
}

Key/value pairs

The text is a list of pairs. pairSep divides one pair from the next, and kvSep divides a pair's key from its value. With nextKey on, the default, a key is the text after the last pairSep before a kvSep, and its value runs to the pairSep before the next key: in msg=user logged in id=5, msg is user logged in. A kvSep inside a value, with no pairSep before it, belongs to the value, so url=http://x/?a=1 keeps http://x/?a=1 whole. The parser trims whitespace around a key or an unquoted value. A word before the first key is a key whose value is the text "true".

With nextKey off, a value ends at the first pairSep, any word without kvSep is a key whose value is the text "true", and the parser keeps whitespace. In level=info debug x=1, level is info and debug is "true".

quote lets a value hold pairSep: the value starts and ends with it. Inside it, a backslash before quote or before another backslash stands for that character. Only pairSep may follow the closing quote, and with nextKey on, spaces before it. Anything else there, or an unclosed quoted value, raises an Unparsable exception.

This step separates pairs with one space and keys from values with =, and declares service as str and status as i64:

Step

Saved JSON

{
  "id": "parse",
  "type": "parser",
  "input": "_raw",
  "kind": { "type": "keyValue", "pairSep": " ", "kvSep": "=" },
  "declared": [
    { "name": "service", "type": "str" },
    { "name": "status", "type": "i64" }
  ]
}

service=checkout status=200

{
  "service": "checkout",
  "status": 200
}

With quote set, a quoted value can contain spaces:

Step

Saved JSON

{
  "id": "parse",
  "type": "parser",
  "input": "_raw",
  "kind": {
    "type": "keyValue",
    "pairSep": " ",
    "kvSep": "=",
    "quote": "\""
  },
  "declared": [
    { "name": "message", "type": "str" },
    { "name": "status", "type": "i64" }
  ]
}

message="order placed" status=200

{
  "message": "order placed",
  "status": 200
}

Regex/Grok patterns

The Regex/Grok format reads regular expressions and Grok patterns. A pattern is a regular expression, and each named group, (?<name>...), captures its match in a field. Grok adds named patterns on top: %{NAME} inserts a built-in or custom named pattern, and %{NAME:field} also captures its match in a field. The two forms mix freely in one pattern.

The captures are the step's fields, each with a type. With several patterns, the first one that matches a record parses it. If none matches, the record has an Unparsable exception. A matched pattern with an absent optional capture leaves that field null.

The parser groups each named pattern it inserts, so a choice or a case-insensitive flag applies within that pattern:

• The built-in LOGLEVEL ignores case. In %{LOGLEVEL} ERROR, the literal ERROR still requires uppercase. WARN ERROR matches; warn error does not.
• A custom pattern AB defined as a|b makes ^%{AB}c$ match ac and bc, not a alone.

This step is a plain regular expression, with status as i64:

Step

Saved JSON

{
  "id": "parse",
  "type": "parser",
  "input": "_raw",
  "kind": {
    "type": "grok",
    "patterns": ["^(?<service>[a-z]+) (?<status>\\d+)$"]
  },
  "declared": [
    { "name": "service", "type": "str" },
    { "name": "status", "type": "i64" }
  ]
}

checkout 200

{
  "service": "checkout",
  "status": 200
}

The same parse with Grok's named patterns is ^%{WORD:service} %{INT:status}$. In both, the anchors ^ and $ require the whole input to match.

Step

Saved JSON

{
  "id": "parse",
  "type": "parser",
  "input": "_raw",
  "kind": {
    "type": "grok",
    "patterns": ["^%{WORD:service} %{INT:status}$"]
  },
  "declared": [
    { "name": "service", "type": "str" },
    { "name": "status", "type": "i64" }
  ]
}

checkout 200

{
  "service": "checkout",
  "status": 200
}

Custom Grok patterns

A custom pattern is a named pattern you define. A Grok pattern inserts it as %{NAME}, the same way it inserts a built-in. A custom pattern has one of two scopes:

• One step: defined under the step's pattern definitions. Only that step can name it.
• A group: defined under Build, Resources, Grok patterns. Any Grok step in the group can name it. A group also sees the patterns of shared/.

A name is looked up in the step first, then the group, then the built-ins.

This step uses the custom pattern AB and the built-in LOGLEVEL:

Step

Saved JSON

{
  "id": "parse",
  "type": "parser",
  "input": "_raw",
  "kind": {
    "type": "grok",
    "patterns": ["^%{AB:kind}c %{LOGLEVEL:level} ERROR$"],
    "customPatterns": [{ "name": "AB", "pattern": "a|b" }]
  },
  "declared": [
    { "name": "kind", "type": "str" },
    { "name": "level", "type": "str" }
  ]
}

ac warn ERROR

bc WARN ERROR

{
  "kind": "a",
  "level": "warn"
}

{
  "kind": "b",
  "level": "WARN"
}

Built-in Grok patterns

A pattern is a regular expression in the syntax of Rust's regex crate. A pattern takes no lookarounds and no backreferences. A field name in %{NAME:field} is one or more dot-separated parts, each of ASCII letters, digits and _, not starting with a digit. The field's type is the step's declaration, so there is no %{NAME:field:type} form. The parser refuses patterns nested more than 32 deep.

Split keys at dots

In the GUI, Split keys at dots is under Advanced.

On, a dot in a field name nests the field: geo.city is the field city inside the object geo. Off, geo.city is one field name with a dot in it, which a later step reads as $["geo.city"]. The default follows the format. JSON and CSV have their own way to nest or none at all, so it is off. Key/value and Regex/Grok captures can nest only through a dot, so it is on.

Turned on for JSON, a dotted field name becomes a nested object:

Step

Saved JSON

{
  "id": "parse",
  "type": "parser",
  "input": "_raw",
  "kind": { "type": "json", "splitDots": true },
  "declared": [{ "name": "status", "type": "i64" }]
}

{"geo.city":"Oslo","status":200}

{
  "geo": {
    "city": "Oslo"
  },
  "status": 200
}

Turned off for Key/value, a dotted key stays one field name:

Step

Saved JSON

{
  "id": "parse",
  "type": "parser",
  "input": "_raw",
  "kind": {
    "type": "keyValue",
    "pairSep": " ",
    "kvSep": "=",
    "splitDots": false
  },
  "declared": []
}

src.ip=10.0.0.1 action=allow

{
  "src.ip": "10.0.0.1",
  "action": "allow"
}

Unparsable records

Text the format cannot read marks the record with an Unparsable exception. Its exception answer decides what happens to it:

• tag (default): the record continues with its exceptions in smeltr.exception. The fields parsed before the exception stay. Route delivery conditions decide where it goes.
• block: the source stops and processing waits for a new deployment. Correct the cause, then review and deploy the change.
• drop: Smeltr counts the record and discards it.
• deliver: the record goes to the destinations you pick, with its exceptions in the tag field.
• ignore: the record goes on unmarked, and Smeltr still counts the exception.

A value the parser cannot read as its declared type marks the record with an UnparsableField exception, and its field becomes null. Status → Issues warns when production stops matching a declared field.

In these examples the second record is cut off after "status":. Tagged, it goes on with the field read before the exception:

Step

The group's exception answers: every exception tag.

Saved JSON

{
  "id": "parse",
  "type": "parser",
  "input": "_raw",
  "kind": { "type": "json" },
  "declared": [{ "name": "status", "type": "i64" }]
}

{"service":"checkout","status":200}

{"service":"checkout","status":

{
  "service": "checkout",
  "status": 200
}

{
  "service": "checkout",
  "smeltr": {
    "exception": {
      "pipeline": "example",
      "exceptions": [
        {
          "step": "parse",
          "kind": "Unparsable",
          "message": "_raw is \"{\\\"service\\\":\\\"checkout\\\",\\\"status\\\":\", which json could not parse"
        }
      ]
    }
  }
}

Dropped, only the record that parsed goes on:

Step

The group's exception answers: every exception drop.

Saved JSON

{
  "id": "parse",
  "type": "parser",
  "input": "_raw",
  "kind": { "type": "json" },
  "declared": [{ "name": "status", "type": "i64" }]
}

{"service":"checkout","status":200}

{"service":"checkout","status":

{
  "service": "checkout",
  "status": 200
}

See also

• Pipeline steps
• Exception answers