Pipeline reference

Code step

A Code step runs a short program on each record it processes.

The program's statements run in order, top to bottom. It reads the record's fields by name and writes fields that later steps and destinations see; assigning null removes a field. Every name has one type. This page gives worked examples of common tasks, each showing a record before and after the step. The expression reference gives the rules its code follows, and lists the operators and functions it can use.

Settings

It also takes the settings every step takes.

Common tasks

Set, rename and remove fields

Assign to set a field, and assign null to remove one:

let label = "checkout";
service = label.toUpperCase();
debug = null;

{  "service": "cart",  "debug": "trace-7",  "status": 200}

{  "service": "CHECKOUT",  "status": 200}

delete removes fields from the record, and move renames one. Inside an if, loop or try, delete cannot remove a field bound outside it; assign it null there instead.

delete(password, debug);
move(msg, message);

{  "user": "ana",  "password": "hunter2",  "debug": true,  "msg": "signed in"}

{  "message": "signed in",  "user": "ana"}

Choose a value

if and else choose which statements run for each record. For a short value selection, write severity = status >= 500 ? "error" : "normal";.

if (status >= 500) {
  severity = "error";
} else {
  severity = "normal";
}

{  "status": 503}

{  "status": 503,  "severity": "error"}

switch compares one value with each case in order. Cases with no statements between them share the next one's, and break ends the switch:

switch (status) {
  case 200:
  case 204:
    outcome = "ok";
    break;
  case 404:
    outcome = "missing";
    break;
  default:
    outcome = "error";
}

{ "status": 204 }

{ "status": 404 }

{ "status": 503 }

{ "outcome": "ok", "status": 204 }

{ "outcome": "missing", "status": 404 }

{ "outcome": "error", "status": 503 }

Filter records

drop() discards the record. It stands under an if, never inside a loop; a loop sets a flag, and an if after the loop drops on it. The step's filter is a condition. Where it is true, the body runs on the record. Where it is false or null, the body does not run, and the record continues with its existing values.

if (path == "/health") drop();

{ "path": "/health" }

{ "path": "/pay" }

{ "path": "/pay" }

Parse a field

The parse functions read text into an object, as a Parser step does, and lookup reads a lookup table, as a Lookup step does. A call stands where a statement writes its object: Object.assign($, f(…)), $ = f(…), x = f(…) and x: T = f(…), or a let. Smeltr reads everything but the text and the key when the pipeline compiles, so you write it as literals. Text the function cannot read marks the record with an Unparsable exception and writes nothing.

Object.assign($, kv(line));

{  "line": "user=ana action=\"sign in\""}

{  "action": "sign in",  "line": "user=ana action=\"sign in\"",  "user": "ana"}

A parse function converts a field that a declared type names to it; every other field keeps the type the format gives, which is str for all but JSON. The declared type turns status into an i64:

let hit: {status: i64, path: str} = grok(line, ["%{VERB:verb} %{URIPATH:path} %{INT:status}"], {patterns: {VERB: "[A-Z]+"}});
status = hit.status;
path = hit.path;

{  "line": "GET /pay 503"}

{  "line": "GET /pay 503",  "path": "/pay",  "status": 503}

The parse functions are json, kv, cef, csv and grok, each with its options.

Look up a table

lookup reads the row of a lookup table that a key matches. fields: each output named for its column, "column", or for its column and its default, ["column", "default"]. A field with no match and no default is left out.

Object.assign($, lookup("users", user, {column: "user", fields: {team: "team", level: ["level", "0"]}}));

{ "user": "alice" }

{ "user": "zed" }

{ "level": "3", "team": "payments", "user": "alice" }

{ "level": "0", "user": "zed" }

Work with arrays

filter keeps the items of an array for which an arrow's expression is true. map replaces each item with the arrow's value. The original array remains unless you assign the result back to it, and the calls chain: values.filter(x => x > 2).map(x => x * 2).

values = [1, 2, 3, 4];
kept = values.filter(x => x > 2);
doubled = kept.map(x => x * 2);

{  "id": 7}

{  "id": 7,  "values": [1,2,3,4],  "kept": [3,4],  "doubled": [6,8]}

for...of visits each item in an array.

let total = 0;
for (let value of [4, 5, 6]) {
  total = total + value;
}
sum = total;

{  "id": 7}

{  "id": 7,  "sum": 15}

Split a record

unroll replaces a record with one record per element of an array field. With true each new record keeps the other fields. It changes how many records there are, so it stands outside every if, loop and try.

unroll(items, true);

{ "order": "A1", "items": ["pen", "ink"] }

{ "items": "pen", "order": "A1" }

{ "items": "ink", "order": "A1" }

Reuse a calculation

A local function names a repeated calculation. The body's last expression, written without a semicolon, is its value, as is a return.

function addTax(amount: f64): f64 {
  return amount * 1.2;
}
price = addTax(10);

{  "id": 7}

{  "id": 7,  "price": 12.0}

Handle bad data

Use items ?? [] when a missing array should default to an empty one. Only the selected branch can mark the record with an exception. For example, d == 0 ? 0 : n / d returns zero when d is zero, without a DivideByZero. A try answers what is left, one kind at a time:

try {
  ratio = bytes / packets;
  count = parseInt(text);
} catch (e: DivideByZero | NaN) {
  ratio = 0.0;
} catch (e) {
  problem = e.kind;
}

{  "bytes": 1200,  "packets": 0,  "text": "many"}

{  "bytes": 1200,  "packets": 0,  "ratio": 0.0,  "text": "many"}

Dividing by zero stops the try before parseInt runs, so the first catch takes the record. A record with packets but unreadable text would run the second catch.

See also

• Expression reference
• Pipeline steps
• Exception answers
• Unroll step