Introduction

jlens evaluates JSONata expressions against JSON documents. You can use it to filter, query and transform payloads, and to express business rules whose outcome must be explained.

It ships in three forms that share a single engine:

To try it without installing anything, open the web editor: it runs jlens in your browser, with completion, diagnostics, and the explanation and audit tree of every result.

What sets jlens apart from other JSONata implementations:

• Explainable evaluation. jlens can record an audit tree of how an expression reached its result: which comparisons passed or failed, which items a filter rejected, and which values were bound. You can attach business labels through ordinary JSONata comments (annotations), so the expression stays valid in any other JSONata implementation.
• Schema-driven assistance. jlens infers structural metadata from sample documents. It then uses that metadata to offer completion, diagnostics and hover information for expressions being written, in the browser and in editors (language server).
• Conformance first. The engine is checked against the official JSONata test suite (2.2.2) on every change, and against jsonata-js itself with generated expressions every week; see Conformance testing.

Status: jlens 0.16 implements all of JSONata 2.2 (1,688 of the 1,690 official test cases, with the other two documented), host functions, evaluation audits, schema-driven editor support with a language server, a command line for scripts and agents that checks and tests expressions, and the browser package. See the roadmap.

Installation

jlens comes in three forms that share one engine. Install the ones you need.

Command line

Each GitHub release has the jlens binary for Linux (x86-64 and ARM64, static), macOS (Intel and Apple silicon) and Windows (x86-64), with a SHA256SUMS file. Download the archive for your platform and put jlens on your PATH, or let cargo binstall do it:

cargo binstall jlens-cli

From 0.6.0 on, each archive has a build attestation: a signed statement that the release workflow of this repository built it, from which commit. Check one with the GitHub CLI:

gh attestation verify jlens-0.16.0-x86_64-unknown-linux-musl.tar.gz --repo thkprado/jlens

Elsewhere, build it from source with Rust 1.88 or later and a C compiler:

cargo install jlens-cli                          # from crates.io
cargo install --git https://github.com/thkprado/jlens jlens-cli
cargo install --path crates/jlens-cli            # from a checkout

$ jlens --version
jlens 0.16.0

On macOS and Linux, Homebrew installs the same binary from the tap thkprado/homebrew-jlens (from 0.14.0 on), and brew upgrade follows the releases:

brew install thkprado/jlens/jlens

On Windows, Scoop installs it from the bucket thkprado/scoop-jlens (from 0.14.0 on), and scoop update jlens follows the releases:

scoop bucket add jlens https://github.com/thkprado/scoop-jlens
scoop install jlens/jlens

In a JavaScript project, the same binary will come from npm, in the package jlens-cli. It is not published yet: at the 0.14.0 release, npm held back the name of the Windows package, and jlens-cli waits for it. Until then, use one of the channels above. Once published:

npx jlens-cli --version              # run it without installing
npm install --save-dev jlens-cli     # or add it to the project; npm scripts then run `jlens`

jlens-cli runs the release binary for your platform, which npm installs from one of its optional dependencies, jlens-cli-<os>-<cpu>, with no install script. Each run starts Node.js first, which adds some tens of milliseconds. An install with --omit=optional leaves the binary out, and so does a platform without a release binary (Windows on ARM64, for one): jlens then says which package it looked for and exits with code 2.

With Docker or Podman, the container image ghcr.io/thkprado/jlens (from 0.14.0 on) runs the static Linux binary, for linux/amd64 and linux/arm64, with nothing else in the image. It is tagged with each version (0.16.0), each minor version (0.16) and latest. Give it -i to read standard input, or mount the current directory on /work to read files:

echo '{"a": [1, 2]}' | docker run --rm -i ghcr.io/thkprado/jlens eval '$sum(a)'
docker run --rm -v "$PWD:/work" ghcr.io/thkprado/jlens eval -f rule.jsonata -i order.json

The image runs as an unprivileged user (UID 65532): add --user "$(id -u):$(id -g)" for files that only you can read. It has a build attestation too:

gh attestation verify oci://ghcr.io/thkprado/jlens:0.16.0 --repo thkprado/jlens

Install the command through one channel. With more than one, the first jlens on the PATH runs, whatever its version; inside npm scripts and npx, the project’s own jlens-cli comes first. To see which one runs:

jlens --version
which -a jlens       # Linux and macOS: every jlens on the PATH, the one that runs first
where.exe jlens      # Windows

The CLI guide describes every command.

Rust crates

[dependencies]
jlens-core = "0.15"    # the language: parsing, evaluation, functions, auditing
jlens-schema = "0.15"  # optional: schema inference and the language service
serde_json = "1"

jlens-core depends on serde_json, indexmap, jlens-regress (a fork of regress) and a few small crates. The optional tracing feature adds tracing spans for evaluations; see the audit guide. The crate guide shows how to use the API.

Browser and JavaScript

The WebAssembly package is the npm package jlens-wasm, an ES module:

npm install jlens-wasm

From 0.6.0 on, the package is published with provenance: its page on npmjs.com links it to the commit and the workflow run that built it, and npm audit signatures checks it in a project that depends on it.

To build it from a checkout instead, you need:

• the wasm32-unknown-unknown Rust target;
• the wasm-bindgen CLI, at the version in Cargo.lock.

rustup target add wasm32-unknown-unknown
cargo install wasm-bindgen-cli --version <version in Cargo.lock> --locked
cargo xtask wasm          # writes crates/jlens-wasm/pkg

The WebAssembly guide covers loading the module in browsers, bundlers and Node.js.

This book

The documentation is an mdBook:

cargo install mdbook
mdbook serve docs        # http://localhost:3000
cargo xtask doc-check    # runs the examples and checks the links

First steps

This tour uses the jlens command line and the files in fixtures/. They describe a store order: a customer and a list of items, each with a type, a color, a unit price, a quantity and a tax. Every command runs from the repository root.

{
  "number": "SO-12345",
  "customer": { "login": "alice.walker99", "name": "Acme Stores" },
  "items": [
    { "code": "1", "color": "blue",  "type": "apparel",    "unitPrice": 10000.0, "quantity": 2, "tax": 1500.0 },
    { "code": "2", "color": "black", "type": "electronic", "unitPrice": 25000.0, "quantity": 1, "tax": 4000.0 },
    { "code": "3", "color": "white", "type": "electronic", "unitPrice": 12000.0, "quantity": 1, "tax": 3000.0 }
  ]
}

Query

A path selects values. Predicates in square brackets filter them:

$ jlens eval -i fixtures/online-store-order.json 'customer.name'
"Acme Stores"

$ jlens eval -i fixtures/online-store-order.json -c 'items[type = "electronic"].code'
["2","3"]

$ jlens eval -i fixtures/online-store-order.json '$sum(items.(unitPrice * quantity))'
57000

The language reference explains paths, operators and functions.

Write a rule

fixtures/shipping-rule.jsonata is a business rule. Variables hold intermediate totals, then a condition chooses the outcome:

(
  $total       := $sum(items.(unitPrice * quantity));
  $electronics := $sum(items[type = 'electronic'].(unitPrice * quantity));

  $approved := (
    /** @label "Authorized customer" @message "The customer login must be on the approved partners list." */
    customer.login in ["alice.walker99", "bob.harrison42"]
    and
    /** @label "Minimum order grand total" @message "The order grand total must exceed 50,000.00." */
    $total > 50000
    and
    /** @label "Electronics group" */
    ($electronics > $total * 0.30 + 1000)
  );

  $approved
    ? { "action": "FREE_SHIPPING", "code": "OK_200" }
    : { "action": "REGULAR_FEE", "code": "SHIP_FEE_402", "delivery": "SLOW" }
)

(The fixture has a few more checks; this is the gist.)

$ jlens eval -f fixtures/shipping-rule.jsonata -i fixtures/online-store-order.json
{
  "action": "FREE_SHIPPING",
  "code": "OK_200"
}

Explain the result

The /** @label ... */ comments are annotations. They are ordinary comments to other JSONata implementations, but jlens uses them to explain results:

$ jlens eval -f fixtures/shipping-rule.jsonata -i fixtures/online-store-order.json --explain
result: {"action":"FREE_SHIPPING","code":"OK_200"}
• $total := $sum(items.(unitPrice * quantity)) = 57000
• $electronics := $sum(items[type = 'electronic'].(unitPrice * quantity)) = 37000
  • [type = 'electronic']: 2 of 3 items passed; rejected: {"code":"1","color":"blue",...}
✔ $approved := ( customer.login in [...] and ... ) = true
  ✔ Authorized customer — customer.login in [...] → true ("alice.walker99" in [...])
  ✔ Minimum order grand total — $total > 50000 → true (57000 > 50000)
  ✔ Electronics group — ... → true
      ✔ $electronics > $total * 0.30 + 1000 → true (37000 > 18100)
• $approved ? { ... } : { ... } → {"action":"FREE_SHIPPING","code":"OK_200"} [then]

(Output shortened.) --audit json prints the full audit tree for programs to read. The audit guide explains audits, levels and annotations.

Get help while writing

jlens can infer a schema from sample documents. Using it, jlens check reports fields that do not exist, and the language service completes field names:

$ jlens schema infer fixtures/online-store-order.json -o order.schema.json
inferred from 1 sample(s): order.schema.json

$ jlens check --schema order.schema.json 'items[type = "electronic"].unitprice'
warning[W0201]: Field "unitprice" is not in the schema; did you mean `unitPrice`?
 --> <expr>:1:28
  |
1 | items[type = "electronic"].unitprice
  |                            ^^^^^^^^^

$ jlens schema complete --schema order.schema.json "items[type = 'electronic']."
code                     field     string
color                    field     string
type                     field     string
unitPrice                field     number
quantity                 field     integer
tax                      field     number

The same service drives editors in the browser; try the examples/web-editor demo (WebAssembly guide).

Next

• Embed the engine in a Rust program: Rust crate.
• Use it in a browser: WebAssembly.
• Process files and streams: Command-line interface.

Language overview

JSONata is a query and transformation language for JSON. An expression selects values from a document with location paths (Account.Order.Product), combines them with operators and functions ($sum(Product.(Price * Quantity))), and shapes the output with ordinary JSON syntax ({"total": ...}). It also has variables, functions and conditionals, so rules of any complexity can be written in it.

jlens implements JSONata 2.2 and is checked against the official test suite. Where it differs from the reference implementation, jsonata-js, Compatibility says so.

This reference is adapted from the JSONata documentation (© IBM Corp. and JSONata contributors, MIT license), reorganized and extended with jlens-specific notes. Every example in it is run against jlens by cargo xtask doc-check.

Reading the examples

Examples show an expression followed by its result after =>:

1 + 2
=> 3

When an example needs an input document, its first line names it. Most examples use one of three sample documents, listed at the end of this page:

input: address.json
Address.City
=> "Winchester"

A result of undefined means the expression produced nothing: JSONata’s empty sequence. The command line prints nothing for it, and the APIs return None or undefined.

Types

JSONata values have the JSON types plus one more:

JSONata is a superset of JSON. Every JSON document is a valid expression that evaluates to itself. A JSON template is therefore a good start for a transformation: replace its values with expressions.

input: address.json
{"name": FirstName & " " & Surname, "adult": Age >= 18, "phones": $count(Phone)}
=> {"name": "Fred Smith", "adult": true, "phones": 4}

Sequences

A path can match zero, one or many values, so JSONata expressions produce sequences. The rules for sequences shape every result:

1. An empty sequence is nothing. It is not null, and it does not appear in the output: an object field whose value is an empty sequence is left out.
2. A sequence of one value is that value. No array surrounds it.
3. A sequence of several values is output as an array.
4. Sequences are flattened. Values of nested sequences are pulled up into the outer one. Arrays from the input, or built with [...], keep their structure.

input: address.json
Phone[type = 'mobile'].number
=> "077 7700 1234"

input: address.json
Phone[type = 'office'].number
=> ["01962 001234", "01962 001235"]

input: address.json
{"fax": Phone[type = 'fax'].number, "name": FirstName}
=> {"name": "Fred"}

The second rule makes a single value and a one-element array interchangeable, which keeps expressions composable. When a consistent array is needed, the [] modifier keeps singleton results as arrays.

Comments

/* ... */ comments can appear anywhere whitespace can:

input: address.json
/* the person's full name */
FirstName & " " & Surname
=> "Fred Smith"

Comments that start with /** are documentation comments. jlens reads @label, @message and @id tags in them to describe checks in audits; see Audit annotations. Other JSONata implementations treat them as ordinary comments.

The rest of this reference

• Paths: navigating objects and arrays, predicates, wildcards, and the path stages (#, @, %).
• Building results: array and object constructors.
• Operators: arithmetic, comparison, Boolean, string, conditional and chaining operators, and their precedence.
• Sorting, grouping and aggregation.
• Programming: variables, blocks, functions, recursion, closures, partial application, function composition and the transform operator.
• Regular expressions and Dates and times.
• The function library, with a page per category.
• Errors: every error code.

Sample documents

address.json:

{
  "FirstName": "Fred",
  "Surname": "Smith",
  "Age": 28,
  "Address": {
    "Street": "Hursley Park",
    "City": "Winchester",
    "Postcode": "SO21 2JN"
  },
  "Phone": [
    { "type": "home", "number": "0203 544 1234" },
    { "type": "office", "number": "01962 001234" },
    { "type": "office", "number": "01962 001235" },
    { "type": "mobile", "number": "077 7700 1234" }
  ],
  "Email": [
    {
      "type": "work",
      "address": ["fred.smith@my-work.com", "fsmith@my-work.com"]
    },
    {
      "type": "home",
      "address": ["freddy@my-social.com", "frederic.smith@very-serious.com"]
    }
  ],
  "Other": {
    "Over 18 ?": true,
    "Misc": null,
    "Alternative.Address": {
      "Street": "Brick Lane",
      "City": "London",
      "Postcode": "E1 6RF"
    }
  }
}

invoice.json:

{
    "Account": {
        "Account Name": "Firefly",
        "Order": [
            {
                "OrderID": "order103",
                "Product": [
                    {
                        "Product Name": "Bowler Hat",
                        "ProductID": 858383,
                        "SKU": "0406654608",
                        "Description": {
                            "Colour": "Purple",
                            "Width": 300,
                            "Height": 200,
                            "Depth": 210,
                            "Weight": 0.75
                        },
                        "Price": 34.45,
                        "Quantity": 2
                    },
                    {
                        "Product Name": "Trilby hat",
                        "ProductID": 858236,
                        "SKU": "0406634348",
                        "Description": {
                            "Colour": "Orange",
                            "Width": 300,
                            "Height": 200,
                            "Depth": 210,
                            "Weight": 0.6
                        },
                        "Price": 21.67,
                        "Quantity": 1
                    }
                ]
            },
            {
                "OrderID": "order104",
                "Product": [
                    {
                        "Product Name": "Bowler Hat",
                        "ProductID": 858383,
                        "SKU": "040657863",
                        "Description": {
                            "Colour": "Purple",
                            "Width": 300,
                            "Height": 200,
                            "Depth": 210,
                            "Weight": 0.75
                        },
                        "Price": 34.45,
                        "Quantity": 4
                    },
                    {
                        "ProductID": 345664,
                        "SKU": "0406654603",
                        "Product Name": "Cloak",
                        "Description": {
                            "Colour": "Black",
                            "Width": 30,
                            "Height": 20,
                            "Depth": 210,
                            "Weight": 2
                        },
                        "Price": 107.99,
                        "Quantity": 1
                    }
                ]
            }
        ]
    }
}

library.json:

{
    "library": {
        "books": [
            {
                "title": "Structure and Interpretation of Computer Programs",
                "authors": ["Abelson", "Sussman"],
                "isbn": "9780262510875",
                "price": 38.90,
                "copies": 2
            },
            {
                "title": "The C Programming Language",
                "authors": ["Kernighan", "Richie"],
                "isbn": "9780131103627",
                "price": 33.59,
                "copies": 3
            },
            {
                "title": "The AWK Programming Language",
                "authors": ["Aho", "Kernighan", "Weinberger"],
                "isbn": "9780201079814",
                "copies": 1
            },
            {
                "title": "Compilers: Principles, Techniques, and Tools",
                "authors": ["Aho", "Lam", "Sethi", "Ullman"],
                "isbn": "9780201100884",
                "price": 23.38,
                "copies": 1
            }
        ],
        "loans": [
            {
                "customer": "10001",
                "isbn": "9780262510875",
                "return": "2016-12-05"
            },
            {
                "customer": "10003",
                "isbn": "9780201100884",
                "return": "2016-10-22"
            },
            {
                "customer": "10003",
                "isbn": "9780262510875",
                "return": "2016-12-22"
            }
        ],
        "customers": [
            {
                "id": "10001",
                "name": "Joe Doe",
                "address": {
                    "street": "2 Long Road",
                    "city": "Winchester",
                    "postcode": "SO22 5PU"
                }
            },
            {
                "id": "10002",
                "name": "Fred Bloggs",
                "address": {
                    "street": "56 Letsby Avenue",
                    "city": "Winchester",
                    "postcode": "SO22 4WD"
                }
            },
            {
                "id": "10003",
                "name": "Jason Arthur",
                "address": {
                    "street": "1 Preddy Gate",
                    "city": "Southampton",
                    "postcode": "SO14 0MG"
                }
            }
        ]
    }
}

Paths

A location path selects values from the input document. Field names separated by . navigate into nested objects, and every array along the way is searched item by item.

Navigating objects

input: address.json
Surname
=> "Smith"

input: address.json
Address.City
=> "Winchester"

A path that matches null returns it. A path that matches nothing returns nothing (an empty sequence), never an error: missing data is normal.

input: address.json
Other.Misc
=> null

input: address.json
Other.Nothing
=> undefined

Field names that are not plain identifiers go between backquotes:

input: address.json
Other.`Over 18 ?`
=> true

Navigating arrays

When a step selects an array, the following steps apply to each of its items, and the results are collected:

input: address.json
Phone.number
=> ["0203 544 1234", "01962 001234", "01962 001235", "077 7700 1234"]

A number in square brackets selects an item by position, from 0. Negative positions count from the end, and non-integer positions are rounded down. A position past the end selects nothing.

input: address.json
Phone[0]
=> {"type": "home", "number": "0203 544 1234"}

input: address.json
Phone[-1].number
=> "077 7700 1234"

input: address.json
Phone[8]
=> undefined

An array of positions selects several items:

input: address.json
Phone[[0, 2]].number
=> ["0203 544 1234", "01962 001235"]

input: address.json
Phone[[0..1]].number
=> ["0203 544 1234", "01962 001234"]

The index applies to the step it follows, not to the whole path so far. Phone.number[0] takes the first number of each phone. Parentheses apply the index to the whole result:

input: address.json
Phone.number[0]
=> ["0203 544 1234", "01962 001234", "01962 001235", "077 7700 1234"]

input: address.json
(Phone.number)[0]
=> "0203 544 1234"

Predicates

Square brackets around any other expression filter the items: an item is kept when the expression, evaluated with the item as context, is true. Values that are not Booleans are converted as by $boolean.

input: address.json
Phone[type = 'mobile'].number
=> "077 7700 1234"

input: invoice.json
Account.Order.Product[Price > 30 and Quantity > 1].`Product Name`
=> ["Bowler Hat", "Bowler Hat"]

input: library.json
library.books["Aho" in authors].title
=> ["The AWK Programming Language", "Compilers: Principles, Techniques, and Tools"]

Always getting an array

Because a single value and a one-item sequence are the same (see Sequences), a path can return a value or an array depending on the data. Empty square brackets [] on any step keep the result an array:

input: address.json
Address[].City
=> ["Winchester"]

input: address.json
Phone[type = 'home'].number[]
=> ["0203 544 1234"]

The context: $ and $$

$ is the context value: the item a step is evaluated against. At the start of an expression it is the whole input document, which is how a top-level array is indexed. $$ is always the root of the input, wherever the evaluation is.

input: [{"ref": [1, 2]}, {"ref": [3, 4]}]
$[0].ref
=> [1, 2]

input: [{"ref": [1, 2]}, {"ref": [3, 4]}]
$.ref
=> [1, 2, 3, 4]

The nested arrays of the last example were flattened into one sequence. To keep the structure, use an array constructor: $.[ref].

input: address.json
Phone[type = 'mobile'].($$.FirstName & ": " & number)
=> "Fred: 077 7700 1234"

Wildcards

* selects the values of all the fields of an object:

input: address.json
Address.*
=> ["Hursley Park", "Winchester", "SO21 2JN"]

input: address.json
*.Postcode
=> "SO21 2JN"

** selects descendants at any depth:

input: address.json
**.Postcode
=> ["SO21 2JN", "E1 6RF"]

Path stages

A path is a pipeline of stages. Each stage takes the sequence from the previous one:

A filter binds more tightly than a map, so books.authors[0] is the first author of each book. Parentheses change the grouping, as (books.authors)[0] does. Parentheses also end the scope of variables bound inside them.

Mapping with expressions

Any expression can be a step, not just a field name. Its context is each item in turn:

input: invoice.json
Account.Order.Product.(Price * Quantity)
=> [68.9, 21.67, 137.8, 107.99]

input: invoice.json
Account.Order.OrderID.$uppercase()
=> ["ORDER103", "ORDER104"]

Positions: #

#$i binds the position of each item, counted before later filters, to $i:

input: library.json
library.books#$i["Kernighan" in authors].{"title": title, "index": $i}
=> [
  {"title": "The C Programming Language", "index": 1},
  {"title": "The AWK Programming Language", "index": 2}
]

Joins: @

@$v binds each item to $v and keeps the context where it was, so the next step navigates from the same place. That cross-references arrays like a join in a database:

input: library.json
library.loans@$l.books@$b[$l.isbn = $b.isbn].{"title": $b.title, "customer": $l.customer}
=> [
  {"title": "Structure and Interpretation of Computer Programs", "customer": "10001"},
  {"title": "Compilers: Principles, Techniques, and Tools", "customer": "10003"},
  {"title": "Structure and Interpretation of Computer Programs", "customer": "10003"}
]

Parents: %

% selects the parent of the context value: the object whose field holds it. %.% is the grandparent. The parent is worked out from the path when the expression is compiled, so % only works in paths that reach the value through its parent (otherwise: error S0217).

input: invoice.json
Account.Order.Product.{"product": `Product Name`, "order": %.OrderID, "account": %.%.`Account Name`}
=> [
  {"product": "Bowler Hat", "order": "order103", "account": "Firefly"},
  {"product": "Trilby hat", "order": "order103", "account": "Firefly"},
  {"product": "Bowler Hat", "order": "order104", "account": "Firefly"},
  {"product": "Cloak", "order": "order104", "account": "Firefly"}
]

Building results

The output of an expression can have any shape. Array and object constructors use JSON syntax, with expressions wherever values (and object keys) go.

Arrays

Square brackets build an array of the values of the expressions inside them:

input: address.json
[Address.City, Other.`Alternative.Address`.City]
=> ["Winchester", "London"]

Unlike a sequence, an array built with [...] keeps its structure: it is neither flattened nor unwrapped when it has a single item. As a path step, it builds one array per context item:

input: address.json
Email.address
=> ["fred.smith@my-work.com", "fsmith@my-work.com", "freddy@my-social.com", "frederic.smith@very-serious.com"]

input: address.json
Email.[address]
=> [
  ["fred.smith@my-work.com", "fsmith@my-work.com"],
  ["freddy@my-social.com", "frederic.smith@very-serious.com"]
]

An array constructor can also select several paths at once, as a more precise alternative to wildcards:

input: address.json
[Address, Other.`Alternative.Address`].City
=> ["Winchester", "London"]

Ranges

Inside an array constructor, from..to produces the integers from from to to, both included. Both ends must be integers (error T2003, T2004), and a range can have at most 10 million items (D2014).

[1..5]
=> [1, 2, 3, 4, 5]

[1..3, 7..9]
=> [1, 2, 3, 7, 8, 9]

[1..5].($ * $)
=> [1, 4, 9, 16, 25]

input: address.json
[1..$count(Phone)].("Phone " & $)
=> ["Phone 1", "Phone 2", "Phone 3", "Phone 4"]

Objects

Braces build an object from key: value pairs. Keys are expressions that must evaluate to strings (error T1003); values are any expressions. Fields whose value is nothing are left out.

input: address.json
{"name": FirstName, "city": Address.City, "fax": Phone[type = 'fax'].number}
=> {"name": "Fred", "city": "Winchester"}

Where the constructor appears changes what it does:

• After . it is a map step and builds one object per item:

  input: address.json
  Phone.{type: number}
  => [
    {"home": "0203 544 1234"},
    {"office": "01962 001234"},
    {"office": "01962 001235"},
    {"mobile": "077 7700 1234"}
  ]
  

• Directly after a path (no .) it groups the items into a single object. Values for the same key are collected together; see Grouping.

  input: address.json
  Phone{type: number}
  => {
    "home": "0203 544 1234",
    "office": ["01962 001234", "01962 001235"],
    "mobile": "077 7700 1234"
  }
  

  With [], every group is an array, whatever its size:

  input: address.json
  Phone{type: number[]}
  => {
    "home": ["0203 544 1234"],
    "office": ["01962 001234", "01962 001235"],
    "mobile": ["077 7700 1234"]
  }
  

JSON literals

Values of every JSON type can be written as they are in JSON: strings (with JSON escapes, and either quote character), numbers, true, false, null, arrays and objects. Because JSONata is a superset of JSON, a JSON document makes a good template for the output.

input: invoice.json
{
  "account": Account.`Account Name`,
  "orders": Account.Order.{
    "id": OrderID,
    "total": $round($sum(Product.(Price * Quantity)), 2),
    "items": $count(Product)
  },
  "currency": "EUR"
}
=> {
  "account": "Firefly",
  "orders": [
    {"id": "order103", "total": 90.57, "items": 2},
    {"id": "order104", "total": 245.79, "items": 2}
  ],
  "currency": "EUR"
}

Operators

Arithmetic

Both operands must be numbers (error T2001, T2002), and a result that is not finite (such as 1/0) is an error (D1001). If either operand is nothing, the result is nothing.

5 + 2 * 3
=> 11

5 / 2
=> 2.5

-7 % 3
=> -1

input: invoice.json
Account.Order[0].Product[0].(Price * Quantity)
=> 68.9

"5" + 1
=> error T2001

missing + 1
=> undefined

Numbers are 64-bit floats, as in JavaScript, and results are printed the way JavaScript prints them:

0.1 + 0.2
=> 0.30000000000000004

Comparison

= and != never fail: values of different types are simply not equal. The ordering operators need two numbers or two strings (errors T2009, T2010). Comparing with nothing gives false.

1 + 1 = 2
=> true

"1" = 1
=> false

{"a": [1, 2]} = {"a": [1, 2]}
=> true

"apple" < "banana"
=> true

"world" in ["hello", "world"]
=> true

"hello" in "hello"
=> true

input: address.json
"01962 001234" in Phone.number
=> true

Boolean

and and or convert their operands to Booleans as $boolean does, and stop as soon as the result is known: the right side of false and x is never evaluated. Negation is a function, $not.

input: library.json
library.books["Aho" in authors and price < 50].title
=> "Compilers: Principles, Techniques, and Tools"

The other book by Aho has no price, and comparing nothing with 50 is false.

0 or "yes"
=> true

$not(true and false)
=> true

Strings

& joins strings. It is the only operator that converts its operands: non-strings become strings as by $string, and nothing becomes the empty string.

input: address.json
FirstName & " " & Surname
=> "Fred Smith"

5 & 0 & true & missing
=> "50true"

Conditionals

condition ? then : else evaluates then when the condition is true, and else otherwise. The condition is converted to a Boolean. Without an else part, a false condition produces nothing.

input: invoice.json
Account.Order.Product.(Price > 100 ? "premium" : "standard")
=> ["standard", "standard", "standard", "premium"]

false ? "yes"
=> undefined

Defaults

input: address.json
Other.Misc ?: "n/a"
=> "n/a"

input: address.json
Other.Misc ?? "n/a"
=> null

0 ?? 1
=> 0

Chaining: ~>

value ~> $f(args) calls $f(value, args). Chains read left to right instead of inside out:

input: invoice.json
Account.Order.Product.(Price * Quantity) ~> $sum() ~> $round(1)
=> 336.4

"  Hello  World  " ~> $trim() ~> $uppercase()
=> "HELLO WORLD"

With functions on both sides, ~> composes them into a new function; see Function composition. A regular expression on the right tests a string, which is useful in filters.

Other operators

Precedence

From the tightest binding to the loosest:

Unary minus binds more tightly than any binary operator. Parentheses override precedence:

(5 + 3) * 4
=> 32

1 < 2 and 2 < 3 or false
=> true

Sorting, grouping and aggregation

Sorting

The order-by stage ^(...) sorts the items of a path. Sort keys are expressions evaluated against each item, and they must give numbers or strings. Strings sort by Unicode code point. A key prefixed with > sorts in descending order; <, the default, sorts in ascending order. Further keys, separated by commas, break ties. The sort is stable.

input: invoice.json
Account.Order.Product^(Price).`Product Name`
=> ["Trilby hat", "Bowler Hat", "Bowler Hat", "Cloak"]

input: invoice.json
Account.Order.Product^(>Price, <Quantity).{"name": `Product Name`, "qty": Quantity}
=> [
  {"name": "Cloak", "qty": 1},
  {"name": "Bowler Hat", "qty": 2},
  {"name": "Bowler Hat", "qty": 4},
  {"name": "Trilby hat", "qty": 1}
]

input: invoice.json
Account.Order.Product^(Price * Quantity).(Price * Quantity)
=> [21.67, 68.9, 107.99, 137.8]

$sort sorts an array with a comparison function, for orders that keys cannot express.

Grouping

An object constructor directly after a path (without .) groups the items: it evaluates the key for each item, and items with the same key fall into the same group. Then it evaluates the value once per group, with the group’s items as context. Aggregations therefore work per group:

input: invoice.json
Account.Order.Product{`Product Name`: Price}
=> {"Bowler Hat": [34.45, 34.45], "Trilby hat": 21.67, "Cloak": 107.99}

input: invoice.json
Account.Order.Product{`Product Name`: $round($sum($.(Price * Quantity)), 2)}
=> {"Bowler Hat": 206.7, "Trilby hat": 21.67, "Cloak": 107.99}

The value expression runs against the whole group. A path in it therefore collects values from all the group’s items. To build one object per item, map over the group with $.:

input: invoice.json
Account.Order.Product{`Product Name`: {"price": Price, "qty": Quantity}}
=> {
  "Bowler Hat": {"price": [34.45, 34.45], "qty": [2, 4]},
  "Trilby hat": {"price": 21.67, "qty": 1},
  "Cloak": {"price": 107.99, "qty": 1}
}

input: invoice.json
Account.Order.Product{`Product Name`: $.{"price": Price, "qty": Quantity}}
=> {
  "Bowler Hat": [{"price": 34.45, "qty": 2}, {"price": 34.45, "qty": 4}],
  "Trilby hat": {"price": 21.67, "qty": 1},
  "Cloak": {"price": 107.99, "qty": 1}
}

Keys must be strings (error T1003). Several key/value pairs can group the same items in different ways, and a group that evaluates to nothing is left out.

input: library.json
library.books{
  "cheap": $count($[price < 30]),
  "expensive": $count($[price >= 30]),
  "unpriced": $count($[$not($exists(price))])
}
=> {"cheap": 1, "expensive": 2, "unpriced": 1}

Aggregation

Aggregation functions reduce an array to one value:

• $sum;
• $count;
• $max, $min;
• $average;
• $join for strings;
• $reduce for anything else.

input: invoice.json
$sum(Account.Order.Product.(Price * Quantity))
=> 336.36

input: invoice.json
{
  "orders": $count(Account.Order),
  "dearest": $max(Account.Order.Product.Price),
  "average quantity": $average(Account.Order.Product.Quantity)
}
=> {"orders": 2, "dearest": 107.99, "average quantity": 2}

input: invoice.json
$join(Account.Order.Product.`Product Name`, ", ")
=> "Bowler Hat, Trilby hat, Bowler Hat, Cloak"

Programming

JSONata is a functional language. Values are immutable, functions are values, and everything is an expression.

Blocks

Parentheses group expressions separated by ;. The expressions are evaluated in order, and the block’s value is the value of the last one. A block is also a scope for variables.

(1 + 2; "ignored"; 10 * 2)
=> 20

Variables

A name that starts with $ is a variable. $name := value binds a variable, and its value is the bound value. The binding is visible in the rest of the block and in nested blocks, but not outside the block.

input: invoice.json
Account.Order.Product.(
  $total := Price * Quantity;
  $tax := $total * 0.2;
  $round($total + $tax, 2)
)
=> [82.68, 26, 165.36, 129.59]

($x := 1; ($x := 2); $x)
=> 1

Predefined variables:

• $ is the context value; see Paths.
• $$ is the root of the input.
• The built-in functions ($sum, $string, …) are variables too, so a binding can replace one in its scope.

Applications can define more variables when they evaluate an expression: bindings, such as jlens eval --bindings, Bindings in Rust, or the bindings argument in JavaScript.

bindings: {"limit": 100, "rate": 0.2}
$limit * (1 + $rate)
=> 120

Functions

Calling functions

A function is called with its arguments in parentheses:

$substring("hello world", 0, 5)
=> "hello"

Many built-in functions use the context value when their first argument is omitted (their signature marks that argument with -). That makes them usable as path steps:

input: invoice.json
Account.Order.OrderID.$uppercase()
=> ["ORDER103", "ORDER104"]

Defining functions

function($a, $b) { body } defines a function. λ can be written instead of function. Functions are values: bind them to variables, pass them to other functions, return them.

(
  $volume := function($l, $w, $h) { $l * $w * $h };
  $volume(10, 10, 5)
)
=> 500

λ($x) { $x * 2 }(21)
=> 42

Missing arguments are undefined inside the function, and extra arguments are ignored.

Signatures

An optional signature after the parameter list declares the types of the arguments and of the result. Arguments are then checked when the function is called (errors T0410 and T0412).

(
  $repeat := function($s, $n)<sn:s> { $join([1..$n].($s)) };
  $repeat("ab", 3)
)
=> "ababab"

(
  $repeat := function($s, $n)<sn:s> { $join([1..$n].($s)) };
  $repeat(3, "ab")
)
=> error T0410

A signature is <params:result>, with one symbol per parameter:

These modifiers can follow a symbol:

• ? makes the parameter optional.
• + takes one or more arguments of the type.
• - takes the context value when the argument is missing.

A single value is accepted where an array is expected. For example, $sum has the signature <a<n>:n> and $length has <s-:n>. The function reference gives the signature of every built-in function.

Recursion and tail calls

A function bound to a variable can call itself through it:

(
  $factorial := function($n) { $n <= 1 ? 1 : $n * $factorial($n - 1) };
  $factorial(10)
)
=> 3628800

Each nested call uses stack, and evaluation stops with D1011 past a depth limit (see Guardrails). A call in tail position, the last thing a function does, does not nest. jlens runs it as a loop, so tail-recursive functions can iterate any number of times:

(
  $sumTo := function($n, $acc) { $n = 0 ? $acc : $sumTo($n - 1, $acc + $n) };
  $sumTo(100000, 0)
)
=> 5000050000

Closures

A function remembers the variables and the context value of the place where it was defined:

(
  $multiplier := function($k) { function($x) { $x * $k } };
  $triple := $multiplier(3);
  [$triple(2), $triple(5)]
)
=> [6, 15]

input: invoice.json
Account.(
  $account := function() { $.`Account Name` };
  Order[OrderID = 'order104'].Product.{"account": $account(), "product": `Product Name`}
)
=> [
  {"account": "Firefly", "product": "Bowler Hat"},
  {"account": "Firefly", "product": "Cloak"}
]

Higher-order functions

Functions that take or return functions are called higher-order. The library has several: $map, $filter, $reduce, $sift, $single and $sort.

(
  $twice := function($f) { function($x) { $f($f($x)) } };
  $twice(function($y) { $y + 3 })(7)
)
=> 13

Partial application

Calling a function with ? in place of some arguments returns a new function that takes the missing ones:

(
  $first5 := $substring(?, 0, 5);
  $first5("Hello, World")
)
=> "Hello"

(
  $firstN := $substring(?, 0, ?);
  $firstN("Hello, World", 3)
)
=> "Hel"

Function composition

When both sides of ~> are functions, the result is their composition: a function that applies the left one, then the right one.

(
  $normalize := $trim ~> $uppercase;
  $normalize("   Some   Words   ")
)
=> "SOME WORDS"

input: address.json
(
  $first5Capitalized := $substring(?, 0, 5) ~> $uppercase(?);
  $first5Capitalized(Address.City)
)
=> "WINCH"

Conditional logic

The conditional operator ? : and the default operators ?: and ?? cover if/then/else. Nested conditionals read like an else if chain:

input: invoice.json
Account.Order.Product.(
  Price > 100 ? "premium" :
  Price > 30 ? "standard" :
  "budget"
)
=> ["standard", "budget", "standard", "premium"]

Transforms

The transform operator |location|update| or |location|update, delete| produces a function. That function copies its argument and, in the copy, changes every object that location matches:

• update must evaluate to an object. Its fields are merged into each matched object, as $merge does. update is evaluated against the matched object.
• delete (optional) names the fields to remove: a string or an array of strings.

The original value is not modified.

input: invoice.json
(
  $copy := $ ~> |Account.Order.Product|{"Price": $round(Price * 1.2, 2)}, ["SKU", "Description"]|;
  $copy.Account.Order[0].Product
)
=> [
  {"Product Name": "Bowler Hat", "ProductID": 858383, "Price": 41.34, "Quantity": 2},
  {"Product Name": "Trilby hat", "ProductID": 858236, "Price": 26, "Quantity": 1}
]

Updates see the original values. Here Total uses the price before the 20% increase:

{"Price": 10, "Quantity": 3} ~> |$|{"Price": Price * 1.2, "Total": Price * Quantity}|
=> {"Price": 12, "Quantity": 3, "Total": 30}

A transform is a function, so it can be bound to a variable and reused, or chained with other transforms. An update that would make an object contain itself is an error (U1001) in jlens; see Compatibility.

Comments and annotations

/* ... */ comments are ignored. jlens reads documentation comments (/** ... */) placed before a subexpression as audit annotations: labels and messages that explain the checks of a rule.

Regular expressions

A regular expression is written between slashes, optionally followed by flags:

/pattern/flags

The syntax and semantics are those of JavaScript regular expressions (ECMAScript). They include character classes, quantifiers (greedy and lazy), groups, backreferences, lookahead and lookbehind, and Unicode escapes. jlens uses the regress engine, which follows the ECMAScript specification. An invalid pattern fails with S0302.

Like JavaScript’s, the engine backtracks, so some patterns take exponential time on inputs they almost match: /^(a+)+$/ on thirty a followed by ! tries half a billion ways to split the run. When the evaluation has a timeout or a cancellation token, a match stops with D1012 or U1003 like any other step (ADR 0009). Set a timeout when the expressions come from users.

Functions that take a regular expression

• $contains tests whether a string matches.
• $match returns the matches, with positions and groups.
• $replace replaces matches, with references to groups.
• $split splits a string around matches.

input: address.json
Email.address[$contains($, /@my-work\./)]
=> ["fred.smith@my-work.com", "fsmith@my-work.com"]

$replace("2024-03-15", /(\d+)-(\d+)-(\d+)/, "$3/$2/$1")
=> "15/03/2024"

Matching in predicates

A regular expression is a function: applied to a string, it returns the first match, or nothing. With the chain operator ~>, that makes a compact test in a filter:

input: invoice.json
Account.Order.Product[`Product Name` ~> /hat/i].`Product Name`
=> ["Bowler Hat", "Trilby hat", "Bowler Hat"]

Matcher functions

Called directly, a regular expression returns an object describing the first match:

• match: the matched text;
• start and end: its positions;
• groups: the captured texts;
• next: a function that returns the next match, or nothing.

(
  $matcher := /[a-z]*an[a-z]*/i;
  $m := $matcher("A man, a plan, a canal, Panama!");
  [$m.match, $m.start, $m.end, $m.next().match]
)
=> ["man", 2, 5, "plan"]

$contains, $match, $replace and $split only rely on this contract. Any function that returns such objects can therefore stand in for a regular expression, for example a lambda that implements a custom matching rule.

Positions in match objects are character offsets as JavaScript counts them, in UTF-16 code units. A character outside the Basic Multilingual Plane counts as two.

Adapted from the JSONata documentation (MIT license).

Dates and times

JSON has no date type. Timestamps are usually strings in ISO 8601 format ("2018-12-10T13:45:00.000Z") or numbers of milliseconds since the Unix epoch. JSONata works with both:

The evaluation time is captured once, when the evaluation starts. Every $now() and $millis() in an expression returns that same value:

input: invoice.json
(
  $start := $now();
  $total := $sum(Account.Order.Product.(Price * Quantity));
  $now() = $start
)
=> true

Durations are differences between millisecond values:

($toMillis("2018-12-10T18:00:00Z") - $toMillis("2018-12-10T09:30:00Z")) / 3600000
=> 8.5

Picture strings

Pictures describe other formats, both for formatting ($fromMillis, $now) and for parsing ($toMillis). The syntax is that of XPath fn:format-dateTime. Text is copied as is, and components in square brackets stand for parts of the timestamp:

A presentation modifier after the component letter chooses how it is written:

Literal square brackets are written [[ and ]].

$fromMillis(1544450400000, "[D01]/[M01]/[Y0001] [H01]:[m01]")
=> "10/12/2018 14:00"

$fromMillis(1544450400000, "[FNn], [D1o] [MNn] [YI]")
=> "Monday, 10th December MMXVIII"

$fromMillis(1544450400000, "[MNn,3-3] [D], [h]:[m01] [P]")
=> "Dec 10, 2:00 pm"

$fromMillis(1544450400000, "[Y]-[M01]-[D01]T[H01]:[m01]:[s01][Z]", "+0530")
=> "2018-12-10T19:30:00+05:30"

A timezone that is not a number is an error:

differs: jsonata-js writes "NaN" in every component
$fromMillis(1544450400000, "[Y]", "CET")
=> error D3150

Parsing

$toMillis parses with the same pictures. Parts missing from the picture take default values: the start of the year, the month or the day.

$toMillis("10/12/2018", "[D]/[M]/[Y]") ~> $fromMillis("[M]/[D]/[Y]")
=> "12/10/2018"

$toMillis("Monday, 10th December 2018", "[FNn], [D1o] [MNn] [Y]")
=> 1544400000000

$toMillis("2018-12-10 14:00 +01:00", "[Y]-[M]-[D] [H]:[m] [Z]") ~> $fromMillis()
=> "2018-12-10T13:00:00.000Z"

Adapted from the JSONata documentation (MIT license).

Function reference

jlens implements every built-in function of JSONata 2.2. Each function has a signature that describes its parameters and result, for example <s-nn?:s>:

Arguments that do not match the signature raise T0410 (wrong type), T0411 (the context value has the wrong type) or T0412 (an array item has the wrong type).

String functions

Details and examples: String functions.

Numeric functions

Details and examples: Numeric functions.

Aggregation functions

Details and examples: Aggregation functions.

Boolean functions

Details and examples: Boolean functions.

Array functions

Details and examples: Array functions.

Object functions

Details and examples: Object functions.

Higher-order functions

Details and examples: Higher-order functions.

Date and time functions

Details and examples: Date and time functions.

Encoding functions

Details and examples: Encoding functions.

Other functions

Details and examples: Other functions.

String functions

Most string functions use the context value when their first argument is omitted, so they work as path steps: Account.Order.OrderID.$uppercase(). Positions and lengths count characters (Unicode code points), except where noted.

$string

$string(arg, prettify?) converts a value to a string:

• strings are returned unchanged;
• functions become the empty string;
• other values are written as JSON.

With prettify true, objects and arrays are indented. A number that is not finite is an error (D3001).

$string(5)
=> "5"

[1..3].$string()
=> ["1", "2", "3"]

$string({"a": [1, true, null]})
=> "{\"a\":[1,true,null]}"

Numbers are written with at most 15 significant digits, as in jsonata-js:

$string(0.1 + 0.2)
=> "0.3"

$length

$length(str) returns the number of characters in str.

$length("Hello World")
=> 11

$length("𝄞é")
=> 2

$substring

$substring(str, start, length?) returns the characters of str from position start (from 0), at most length of them. A negative start counts from the end.

$substring("Hello World", 3)
=> "lo World"

$substring("Hello World", 3, 5)
=> "lo Wo"

$substring("Hello World", -4, 2)
=> "or"

$substringBefore

$substringBefore(str, chars) returns the part of str before the first occurrence of chars, or all of str if chars does not occur.

$substringBefore("Hello World", " ")
=> "Hello"

$substringAfter

$substringAfter(str, chars) returns the part of str after the first occurrence of chars, or all of str if chars does not occur.

$substringAfter("Hello World", " ")
=> "World"

$uppercase

$uppercase(str) converts str to upper case.

$uppercase("Hello World")
=> "HELLO WORLD"

$lowercase

$lowercase(str) converts str to lower case.

$lowercase("Hello World")
=> "hello world"

$trim

$trim(str) normalizes whitespace in three steps:

1. Tabs, carriage returns and line feeds become spaces.
2. Runs of spaces become one space.
3. Leading and trailing spaces are removed.

$trim("   Hello  \n World  ")
=> "Hello World"

$pad

$pad(str, width, char?) pads str to at least |width| characters with char (a space by default). A positive width pads on the right, a negative one on the left.

$pad("foo", 5)
=> "foo  "

$pad("foo", -5, "#")
=> "##foo"

$formatBase(35, 2) ~> $pad(-8, "0")
=> "00100011"

$contains

$contains(str, pattern) tests whether str contains pattern, which is either a string or a regular expression.

$contains("abracadabra", "bra")
=> true

$contains("Hello World", /wo/i)
=> true

input: address.json
Phone[$contains(number, /^077/)].type
=> "mobile"

$split

$split(str, separator, limit?) splits str into an array of strings around separator, which is a string or a regular expression. An empty string separator splits str into characters. limit caps the number of parts.

$split("so many words", " ")
=> ["so", "many", "words"]

$split("so many words", " ", 2)
=> ["so", "many"]

$split("too much, punctuation. hard; to read", /[ ,.;]+/)
=> ["too", "much", "punctuation", "hard", "to", "read"]

$join

$join(strings, separator?) joins an array of strings, with separator (empty by default) between them. Every item must be a string.

$join(["a", "b", "c"])
=> "abc"

$split("too much, punctuation. hard; to read", /[ ,.;]+/, 3) ~> $join(", ")
=> "too, much, punctuation"

$match

$match(str, pattern, limit?) returns the matches of a regular expression in str. Each match is an object with three fields:

• match: the matched text;
• index: its position;
• groups: the texts captured by the parenthesized groups.

limit caps the number of matches. A single match is returned as a single object, following the sequence rules.

$match("ababbabbcc", /a(b+)/)
=> [
  {"match": "ab", "index": 0, "groups": ["b"]},
  {"match": "abb", "index": 2, "groups": ["bb"]},
  {"match": "abb", "index": 5, "groups": ["bb"]}
]

$match("ababbabbcc", /a(b+)/, 1).groups
=> ["b"]

$replace

$replace(str, pattern, replacement, limit?) replaces the occurrences of pattern (a string or a regular expression) in str. limit caps the number of replacements.

A string replacement with a regex pattern can refer to what was matched:

• $0 is the whole match;
• $1, $2, … are the captured groups (empty when there is no such group);
• $$ is a literal $.

replacement can also be a function. It receives each match, as $match describes it, and returns the replacement string.

$replace("John Smith and John Jones", "John", "Mr")
=> "Mr Smith and Mr Jones"

$replace("John Smith and John Jones", "John", "Mr", 1)
=> "Mr Smith and John Jones"

$replace("John Smith", /(\w+)\s(\w+)/, "$2, $1")
=> "Smith, John"

$replace("265USD", /([0-9]+)USD/, "$$$1")
=> "$265"

(
  $celsius := function($m) { ($number($m.groups[0]) - 32) * 5 / 9 & "C" };
  $replace("temperature = 68F today", /(\d+)F/, $celsius)
)
=> "temperature = 20C today"

Adapted from the JSONata documentation (MIT license). The functions that format numbers as strings are on the numeric functions page.

Numeric functions

Numbers are 64-bit floats. A function whose result would not be a finite number fails with D3060 ($sqrt) or D3061 ($power).

$number

$number(arg) converts a value to a number:

$number("5")
=> 5

$number("0x12")
=> 18

["1", "2.5", "1e3"].$number()
=> [1, 2.5, 1000]

$number("five")
=> error D3030

$abs

$abs(n) returns the absolute value of n.

$abs(-5)
=> 5

$floor

$floor(n) rounds n down to an integer.

[$floor(5.8), $floor(-5.3)]
=> [5, -6]

$ceil

$ceil(n) rounds n up to an integer.

[$ceil(5.3), $ceil(-5.3)]
=> [6, -5]

$round

$round(n, precision?) rounds n to precision decimal places (0 by default). A negative precision rounds to tens, hundreds, and so on. Halves round to the even neighbour (banker’s rounding), as in IEEE 754 and financial calculations.

[$round(123.456), $round(123.456, 2), $round(123.456, -1)]
=> [123, 123.46, 120]

[$round(11.5), $round(12.5), $round(125, -1)]
=> [12, 12, 120]

$power

$power(base, exponent) raises base to exponent.

[$power(2, 8), $power(2, -2), $power(2, 0.5)]
=> [256, 0.25, 1.4142135623730951]

$power(-1, 0.5)
=> error D3061

$sqrt

$sqrt(n) returns the square root of n, which must not be negative.

$sqrt(16)
=> 4

$random

$random() returns a pseudo-random number in [0, 1). It is not suitable for cryptography.

$random() < 1
=> true

$formatNumber

$formatNumber(n, picture, options?) formats a number with an XPath fn:format-number picture string:

options replaces the characters used, for example {"decimal-separator": ",", "grouping-separator": "."}. The other options are zero-digit, percent, per-mille, minus-sign, infinity, NaN, exponent-separator and pattern-separator.

$formatNumber(12345.6, "#,###.00")
=> "12,345.60"

$formatNumber(1234.5678, "00.000e0")
=> "12.346e2"

[$formatNumber(34.555, "#0.00;(#0.00)"), $formatNumber(-34.555, "#0.00;(#0.00)")]
=> ["34.56", "(34.56)"]

$formatNumber(0.14, "01%")
=> "14%"

$formatNumber(1234567.891, "#.##0,00", {"decimal-separator": ",", "grouping-separator": "."})
=> "1.234.567,89"

$formatBase

$formatBase(n, radix?) writes the integer part of n in base radix, from 2 to 36 (10 by default).

[$formatBase(100, 2), $formatBase(2555, 16)]
=> ["1100100", "9fb"]

$formatInteger

$formatInteger(n, picture) writes an integer with an XPath fn:format-integer picture:

$formatInteger(2789, "w")
=> "two thousand, seven hundred and eighty-nine"

[$formatInteger(1999, "I"), $formatInteger(28, "A"), $formatInteger(3, "w;o")]
=> ["MCMXCIX", "AB", "third"]

$formatInteger(1234567, "#,##0")
=> "1,234,567"

$parseInteger

$parseInteger(str, picture) reads an integer written as $formatInteger writes it.

$parseInteger("twelve thousand, four hundred and seventy-six", "w")
=> 12476

$parseInteger("12,345,678", "#,##0")
=> 12345678

Adapted from the JSONata documentation (MIT license).

Aggregation functions

Aggregation functions reduce an array to a single value. A single value counts as an array of one item, and an empty array gives 0 for $sum and $count, and nothing for the others. The items of $sum, $max, $min and $average must be numbers (error T0412).

$sum

$sum(array) adds the numbers of array.

$sum([5, 1, 3, 7, 4])
=> 20

input: invoice.json
$sum(Account.Order.Product.Quantity)
=> 8

$count

$count(array) returns the number of items. It accepts any values.

[$count([1, 2, 3, 1]), $count("hello"), $count([])]
=> [4, 1, 0]

input: address.json
$count(Phone[type = 'office'])
=> 2

$max

$max(array) returns the largest number.

$max([5, 1, 3, 7, 4])
=> 7

$min

$min(array) returns the smallest number.

$min([5, 1, 3, 7, 4])
=> 1

$min([])
=> undefined

$average

$average(array) returns the mean of the numbers.

$average([5, 1, 3, 7, 4])
=> 4

Adapted from the JSONata documentation (MIT license).

Boolean functions

$boolean

$boolean(arg) converts a value to a Boolean. The same rules apply wherever JSONata needs a Boolean: in predicates, with and, or, ?: and in conditions.

[$boolean("0"), $boolean(0), $boolean([0, ""]), $boolean([0, 1]), $boolean({})]
=> [true, false, false, true, false]

$not

$not(arg) converts arg to a Boolean and negates it.

[$not(true), $not(""), $not([1])]
=> [false, true, false]

input: library.json
library.books[$not("Kernighan" in authors)].title
=> ["Structure and Interpretation of Computer Programs", "Compilers: Principles, Techniques, and Tools"]

$exists

$exists(arg) returns true if arg produces a value, even null or false, and false if it produces nothing.

input: address.json
[$exists(Other.Misc), $exists(Other.Nothing), $exists(Phone[type = 'fax'])]
=> [true, false, false]

Adapted from the JSONata documentation (MIT license).

Array functions

A single value passed where an array is expected counts as an array of one item. Results that are arrays are proper arrays: they keep their structure even with one item.

$append

$append(array1, array2) returns the items of both arrays, in order. A value that is not an array is appended as one item.

$append([1, 2, 3], [4, 5, 6])
=> [1, 2, 3, 4, 5, 6]

$append("Hello", "World")
=> ["Hello", "World"]

$reverse

$reverse(array) returns the items in reverse order.

[1..5] ~> $reverse()
=> [5, 4, 3, 2, 1]

$sort

$sort(array, comparator?) returns the items in order. Without a comparator, the items must be all numbers or all strings (error D3070), and they sort in ascending order.

comparator($a, $b) returns true when $a must come after $b. The sort is stable: items the comparator considers equal keep their order.

$sort([5, 1, 3, 7, 4])
=> [1, 3, 4, 5, 7]

input: invoice.json
$sort(Account.Order.Product, function($l, $r) {
  $l.Description.Weight > $r.Description.Weight
}).`Product Name`
=> ["Trilby hat", "Bowler Hat", "Bowler Hat", "Cloak"]

For sorting by keys, the order-by stage ^(...) is shorter.

$distinct

$distinct(array) removes duplicates, keeping the first occurrence of each value. Values are compared as by =, deeply for arrays and objects.

$distinct([1, 2, 3, 3, 4, 3, 5])
=> [1, 2, 3, 4, 5]

input: invoice.json
$distinct(Account.Order.Product.Description.Colour)
=> ["Purple", "Orange", "Black"]

$shuffle

$shuffle(array) returns the items in random order.

$count($shuffle([1..9]))
=> 9

$zip

$zip(array1, array2, ...) groups the items at each position of its arguments into arrays. The result is as long as the shortest argument.

$zip([1, 2, 3], [4, 5, 6])
=> [[1, 4], [2, 5], [3, 6]]

$zip([1, 2, 3], [4, 5], [7, 8, 9])
=> [[1, 4, 7], [2, 5, 8]]

$count is described with the aggregation functions.

Adapted from the JSONata documentation (MIT license).

Object functions

Object fields keep the order in which they were inserted.

$keys

$keys(object) returns the field names of object. For an array of objects, it returns the names of all their fields, without duplicates.

input: address.json
$keys(Address)
=> ["Street", "City", "Postcode"]

input: address.json
$keys(Phone)
=> ["type", "number"]

$lookup

$lookup(object, key) returns the value of field key, for names that are not known in advance. For an array of objects, it returns the values from all of them.

input: address.json
$lookup(Address, "City")
=> "Winchester"

input: address.json
(
  $field := "number";
  $lookup(Phone[type = 'mobile'], $field)
)
=> "077 7700 1234"

$spread

$spread(object) splits an object into an array of objects with one field each. For an array of objects, it splits each of them.

input: address.json
$spread(Address)
=> [{"Street": "Hursley Park"}, {"City": "Winchester"}, {"Postcode": "SO21 2JN"}]

$merge

$merge(array) merges an array of objects into one. When several objects have the same field, the last one wins.

$merge([{"a": 1, "b": 2}, {"b": 3, "c": 4}])
=> {"a": 1, "b": 3, "c": 4}

input: address.json
$merge($spread(Address)[$keys($) != "Postcode"])
=> {"Street": "Hursley Park", "City": "Winchester"}

$each

$each(object, function) calls function($value, $key) for each field and returns the results.

input: address.json
$each(Address, function($v, $k) { $k & ": " & $v })
=> ["Street: Hursley Park", "City: Winchester", "Postcode: SO21 2JN"]

$sift, which keeps the fields that satisfy a function, is described with the higher-order functions.

Adapted from the JSONata documentation (MIT license).

Higher-order functions

These functions take a function as an argument. The function receives an item and, if it declares more parameters, the item’s position and the whole array (or, for $sift, the field name and the whole object).

$map

$map(array, function) returns the results of function($value, $index, $array) for each item. Results follow the sequence rules: one result is returned as itself, and none as nothing.

$map([1, 2, 3], function($v) { $v * 2 })
=> [2, 4, 6]

$map([1..5], $string)
=> ["1", "2", "3", "4", "5"]

input: address.json
$map(Email.address, function($v, $i, $a) { "Item " & ($i + 1) & " of " & $count($a) & ": " & $v })
=> [
  "Item 1 of 4: fred.smith@my-work.com",
  "Item 2 of 4: fsmith@my-work.com",
  "Item 3 of 4: freddy@my-social.com",
  "Item 4 of 4: frederic.smith@very-serious.com"
]

$filter

$filter(array, function) keeps the items for which function($value, $index, $array) is true.

input: invoice.json
$filter(Account.Order.Product, function($v, $i, $a) { $v.Price > $average($a.Price) }).`Product Name`
=> "Cloak"

$single

$single(array, function?) returns the only item for which function is true. More than one match is an error (D3138), and so is none (D3139). Without a function, the array must have exactly one item.

input: invoice.json
$single(Account.Order.Product, function($v) { $v.SKU = "0406654608" }).`Product Name`
=> "Bowler Hat"

input: invoice.json
$single(Account.Order.Product, function($v) { $v.Price > 30 })
=> error D3138

$reduce

$reduce(array, function, init?) folds the array into one value. It calls function($accumulator, $value, $index, $array) for each item, carrying the result to the next call. Without init, the first item is the initial accumulator.

$reduce([1..5], function($acc, $v) { $acc * $v })
=> 120

input: invoice.json
$reduce(Account.Order.Product, function($acc, $p) { $acc + $p.Quantity }, 0)
=> 8

$sift

$sift(object, function) keeps the fields of object for which function($value, $key, $object) is true. With one argument it applies to the context value.

input: invoice.json
(Account.Order.Product)[0].$sift(function($v, $k) { $k ~> /^Product/ })
=> {"Product Name": "Bowler Hat", "ProductID": 858383}

$sift({"a": 1, "b": null, "c": 0}, function($v) { $v != null })
=> {"a": 1, "c": 0}

$sort takes a comparison function too; see array functions.

Adapted from the JSONata documentation (MIT license).

Date and time functions

JSON has no date type, so timestamps are strings, usually in ISO 8601 format, or numbers of milliseconds since the Unix epoch (1970-01-01T00:00:00Z). These functions convert between the two, with picture strings for other formats. Dates and times describes the picture string syntax.

$now

$now(picture?, timezone?) returns the time the evaluation started, as an ISO 8601 string in UTC, or formatted as $fromMillis would format it. Every call in one evaluation returns the same time.

$now() = $now()
=> true

$now() ~> $contains(/^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d\.\d{3}Z$/)
=> true

$millis

$millis() returns the time the evaluation started, in milliseconds since the epoch. Like $now(), it is the same throughout one evaluation.

$millis() = $toMillis($now())
=> true

$fromMillis

$fromMillis(millis, picture?, timezone?) formats a number of milliseconds as a timestamp. It uses ISO 8601 without a picture, and an XPath fn:format-dateTime picture string otherwise. timezone, written ±HHMM, shifts the time from UTC.

$fromMillis(1510067557121)
=> "2017-11-07T15:12:37.121Z"

$fromMillis(1510067557121, "[M01]/[D01]/[Y0001] [h#1]:[m01][P]")
=> "11/07/2017 3:12pm"

$fromMillis(1510067557121, "[H01]:[m01]:[s01] [z]", "-0500")
=> "10:12:37 GMT-05:00"

$fromMillis(1510067557121, "[FNn], [D1o] [MNn] [Y]")
=> "Tuesday, 7th November 2017"

$toMillis

$toMillis(timestamp, picture?) parses a timestamp into milliseconds since the epoch. Without a picture it expects ISO 8601. With one, it parses according to the picture. A timestamp that does not match fails with D3110.

$toMillis("2017-11-07T15:07:54.972Z")
=> 1510067274972

$toMillis("10/12/2018", "[D]/[M]/[Y]") ~> $fromMillis("[Y]-[M01]-[D01]")
=> "2018-12-10"

A date and time without a time zone is read as UTC. jsonata-js reads it in the machine’s local time zone, so its results depend on where it runs; see Compatibility.

differs: jsonata-js reads the time in the local time zone
$toMillis("2018-12-10T10:00:00")
=> 1544436000000

Adapted from the JSONata documentation (MIT license).

Encoding functions

$base64encode

$base64encode(str) encodes a string in base 64. Each character is taken as one byte, so the string must be in the Latin-1 range (\u0000 to ÿ). Encode other text with $encodeUrlComponent first, if needed.

$base64encode("myuser:mypass")
=> "bXl1c2VyOm15cGFzcw=="

$base64decode

$base64decode(str) decodes base 64 into a string, reading the bytes as UTF-8.

$base64decode("bXl1c2VyOm15cGFzcw==")
=> "myuser:mypass"

$encodeUrlComponent

$encodeUrlComponent(str) percent-encodes the UTF-8 bytes of every character that is not allowed in a URL component. It behaves like JavaScript’s encodeURIComponent.

$encodeUrlComponent("?x=test")
=> "%3Fx%3Dtest"

$encodeUrl

$encodeUrl(str) percent-encodes a whole URL, keeping the characters that have a meaning in URLs (:/?#[]@!$&'()*+,;=). It behaves like JavaScript’s encodeURI.

$encodeUrl("https://mozilla.org/?x=шеллы")
=> "https://mozilla.org/?x=%D1%88%D0%B5%D0%BB%D0%BB%D1%8B"

$decodeUrlComponent

$decodeUrlComponent(str) reverses $encodeUrlComponent. Malformed escapes fail with D3140.

$decodeUrlComponent("%3Fx%3Dtest")
=> "?x=test"

$decodeUrl

$decodeUrl(str) reverses $encodeUrl.

$decodeUrl("https://mozilla.org/?x=%D1%88%D0%B5%D0%BB%D0%BB%D1%8B")
=> "https://mozilla.org/?x=шеллы"

Adapted from the JSONata documentation (MIT license).

Other functions

$type

$type(value) returns the type of value: "null", "number", "string", "boolean", "array", "object" or "function". It returns nothing for nothing.

[$type(1), $type("1"), $type(null), $type([]), $type({}), $type($type)]
=> ["number", "string", "null", "array", "object", "function"]

$type(missing)
=> undefined

$error

$error(message?) stops the evaluation with error D3137 and the given message.

bindings: {"limit": -1}
$limit >= 0 ? $limit : $error("The limit must not be negative")
=> error D3137

$assert

$assert(condition, message?) stops the evaluation with error D3141 and the message when condition is false. When it is true, it returns nothing.

input: address.json
($assert(Age >= 18, "Adults only"); FirstName)
=> "Fred"

input: address.json
($assert(Age >= 30, "Too young"); FirstName)
=> error D3141

$eval

$eval(expression, context?) parses and evaluates a JSONata expression given as a string. The context value is context if given, the current context otherwise. A syntax error in expression fails with D3120; an evaluation error, with D3121.

$eval("[1, $string(2), 3]")
=> [1, "2", 3]

input: address.json
$eval("FirstName & ' ' & Surname")
=> "Fred Smith"

$eval("a + b", {"a": 1, "b": 2})
=> 3

$clone

$clone(value) returns a deep copy of an object or array. Transforms use it to copy their input.

$clone({"a": [1, {"b": 2}]})
=> {"a": [1, {"b": 2}]}

Adapted from the JSONata documentation (MIT license).

Error reference

Every error jlens reports carries a stable code. The codes and messages match those of jsonata-js, so code that handles JSONata errors works unchanged. In a message template, {{name}} is replaced with the JSON form of the error detail name, and {{{name}}} with its raw text.

Errors expose:

• code();
• message();
• position(), a UTF-16 offset just past the offending token, as in jsonata-js;
• span(), the byte range of the offending text;
• details such as token() and value().

Static errors (S)

Raised while compiling an expression.

Type errors (T)

Raised while evaluating, when a value has the wrong type.

Dynamic errors (D)

Raised while evaluating, for other run-time problems.

Guardrail errors (U)

Raised when an evaluation exceeds its configured limits, is cancelled, or needs what jlens does not support. U2001 is raised by jlens eval --json for an input document that is not valid JSON.

Warnings (W)

jlens-specific. Warnings are reported as diagnostics by jlens check and parse_recover, and never stop evaluation.

Audit annotations

An annotation attaches business metadata to part of an expression: a label, a message and an identifier. jlens uses annotations to make audit trees and explanations readable. Annotations are written as JSONata comments, so an annotated expression is still valid JSONata and evaluates identically in any other implementation.

/** @label "Authorized customer" @message "The customer login must be on the approved list." */
customer.login in $approved
and
/** @label "Minimum total" @id "RULE-042" */
$sum(items.(unitPrice * quantity)) > 50000

Syntax

An annotation is a block comment that starts with /**. The empty comment /**/ and ordinary /* ... */ comments are not annotations.

Tag values are JSON string literals, so \", \n and é escapes work. Leading * characters on continuation lines are ignored, so multi-line comments can be written JSDoc-style:

/**
 * Customers must be registered partners.
 * @label "Authorized customer"
 * @id "IMP-001"
 */
customer.login in $approved

When several annotations precede the same expression, they are merged and later tags win.

What an annotation covers

An annotation covers the operand that follows it. It extends to the right as far as the surrounding operators allow, but never past and or or, nor past operators that bind more loosely: ? (the conditional), := and ... In practice:

Annotations never change how an expression is parsed. The syntax tree of an annotated expression is the same as the tree of the expression with its comments removed; only the annotation metadata is added to one node.

To label a whole and/or chain, wrap it in parentheses, as the rule in fixtures/shipping-rule.jsonata does:

/** @label "Order category check" */
(
  /** @label "Electronics group" */ ('electronic' in items.type and $electronics > $total * 0.3)
  or
  /** @label "Blue items group" */ ('blue' in items.color and $blueTax < $averageTax)
)

Warnings

Annotation problems are warnings: they are reported, but they never stop compilation or evaluation.

Run jlens check to see them:

$ jlens check '/** @label "Total" @severity "high" */ $sum(items.price) > 100'
warning[W0102]: Unknown annotation tag @severity; expected @label, @message or @id
 --> <expr>:1:20
  |
1 | /** @label "Total" @severity "high" */ $sum(items.price) > 100
  |                    ^^^^^^^^^
<expr>: 0 error(s), 1 warning(s)

In the API

After parsing, the annotation is available on the annotated node as Node::annotation. It also appears in the JSON form of the syntax tree (jlens ast):

{ "type": "binary", "op": "in", "annotation": { "label": "Authorized customer", "span": { … } } }

The audit guide explains how annotations appear in audit trees.

Compatibility with jsonata-js

jlens targets the behaviour of jsonata-js 2.2.2, as specified by its official test suite. It copies that behaviour, quirks included, unless copying would cost precision, correctness or performance: a result made inexact by JavaScript’s rounding, a result that is wrong by the JSONata documentation or exposes a JavaScript internal, or a slower common path. This page records both kinds of decision:

• Differences: where jlens behaves otherwise. A difference that affects the conformance suite is also an override there, with its reason.
• Extensions: features of jlens alone, which do not change the meaning of valid expressions.
• Quirks reproduced on purpose: jsonata-js behaviour that looks wrong, which jlens copies because its cost stays in unusual input or error paths, where users can expect the same answer from both engines.

Differences

Most differences have one of four causes: Rust strings are UTF-8 where JavaScript’s are UTF-16; jlens never panics, overflows its stack or exhausts memory on user input, and reports a coded error instead; an evaluation never changes its input; and a result never depends on the machine, its time zone or JavaScript’s internals.

Strings and Unicode

Two suite cases pass a lone surrogate to $encodeUrl / $encodeUrlComponent to provoke an error. jlens cannot represent that input, so it skips those two cases.

Parsing

• Error recovery (parse_recover) produces the same codes and positions as jsonata-js on its recovery tests. It also recovers from lexer errors and post-processing errors, which jsonata-js throws even in recover mode.
• An expression that starts right after a recovered error reuses that error. jsonata-js reports an extra S0211 with an undefined token in that situation.
• Doc-comments (/** ... */) are parsed as audit annotations. Like any comment, they never change how an expression is parsed or evaluated.

Evaluation

The suite runner’s own guard raises U1001 for its timelimit/depth cases; jlens enforces the same limits through Options and reports D1011/D1012, which the conformance harness accepts in place of U1001.

The input in WebAssembly

jsonata-js evaluates the JavaScript object it is given. jlens-wasm evaluate reads that object as JSON.stringify does, one object or array at a time, when the expression reaches it (ADR 0010):

Host functions

Functions provided by the application through the bindings (Bindings::function in Rust, functions in the bindings object or registerFunction in the WebAssembly package):

Extensions

These features exist only in jlens. Expressions that use them remain valid JSONata, because the extensions live in comments or in the host API, not in the language syntax.

• Audit annotations and the warnings W0101–W0103.
• Ast node ids and spans, and parse_recover diagnostics with severities.
• Static warnings (W0201–W0205) and the language service.
• Document, for evaluating many expressions against one converted document.

Quirks reproduced on purpose

Both engines behave as described here. Each row gives the reason for copying it.

Command-line interface

The jlens binary evaluates, checks, explains and inspects JSONata expressions.

jlens eval      <EXPR> | -f FILE  [-i INPUT | --no-input] [--ndjson] [--filter]
                [-b BINDINGS] [-c | -r | --json] [--timeout MS] [--max-depth N]
                [--max-string-length BYTES] [--max-array-length N] [--max-result-size BYTES]
                [--now TIME] [--seed N]
                [--explain | --audit json|text] [--audit-level decisions|full] [--trace-log FILE]
jlens check     <EXPR> | -f FILE...  [--json] [--deny-warnings] [--schema SCHEMA]
                [--config FILE] [--path FILE]
jlens test      FILE...  [--json] [--timeout MS]
                [--max-string-length BYTES] [--max-array-length N] [--max-result-size BYTES]
jlens functions [NAME] [--json]
jlens errors    [CODE] [--json]
jlens schema infer    SAMPLES...  [--each] [--overrides FILE] [-o SCHEMA]
jlens schema complete <EXPR> | -f FILE  [--schema SCHEMA] [--cursor N] [--json]
jlens ast       <EXPR> | -f FILE  [--tokens | --raw] [--recover]

Every command takes the expression either as an argument or from a file with -f. Prefer -f for anything longer than a one-liner. Shells interpret $, " and * differently, and Windows shells (cmd.exe, PowerShell) in particular make quoting expressions awkward. Glob patterns in jlens check -f, jlens test and jlens schema infer are expanded by jlens itself, so they work in every shell.

Exit codes

Writing to a closed pipe, as in jlens functions | head, ends quietly with code 0.

For scripts and agents

jlens keeps a contract with the programs that run it, recorded in ADR 0008:

• Standard output is data: results, JSON from --json, catalogs. Standard error is for people: rendered diagnostics, summaries, usage errors. Never parse standard error.
• Decide on the exit code (see the table above), not on whether anything was printed: an undefined result prints nothing and still exits with 0.
• Match errors by code (S0202, T2001, D1012…), never by message. Codes are listed in the error reference.
• The format never depends on the terminal. A command prints the same thing when an agent runs it and when a person copies it into a shell. Ask for machine output with a flag.
• Pass the expression with -f FILE and the input with -i FILE or --no-input. This avoids shell quoting. An eval that would read its input from a terminal fails with code 2 instead of waiting.
• Use eval --json to get every outcome, errors included, as one line of JSON on standard output (see below), and check --json for the diagnostics of every file as one JSON array.
• Test the expressions you write with jlens test and a file of cases, and read the failures from test --json (see below).
• Fix the clock and the seed with --now and --seed when a result depends on $now(), $millis(), $random() or $shuffle(), so that the same command prints the same bytes when it is run again.

jlens eval

Evaluates the expression against a JSON document and prints the result. The document comes from -i FILE, or from standard input. An undefined result prints nothing. When standard input is a terminal and neither -i nor --no-input is given, eval fails with exit code 2 instead of waiting for a document.

$ jlens eval -f fixtures/shipping-rule.jsonata -i fixtures/online-store-order.json
{
  "action": "FREE_SHIPPING",
  "code": "OK_200"
}

Results are printed the way JavaScript prints JSON, so they match jsonata-js output: 2^64 prints as 18446744073709552000 and 1e21 as 1e+21.

Machine output: --json

With --json, every outcome is one compact JSON object, the envelope, on one line of standard output. Nothing needs to be read from standard error.

$ jlens eval --json -i fixtures/online-store-order.json 'customer.name + 1'
{"ok":false,"error":{"code":"T2001","message":"The left side of the \"+\" operator must evaluate to a number","position":15,"line":1,"column":1}}

• Decide on ok, and match errors by code; message is for people and may change.
• line and column (1-based, the column in characters) mark the start of the offending text in the expression, as the human report does. position is the jsonata-js position, a UTF-16 offset usually just past the offending token. The three are absent when the error has no location.
• An input document that is not valid JSON gives an envelope with code U2001, and exit code 2 as without --json.
• With --filter, a matching document is the result, and a document that does not match prints nothing.
• With --ndjson, every envelope also carries line, the number of its input line, so that results and errors can be paired with their input. A line that is not valid JSON gives a U2001 envelope, and processing continues.
• --json cannot be combined with -r, --explain or --audit.
• Fields are only ever added to the envelope: ignore the ones you do not know.

Many documents: --ndjson

With --ndjson, the input has one JSON document per line, and the expression is evaluated for each one. Results are printed one per line, as they are produced, so arbitrarily large inputs stream through in constant memory.

• Blank lines are skipped.
• Lines that are not valid JSON, and evaluations that fail, are reported on standard error with their line number. Processing continues, and the exit code becomes 1.
• With --json, each result and each error is an envelope that carries its line number, and nothing is written to standard error.

$ jlens eval --ndjson -i fixtures/orders.ndjson '$sum(items.(unitPrice * quantity))'
50000
300
60000
line 5: invalid JSON: expected ident at line 1 column 2

$ jlens eval --ndjson --json -i fixtures/orders.ndjson '$sum(items.(unitPrice * quantity))'
{"ok":true,"line":1,"result":50000}
{"ok":true,"line":2,"result":300}
{"ok":true,"line":4,"result":60000}
{"ok":false,"line":5,"error":{"code":"U2001","message":"The input is not valid JSON: expected ident at line 1 column 2"}}
{"ok":true,"line":6,"undefined":true}

Selecting documents: --filter

With --filter, jlens prints the input documents for which the expression is truthy, instead of the results. Truthiness follows $boolean: false, 0, "", null, empty arrays, empty objects and undefined are false. Combined with --ndjson, this works like a grep for JSON:

$ jlens eval --ndjson --filter -i fixtures/orders.ndjson \
    "customer.login in ['alice.walker99', 'bob.harrison42'] and \$sum(items.(unitPrice * quantity)) > 50000"
{"number":"SO-3","customer":{"login":"bob.harrison42"},"items":[{"type":"toy","unitPrice":60000,"quantity":1}]}

Explaining results

$ jlens eval --no-input --explain '/** @label "Positive" */ 3 > 0 and /** @label "Small" */ 3 < 2'
result: false
✘ Small — 3 < 2 → false (3 < 2)

These options apply to a single document; they cannot be combined with --ndjson or --filter.

jlens check

Parses the expression with error recovery and reports every problem it finds. It warns about calls of functions that are neither built in nor bound in the expression (W0202, with a suggestion such as $summ → $sum), and about calls with a number of arguments the function’s signature does not accept (W0203) or an argument whose type it does not accept (W0205, as in $length(1234)). With --schema, it also reports field names that the schema does not have (see the schema guide), and checks the types of fields passed to functions. Each problem is shown in context:

$ jlens check 'Account.Order[0;].Product'
error[S0202]: Expected "]", got ";"
 --> <expr>:1:16
  |
1 | Account.Order[0;].Product
  |                ^
<expr>: 1 error(s), 0 warning(s)

-f takes several files, and glob patterns such as rules/*.jsonata. Each file is reported in turn, followed by a total. Errors make the exit code 1; warnings alone do not, unless --deny-warnings is given. An unreadable file, or a pattern that matches nothing, ends with exit code 2 before anything is checked.

With --json, the diagnostics of all the files are printed as one compact JSON array on standard output, which suits editors, CI scripts and agents. Each diagnostic has file (absent for an expression given as an argument), the 1-based line and column (counted in characters), then the severity, code, message and span, in byte offsets into the file:

$ jlens check --json 'a +'
[{"line":1,"column":4,"severity":"error","code":"S0207","message":"Unexpected end of expression","span":{"start":3,"end":3}}]

Configuration: jlens.json

check reads the same jlens.json as the language server, so that the command line and the editors report the same problems. It maps file patterns to a schema, the variables and the host functions the expressions are evaluated with: calls of declared host functions are checked like those of built-in ones, and, once variables are declared, references to undeclared ones get W0204. By default, check reads the jlens.json of the current directory, if there is one; --config names another file, whose patterns and paths are relative to its folder.

The configuration applies to files given with -f, by their path. An expression given as an argument matches no pattern, unless --path gives it the path of a file it would be saved as, which need not exist; that suits checking an expression before writing it:

jlens check --path rules/discount.jsonata '$rate(type) * unitPrice'

--schema replaces the schema that the configuration gives a file, and keeps its variables and functions. A configuration that cannot be read, or whose schema cannot be, ends with exit code 2.

To validate a directory of rules in CI, warnings included:

jlens check --deny-warnings -f 'rules/*.jsonata'

jlens test

Runs files of test cases. A case file is a JSON object that gives one expression, inline with expr or as the path of a file with file (relative to the case file), and the cases it is evaluated with:

{
  "file": "../rules/discount.jsonata",
  "bindings": {"discount": 0.1},
  "cases": [
    {"name": "ten percent off", "input": {"items": [{"price": 10}, {"price": 30}]}, "result": 36},
    {"name": "no items", "input": {}, "undefined": true},
    {"name": "text prices", "input": {"items": [{"price": "10"}]}, "error": "T0412"}
  ]
}

Each case has an optional name, the input document (absent for none; null is a document), its own bindings, which are added to those of the file and win over them, and exactly one expectation:

The file may also fix the clock with now (milliseconds or an ISO 8601 timestamp, as eval --now) and the random numbers with seed. Unknown keys are errors, so that a misspelt expectation does not pass unnoticed.

jlens test takes several files, and glob patterns such as tests/*.json. It reports each failed case, with what was expected and what the expression gave, then a summary per file. Had the third case above expected T2001:

$ jlens test tests/discount.json
FAIL tests/discount.json case 3: text prices
  expected: error T2001
  actual:   error T0412: Argument 1 of function "sum" must be an array of "numbers"
tests/discount.json: 2 passed, 1 failed

A failed case makes the exit code 1. A case file that cannot be read, or is invalid, ends with exit code 2 before any case runs. --timeout stops each evaluation after a number of milliseconds (error D1012), and --max-string-length, --max-array-length and --max-result-size cap the size of what it builds, as they do for eval (error U1001).

With --json, the outcome is one compact JSON object on standard output, with the number of cases that passed and failed, and the failures. Each failure has the file, the 1-based case number, the name if there is one, and what was expected and what is actual, both in the shape of the eval --json envelope:

$ jlens test --json tests/discount.json
{"passed":2,"failed":1,"failures":[{"file":"tests/discount.json","case":3,"name":"text prices","expected":{"ok":false,"error":{"code":"T2001"}},"actual":{"ok":false,"error":{"code":"T0412","message":"Argument 1 of function \"sum\" must be an array of \"numbers\"","position":5,"line":1,"column":1}}}]}

jlens functions

Lists the built-in functions by category, with their parameters and a summary. Given a name (with or without $), it describes one function; --json prints the catalog as JSON.

$ jlens functions formatNumber
$formatNumber($value, $picture, $options)
  signature: <n-so?:s>
  category:  Numeric
  Formats a number with an XPath picture string

The function reference is generated from the same catalog.

jlens errors

Lists the error and warning codes by category, with their message templates. Given a code (in any case), it describes one; --json prints the catalog as JSON, each entry with its code, category (static, type, dynamic, guardrail or warning) and template.

$ jlens errors S0202
S0202
  category: Static errors (S)
  message:  Expected {{value}}, got {{token}}

The error reference is generated from the same catalog.

jlens schema

Both are described in the schema guide.

jlens lsp

Runs the language server on standard input and output. Editors start it; it is configured by the jlens.json of each workspace folder. See the editors guide.

jlens ast

Prints the processed syntax tree as JSON. This is the tree the evaluator walks, with node ids, spans, stages and annotations. Flags show earlier stages of the pipeline:

jlens ast is mostly useful for understanding how an expression is read, for example which part an annotation covers. The parsing chapter explains the tree.

Rust crate

jlens-core is the JSONata engine. jlens-schema adds schema inference and the language service; see the schema guide. This chapter covers jlens-core. Its API reference is on docs.rs.

[dependencies]
jlens-core = "0.2"
serde_json = "1"

The snippets in this chapter come from crates/jlens-core/tests/guide.rs, which runs them as tests.

Compile once, evaluate many times

    use jlens_core::{Bindings, Expression};
    use serde_json::json;

    let expr = Expression::compile("$sum(items[type = $kind].(price * qty))")?;
    let order = json!({"items": [
        {"type": "book", "price": 12.5, "qty": 2},
        {"type": "pen", "price": 1.5, "qty": 10}
    ]});
    let bindings = Bindings::new().var("kind", "book");
    let total = expr.evaluate(Some(&order), &bindings)?;
    assert_eq!(total, Some(json!(25)));

Expression::compile parses the expression and prepares it for evaluation. A compiled expression is immutable and cheap to clone. Keep it, and evaluate it as often as needed, from any thread (see Threads).

Documents and results are serde_json::Values. Object fields keep their order (jlens enables serde_json’s preserve_order). Variables come from Bindings: $kind above. Bindings can also be collected from (String, Value) pairs, such as the fields of a JSON object.

Undefined

JSONata distinguishes nothing (an empty sequence) from null. In the API, nothing is None, both for the input and for the result:

    let expr = Expression::compile("customer.email")?;
    // No input document: `None`. A result of `None`: the expression produced nothing.
    assert_eq!(expr.evaluate(None, &Bindings::new())?, None);
    assert_eq!(
        expr.evaluate(Some(&json!({"customer": {}})), &Bindings::new())?,
        None
    );

Guardrails

Expressions can recurse and loop. Bound their cost with Options, especially for expressions that users write:

    use jlens_core::Options;
    use std::time::Duration;

    let options = Options::new()
        .timeout(Duration::from_millis(100)) // D1012 when exceeded
        .max_depth(500); // D1011 when exceeded
    let runaway = Expression::compile("($f := function($n) { $f($n + 1) }; $f(0))")?;
    let err = runaway
        .evaluate_with(None, &Bindings::new(), &options)
        .unwrap_err();
    assert_eq!(err.code(), "D1012");

The default depth limit is 50,000 levels natively; the stack grows as needed. There is no default timeout. Expressions nested more than 1,000 levels deep (counting each operator of a chain such as 1 + 2 + 3) are rejected when compiled (U1001). A string longer than Options::max_string_length (256 MiB), an array longer than Options::max_array_length (16,777,216 items) or a result larger than Options::max_result_size (256 MiB of JSON text) stops the evaluation with U1001 instead of exhausting memory. See Guardrails.

Cancellation

A CancelToken stops an evaluation from another thread, for example when the request that started it goes away. Clones of a token share one flag:

    use jlens_core::CancelToken;
    use std::{thread, time::Duration};

    let token = CancelToken::new();
    let options = Options::new().cancel_token(token.clone());
    let forever = Expression::compile("($loop := function() { $loop() }; $loop())")?;
    let worker = thread::spawn(move || forever.evaluate_with(None, &Bindings::new(), &options));
    thread::sleep(Duration::from_millis(50));
    token.cancel(); // from any thread that holds a clone
    let err = worker.join().unwrap().unwrap_err();
    assert_eq!(err.code(), "U1003");

The evaluation checks the token between steps, as it checks the timeout, and fails with U1003. A host function that is running finishes first. A cancelled token stays cancelled, so create one per job.

Reproducible results

$now() and $millis() read the system clock once per evaluation, and $random() and $shuffle() draw from a generator seeded from it. Fix both in Options to get the same result from the same expression, input and options, for tests or for replaying a decision:

    let options = Options::new()
        .now(1_700_000_000_000) // $now(), $millis()
        .seed(42); // $random(), $shuffle()
    let expr = Expression::compile("{'at': $now(), 'order': $shuffle([1, 2, 3, 4, 5])}")?;
    let bindings = Bindings::new();
    let first = expr.evaluate_with(None, &bindings, &options)?.unwrap();
    assert_eq!(first["at"], json!("2023-11-14T22:13:20.000Z"));
    // Same expression, input and options: the same result.
    assert_eq!(expr.evaluate_with(None, &bindings, &options)?, Some(first));

A seed reproduces results with the same version of jlens, on any platform; it does not reproduce the numbers jsonata-js draws from Math.random().

Errors

Every error has a stable code: S0xxx for syntax errors, T for type errors, D for dynamic errors, and U1001 for limits. Match on the code, not on the message:

    let err = Expression::compile("Account.Order[0;]").unwrap_err();
    assert_eq!(err.code(), "S0202");
    assert_eq!(err.message(), r#"Expected "]", got ";""#);
    assert_eq!(err.position(), Some(16)); // UTF-16 offset, as in jsonata-js
    let span = err.span().unwrap(); // byte range of the offending token
    assert_eq!((span.start, span.end), (15, 16));
    assert_eq!(err.token().as_deref(), Some(";"));

    let expr = Expression::compile("name + 1").unwrap();
    let err = expr
        .evaluate(Some(&json!({"name": "x"})), &Bindings::new())
        .unwrap_err();
    assert_eq!(err.code(), "T2001");
    assert_eq!(err.value(), Some(&json!("x")));
    // `Display` gives the code, the message and the position.
    assert_eq!(
        err.to_string(),
        r#"T2001: The left side of the "+" operator must evaluate to a number (at position 6)"#
    );

position matches jsonata-js: a UTF-16 offset, usually just after the offending token. span is a byte range, for slicing the source in Rust. Errors serialize to JSON with their code, message and details.

Host functions

The application can provide its own functions, called from expressions like built-in ones. Bind a HostFunction with Bindings::function:

    use jlens_core::{Error, HostFunction};

    // `$shipping(weight, region?)`: the region defaults to the context value's `region`.
    let shipping = HostFunction::with_signature("<ns?:n>", |args, call| {
        let weight = args[0].as_ref().and_then(|w| w.as_f64()).unwrap_or(0.0);
        let region = match &args[1] {
            Some(region) => region.clone(),
            None => call
                .input()
                .map_or(json!(null), |order| order["region"].clone()),
        };
        let rate = match region.as_str() {
            Some("north") => 1.5,
            Some("south") => 2.0,
            _ => return Err(Error::new("SHIP01").with("message", format!("No rate for {region}"))),
        };
        Ok(Some(json!(weight * rate)))
    })?;
    let bindings = Bindings::new().function("shipping", shipping);

    let expr = Expression::compile("$shipping($sum(items.weight))")?;
    let order = json!({"region": "south", "items": [{"weight": 2}, {"weight": 1.5}]});
    assert_eq!(expr.evaluate(Some(&order), &bindings)?, Some(json!(7)));

    let err = Expression::compile("$shipping(3, 'west')")?
        .evaluate(None, &bindings)
        .unwrap_err();
    assert_eq!(err.code(), "SHIP01");
    assert_eq!(err.message(), r#"No rate for "west""#);
    assert_eq!(err.position(), Some(10)); // the call, as for built-in functions

• Arguments and results are JSON. None stands for undefined. A function passed as an argument (such as a lambda) arrives as None; HostCall::function gives a handle to call it (below).
• Signatures use the notation of the built-in functions. HostFunction::with_signature checks the arguments before each call, with the same errors (T0410, T0412). A parameter with - takes the context value when the argument is missing. HostFunction::new(arity, ...) skips the check and passes the arguments as given.
• The arity tells higher-order functions how many arguments to pass: $map passes the index only to functions with two parameters or more. Partial application ($shipping(?, 'north')) uses it too.
• The context value is available through HostCall::input, which converts it to JSON only when called.
• Errors returned by the function stop the evaluation. Give them a code of your own and a message detail. Like other errors, they get the position and token of the call.
• Bindings take precedence over built-in functions of the same name, as in jsonata-js. A name holds a value or a function: binding one replaces the other.

Host functions are Send + Sync, so bindings that hold them can be shared between threads. A host function runs to completion: a timeout is checked again only once it returns, or when it calls a function it received.

Functions as arguments

A host function can call a function it receives, such as a fallback or a step to retry. HostCall::function(i) gives a Callable for argument i when it is a function: a lambda, a built-in function, a partial application, a chain or another host function.

    // `$rate(region, fallback)`: the rate from the application's table, or `fallback(region)`.
    let rate = HostFunction::with_signature("<sf:n>", |args, call| {
        match args[0].as_ref().and_then(|r| r.as_str()) {
            Some("north") => Ok(Some(json!(1.5))),
            Some("south") => Ok(Some(json!(2))),
            _ => {
                let fallback = call.function(1).expect("the signature requires a function");
                fallback.call(&[args[0].clone()])
            }
        }
    })?;
    let bindings = Bindings::new().function("rate", rate);

    let expr = Expression::compile("$rate(region, function($r) { $length($r) / 10 })")?;
    let order = json!({"region": "west"});
    assert_eq!(expr.evaluate(Some(&order), &bindings)?, Some(json!(0.4)));

    // An error in the fallback comes back to the host function, which returns it here.
    let err = Expression::compile("$rate('west', function($r) { $error('No rate for ' & $r) })")?
        .evaluate(None, &bindings)
        .unwrap_err();
    assert_eq!(err.code(), "D3137");
    assert_eq!(err.message(), "No rate for west");

• The call is part of the evaluation, as when $map calls its function. The context value is null, and a lambda keeps the variables of its definition. The call counts as a level of depth, so a recursion through a host function ends with D1011; it checks the timeout and the cancellation token; its result is limited by max_result_size. An audit shows its steps under the host function’s call.
• Arguments and the result are JSON, as for the host function itself. A sequence comes back as an array or a single value; a function crosses as None, in both directions.
• Errors come back as Err, with their codes and positions. The host function can handle them, or return them, which ends the evaluation.
• The handle borrows the call: it cannot be kept and called after the host function returns.

Many expressions, one document

Each evaluate call converts the parts of its input document that the expression reads into jlens’s internal representation, one level at a time: a path costs the width of the objects it crosses. An expression that reads most of a document of a few dozen kilobytes (a filter over every item, say) spends more on that conversion than on the rule. To evaluate several rules against the same document, convert it once into a Document:

    use jlens_core::Document;

    let rules = [
        ("minimum total", "$sum(items.(price * qty)) >= 20"),
        ("at most 50 items", "$sum(items.qty) <= 50"),
        ("no pens", "$not('pen' in items.type)"),
    ];
    let compiled: Vec<_> = rules
        .iter()
        .map(|(name, source)| Ok((*name, Expression::compile(source)?)))
        .collect::<Result<_>>()?;

    let order = Document::parse(
        r#"{"items": [{"type": "book", "price": 12.5, "qty": 2},
                      {"type": "pen", "price": 1.5, "qty": 10}]}"#,
    )?;
    let failed: Vec<&str> = compiled
        .iter()
        .filter(|(_, rule)| {
            let outcome = rule.evaluate_document(&order, &Bindings::new(), &Options::default());
            outcome.ok().flatten() != Some(json!(true))
        })
        .map(|(name, _)| *name)
        .collect();
    assert_eq!(failed, ["no pens"]);

Document::parse reads JSON text straight into the internal representation, and Document::new converts a serde_json::Value. A Document is not Send: use it on the thread that created it. Performance has measurements.

Other input representations

An input held in another representation than serde_json (another JSON library, a database row, a host language’s objects) can be read the same way, one level at a time, without being converted first. Implement input::Source for it, and evaluate with Expression::evaluate_source. The documentation of the input module has an example. A level that the source fails to read stops the evaluation with U1004. The WebAssembly package reads JavaScript objects this way.

Explaining results

evaluate_audited records an audit tree of the evaluation: comparisons and their operands, and/or short circuits, conditionals, bindings, filters, and the labels of annotations. explain summarizes it:

    use jlens_core::audit::{AuditOptions, explain};

    let rule = Expression::compile(
        r#"/** @label "Big order" @message "Orders must total at least 100." */
           $sum(items.(price * qty)) >= 100"#,
    )?;
    let order = json!({"items": [{"price": 12.5, "qty": 2}]});
    let (result, audit) = rule.evaluate_audited(
        Some(&order),
        &Bindings::new(),
        &Options::default(),
        &AuditOptions::decisions(),
    );
    assert_eq!(result?, Some(json!(false)));
    let explanation = explain(&audit);
    assert_eq!(explanation.result, "false");
    assert_eq!(
        explanation.lines[0].text,
        "Big order — $sum(items.(price * qty)) >= 100 → false (25 >= 100)"
    );
    // The whole tree serializes to JSON, to store it with the decision.
    let stored = serde_json::to_string(&audit)?;
    assert!(stored.contains(r#""label":"Big order""#));

The audit guide describes the tree, the levels and the bounds that keep audits of large documents small.

Custom tracers

Audits are built by a tracer, an implementation of the Tracer trait that receives evaluation events. Every method has a default that does nothing, so a tracer implements only what it needs:

    use jlens_core::Node;
    use jlens_core::audit::{Tracer, ValueRef};

    /// Counts the comparisons an evaluation makes.
    #[derive(Default)]
    struct Comparisons(usize);

    impl Tracer for Comparisons {
        fn operands(&mut self, _node: &Node, _lhs: ValueRef<'_>, _rhs: ValueRef<'_>) {
            self.0 += 1;
        }
    }

    let expr = Expression::compile("items[price > 10 and qty < 5]")?;
    let mut counter = Comparisons::default();
    expr.evaluate_traced(
        Some(&json!({"items": [{"price": 12, "qty": 1}, {"price": 5, "qty": 9}]})),
        &Bindings::new(),
        &Options::default(),
        &mut counter,
    )?;
    assert_eq!(counter.0, 3); // the second item fails `price > 10`, so `qty < 5` is skipped

jlens provides three tracers:

• AuditCollector builds audit trees;
• JsonlTracer writes every event as a line of JSON;
• TracingTracer, with the tracing feature, emits tracing events.

Without a tracer, the evaluator runs its fast path, so tracing costs nothing when unused.

Tools for editors

parse_recover keeps going after syntax errors. The syntax tree is public, with node ids, spans and annotations:

    use jlens_core::{NodeKind, Severity, parse_recover};

    // Recovering parsing never fails: it returns a partial tree and every problem found.
    let ast = parse_recover("Account.Order[0;].Product");
    let errors: Vec<_> = ast
        .diagnostics
        .iter()
        .filter(|d| d.severity == Severity::Error)
        .map(|d| (d.code.as_str(), d.span.start))
        .collect();
    assert_eq!(errors, [("S0202", 15)]);

    // Walk the tree, for example to list the variables an expression uses (functions are
    // variables too).
    let ast = jlens_core::parse("$a + $b * $sum(items)")?;
    let mut variables = Vec::new();
    ast.root.walk(&mut |node| {
        if let NodeKind::Variable { name } = &node.kind {
            variables.push(name.clone());
        }
    });
    assert_eq!(variables, ["a", "b", "sum"]);

The function catalog describes every built-in function, for completion lists and documentation:

    let catalog = jlens_core::function_catalog();
    let substring = catalog.iter().find(|f| f.name == "substring").unwrap();
    assert_eq!(substring.signature, "<s-nn?:s>");
    assert_eq!(substring.params, ["str", "start", "length"]);

For completion, diagnostics and hover against a schema, use jlens_schema::LanguageService; see the schema guide.

Threads

Expression is Send + Sync. Share one compiled expression between threads, for example behind an Arc:

    use std::sync::Arc;
    use std::thread;

    let expr = Arc::new(Expression::compile("$uppercase(name)")?);
    let workers: Vec<_> = ["ann", "bob"]
        .into_iter()
        .map(|name| {
            let expr = Arc::clone(&expr);
            thread::spawn(move || expr.evaluate(Some(&json!({"name": name})), &Bindings::new()))
        })
        .collect();
    for worker in workers {
        assert!(worker.join().unwrap()?.is_some());
    }

Each evaluation is independent and single-threaded. Values created during an evaluation never leave it, except as the serde_json::Value result.

Features

The minimum supported Rust version is 1.88. jlens-core compiles to wasm32-unknown-unknown; the WebAssembly guide covers the browser package built on it.

WebAssembly (browser)

jlens-wasm compiles jlens to WebAssembly as an ES module for browsers. It covers what the Rust crates do:

• evaluation;
• audit trees and explanations;
• schema inference;
• the language service (completion, diagnostics, hover, signature help) for editors.

It also runs in Node.js 20+ and Deno, provided init is given the .wasm bytes.

import init, { Expression } from "jlens-wasm";

await init();

const expr = new Expression("$sum(items[type = $kind].(price * qty))");
expr.evaluate({ items: [{ type: "a", price: 2.5, qty: 4 }] }, { kind: "a" }); // 10

Building the package

rustup target add wasm32-unknown-unknown
cargo install wasm-bindgen-cli --version <the wasm-bindgen version in Cargo.lock> --locked
cargo xtask wasm            # release build into crates/jlens-wasm/pkg
cargo xtask wasm --dev      # unoptimized, with Rust panics reported on the console

cargo xtask wasm does four things:

1. It compiles the crate with the wasm-release profile, which optimizes for size and speed (opt-level = "s").
2. It generates the JavaScript glue and the TypeScript declarations with wasm-bindgen --target web.
3. It runs wasm-opt -Os from binaryen: the installed one, or the npm package through npx. Without either, the module is about 45% larger.
4. It writes package.json.

The wasm-bindgen CLI must be the exact version in Cargo.lock, and the task checks this.

A size budget for the gzipped module is enforced in xtask/src/wasm.rs.

Loading the module

Call init() once, and await it, before using anything else. By default it fetches jlens_bg.wasm from next to jlens.js:

<script type="module">
  import init, { Expression } from "./pkg/jlens.js";
  await init();
</script>

Bundlers that understand new URL(..., import.meta.url) handle this as is. Vite, webpack 5, Rollup and esbuild all do (esbuild with a loader for .wasm). Otherwise, pass the module’s URL or its bytes:

// Vite: get the URL of the asset.
import wasmUrl from "jlens-wasm/jlens_bg.wasm?url";
await init({ module_or_path: wasmUrl });

// Node.js or Deno: pass the bytes.
import { readFile } from "node:fs/promises";
await init({ module_or_path: await readFile(new URL("./pkg/jlens_bg.wasm", import.meta.url)) });

initSync({ module }) instantiates synchronously from bytes or a WebAssembly.Module. This is useful in workers.

Evaluating expressions

const expr = new Expression(source);         // compile once (throws JlensError on syntax errors)
expr.evaluate(input, bindings, options);     // evaluate many times
expr.evaluateJson(jsonText, bindings);       // JSON text in, JSON text out
expr.free();                                 // release it (or let the garbage collector do so)

• input can be any JSON-compatible value. undefined, or no argument, means there is no input document, so $ is undefined. Values are read as JSON.stringify reads them. As a result, Date objects become strings, and functions and undefined fields are dropped. See how the input is read.
• bindings is an object of variables and functions: {limit: 10} makes $limit available, and a function becomes a host function.
• options sets the limits (timeout, maxDepth, maxStringLength, maxArrayLength, maxResultSize), and the clock and seed that make an evaluation reproducible: now fixes $now() and $millis() (milliseconds since the epoch, a Date or an ISO 8601 timestamp), and seed fixes $random() and $shuffle().
• The result is a plain value, or undefined when the expression evaluates to undefined. Numbers and object key order are the same as in jsonata-js.

evaluateJson takes the input as JSON text and returns the result as JSON text. For large documents that you already have as text, such as a fetch response body, it avoids converting the document to JavaScript objects and back.

How the input is read

evaluate reads its input during the evaluation, one object or array at a time, when the expression first reaches it (ADR 0010). A path such as customer.name reads two small objects, however large the rest of the document is. An object or array of more than 16 values is read whole, with everything it holds, through JSON.stringify, which is faster than reading its values one by one.

Two consequences follow:

• Errors appear when their part is read. A cycle, a BigInt, or a toJSON method that throws makes JSON.stringify throw. evaluate throws the same exception, but only when the expression reads the object or array that holds it: an expression that never reaches it succeeds.
• The input must not change during the evaluation. A host function that modifies the input object can change what the rest of the evaluation reads. Copy the input first if a function must modify it.

evaluateJson, evaluateWithAudit and Document convert their whole input first.

Reusing a document

evaluate reads the parts of the input that the expression reaches, on every call. For an expression that reads most of a document of a few dozen kilobytes, that costs hundreds of microseconds, often more than the evaluation itself. To evaluate several expressions against one document (a set of rules, say), convert it once:

import { Document } from "jlens-wasm";

const doc = new Document(input);              // or Document.fromJson(text)
const results = rules.map((rule) => rule.evaluateDocument(doc, bindings));
doc.free();

Evaluations never modify a document. Performance has measurements.

With explicit resource management, using expr = new Expression(...) frees the expression at the end of the block.

Host functions

JavaScript functions, in the bindings or registered on an expression, are called from the expression like built-in functions:

const expr = new Expression("$shipping($sum(items.weight), region)");
expr.evaluate(order, { shipping: (weight, region) => weight * rates[region] });

// As in jsonata-js: available to every evaluation, with an optional signature.
expr.registerFunction("shipping", (weight, region) => weight * rates[region], "<ns:n>");

• Arguments and the result pass through JSON, like the input. undefined stands for undefined, and a function argument arrives as a JavaScript function that calls it (below).
• A signature (registerFunction’s third argument) checks the arguments before each call, with the errors of the built-in functions (T0410, T0412). A parameter with - receives the context value when the argument is missing: <n-:n> makes price.$double() work.
• this is undefined. Use a signature with - to receive the context value.
• The function’s length tells higher-order functions how many arguments to pass: $map passes the index only to functions with two parameters or more.
• An exception thrown by the function is rethrown by evaluate as is. Its position and token are set to the call when it has none. evaluateWithAudit reports it with its code, or its name, and its message.
• Asynchronous functions are not supported. A function that returns a promise fails with U1002. Fetch what the expression needs before evaluating it.
• Precedence: a function in the bindings replaces a registered function of the same name, and both replace a built-in function.

The Rust crate guide describes the same model.

Functions as arguments

A host function can call a JSONata function it receives, such as a fallback or a step to retry: a lambda, a built-in function, a partial application, a chain or another host function.

const retry = (step, times) => {
  for (let i = 1; ; i++) {
    try {
      return step(i);
    } catch (e) {
      if (i >= times) throw e;
    }
  }
};
new Expression("$retry(function($i) { $lookup(sku, $i) }, 3)").evaluate(order, { retry, lookup });

• The call is part of the evaluation, as when $map calls its function. The context value is null, and a lambda keeps the variables of its definition. The call counts as a level of depth, so a recursion through a host function ends with D1011; it checks the timeout; its result is limited by maxResultSize.
• It is synchronous: it returns the result, where jsonata-js returns a promise. Code written for jsonata-js with await step(i) keeps working.
• Arguments and the result pass through JSON. A sequence comes back as an array or a single value; a JavaScript function among the arguments becomes undefined, and a function result comes back as undefined.
• Its length is 0 and its arity property is the number of parameters, as in jsonata-js.
• Errors are thrown as a JlensError with their codes, or as the exception a host function threw. The host function can catch them, or let them through, which ends the evaluation with that error.
• It is valid until the host function returns. Called later, from a promise or a timer, it throws U1005.

Errors

Errors are Error objects named JlensError. They carry the JSONata error fields; every offset is in UTF-16 code units, like JavaScript string indexes:

try {
  new Expression("Account.Order[0;]");
} catch (err) {
  if (err.name !== "JlensError") throw err;
  err.code;      // "S0202"
  err.message;   // 'Expected "]", got ";"'
  err.position;  // 16: where it was detected, as in jsonata-js
  err.start;     // 15: the offending source range
  err.end;       // 16
  err.token;     // ";"
}

Evaluation errors carry value too, when there is one (for T2001, the value that is not a number). Invalid arguments, such as bindings that are not an object or malformed JSON for evaluateJson, throw a TypeError instead. The error reference lists every code.

Explaining results

evaluateWithAudit evaluates and records how the result was reached; see the audit guide. Evaluation errors are reported in the result, not thrown:

const expr = new Expression('/** @label "Big order" */ $sum(items.(price * qty)) > $limit');
const { result, error, audit, explanation, explanationText } = expr.evaluateWithAudit(
  { items: [{ price: 2.5, qty: 4 }, { price: 1, qty: 3 }] },
  { limit: 20 },
  { level: "decisions", limits: { maxNodes: 1000 } },
);
result;           // false
explanationText;  // 'result: false\n✘ Big order — $sum(items.(price * qty)) > $limit → false (13 > 20)'
audit.root;       // { kind: "binary", op: ">", label: "Big order", operands: [13, 20], outcome: "false", ... }

The options are those of evaluate, plus two more:

• level is "off", "decisions" (the default) or "full".
• limits sets the capture limits: maxItems, maxString, maxDepth, maxRejected, maxNodes and maxSource.

explanation.lines holds structured lines (depth, passed, text, message, node) for rendering in a user interface. Audit node spans are UTF-16 offsets, so a viewer can highlight a node’s source in the editor. The web editor example does this.

Tooling

parseRecover(source) never throws. It returns the partial syntax tree and every diagnostic:

parseRecover("Account.Order[0;].");
// { ast: {...}, diagnostics: [{ severity: "error", code: "S0202", message: ..., span: {start: 15, end: 16} }] }

expr.ast() returns the processed tree of a compiled expression, and functions() returns the catalog of built-in functions. Spans are UTF-16 offsets throughout.

Schemas

inferSchema(samples, overrides?) infers a JSON Schema 2020-12 from sample documents. The schema guide describes what it records and the overrides format:

const schema = inferSchema([doc1, doc2], {
  "items[].type": { description: "Product category", enum: ["apparel", "electronic", "toy"] },
  "items[].internalCode": { hidden: true },
});

The result is an ordinary JSON Schema object. Store it, edit it by hand, or pass it to LanguageService.

The language service

const service = new LanguageService(schema, { limit: { type: "number" } });

const text = "items[type = 'a'].p";
service.complete(text, text.length);
// { range: {start: 18, end: 19},
//   items: [{ label: "price", kind: "field", detail: "integer", documentation: "Examples: `1`", insertText: "price" }] }

service.diagnostics("customer.Login");
// [{ severity: "warning", code: "W0201", span: {start: 9, end: 14},
//    message: 'Field "Login" is not in the schema; did you mean `login`?' }]

service.hover("customer.login", 10);
// { range: {start: 9, end: 14}, contents: '`login`: string\n\nExamples: `"1"`' }

service.signatureHelp("$substring(name, ", 17);
// { name: "$substring", signature: "<s-nn?:s>", params: ["$str", "$start", "$length"], activeParam: 1, ... }

schema is optional: without one, functions and variables are still completed. The second argument declares the variables the expressions are evaluated with. Its keys are variable names, with or without $, and its values are schemas, or null when the shape is unknown. The third argument declares host functions, as {name, signature, params, summary} objects. They are completed first, and hover and signature help show them:

new LanguageService(schema, null, [
  { name: "shipping", signature: "<ns:n>", params: ["weight", "region"], summary: "Shipping cost" },
]);

Create a new service when the schema, the variables or the functions change. Construction is cheap.

Completion items are ranked best first, and insertText replaces range. Field names that are not plain identifiers come quoted with backquotes. documentation and hover contents are Markdown.

CodeMirror 6

CodeMirror positions are UTF-16 offsets, like jlens offsets, so no conversion is needed. examples/web-editor/jlens-codemirror.js in the repository is a self-contained module to copy. It provides highlighting, completion, linting, hover and signature help. The core of it is:

autocompletion({
  override: [(context) => {
    const { range, items } = service.complete(context.state.doc.toString(), context.pos);
    return {
      from: range.start, to: range.end, filter: false,
      options: items.map((i) => ({ label: i.label, detail: i.detail, apply: i.insertText })),
    };
  }],
}),
linter((view) =>
  service.diagnostics(view.state.doc.toString()).map((d) => ({
    from: d.span.start, to: d.span.end, severity: d.severity, message: d.message,
  })),
),

Monaco

Monaco works with line and column positions; convert with the model:

monaco.languages.registerCompletionItemProvider("jsonata", {
  triggerCharacters: [".", "[", "$"],
  provideCompletionItems(model, position) {
    const { range, items } = service.complete(model.getValue(), model.getOffsetAt(position));
    const start = model.getPositionAt(range.start);
    const end = model.getPositionAt(range.end);
    return {
      suggestions: items.map((i) => ({
        label: i.label,
        insertText: i.insertText,
        detail: i.detail,
        documentation: i.documentation && { value: i.documentation },
        kind: monaco.languages.CompletionItemKind.Field, // map i.kind as needed
        range: new monaco.Range(start.lineNumber, start.column, end.lineNumber, end.column),
      })),
    };
  },
});
// Diagnostics: monaco.editor.setModelMarkers(model, "jlens", ...) with the same conversion.

Limits and memory

The size of the result counts a value each time it appears, as the JSON text does, and also limits the arguments passed to a host function. Wasm evaluations are synchronous, so there is no cancellation: use timeout. In a browser, an evaluation can also run in a Web Worker, and terminating the worker stops it; the web editor does this.

The depth limit is lower than on native targets (50,000). WebAssembly runs on the JavaScript engine’s stack, which is about 1 MiB in V8, and deep recursion beyond it raises a JavaScript RangeError. Such an error, like any trap, leaves the module instance unusable. The defaults keep evaluations inside the engine’s stack, with headroom. Raise maxDepth only if you have measured what your target engines allow. The jlens errors themselves (JlensError, TypeError) never affect the instance.

How deep an evaluation fits depends on the shape of the recursion and on V8’s compiler. V8 first runs the module with its baseline compiler (Liftoff), then recompiles the functions an evaluation uses often with its optimizing compiler (TurboFan), whose frames for jlens’s evaluator are about 1.6 times larger. Measured in Node.js 24 once the module is optimized, an audited recursion overflows V8’s stack near these depths (an unaudited one fits about 10% more):

The default of 500 leaves room for shapes not measured here, and for engines with smaller stacks. The evaluator chapter has the details.

Expression, Document and LanguageService objects live in WebAssembly memory. The garbage collector frees them through FinalizationRegistry. Call free(), or use using, to release them promptly in loops.

TypeScript

The package ships jlens.d.ts. Every argument and result is typed: AuditResult, AuditTree, AuditNode, Explanation, Diagnostic, Completions, CompletionItem, Hover, SignatureHelp, FunctionInfo, JsonSchema, SchemaOverride, EvaluateOptions, AuditOptions and JlensError. Results of evaluate are unknown; narrow them as usual.

The web editor example

examples/web-editor is a complete page with no build step, published with this book as the web editor:

• a JSONata editor (CodeMirror 6) with completion, diagnostics, hover and signature help, driven by a schema inferred from the input document;
• live evaluation, in a Web Worker (worker.js), so that a slow expression never blocks typing;
• the explanation and an audit tree viewer; clicking a node selects its source;
• the inferred schema;
• ready-made examples, including the samples of try.jsonata.org;
• a host function, $try (host-functions.js), that calls the function it receives, with an example that uses it: the worker registers it on each expression, and the page declares it to the language service.

cargo xtask wasm
cargo xtask serve        # then open the printed /examples/web-editor/ address

CodeMirror is loaded from esm.sh through an import map, so the page needs network access. The page imports the package by its name, jlens-wasm, which the same import map resolves. cargo xtask site builds the published site in target/site, the book with the editor in editor/, and cargo xtask serve --site serves it.

Testing

cargo xtask wasm-test builds the package, then runs crates/jlens-wasm/tests/suite.mjs twice:

• in Node.js;
• in a headless Chrome, Chromium or Edge (set JLENS_BROWSER to choose the executable).

--require-browser fails when no browser is found, instead of skipping. --example also checks that the web editor loads and evaluates its sample, from the repository and as the site serves it.

Audit and explainability

A JSONata expression returns a value. jlens can also tell you why it returned that value:

• which checks passed and which failed;
• what the compared values were;
• which items a filter rejected;
• which values were bound to variables.

The same evaluation, with the same result, produces an audit tree. An explanation is a readable summary of that tree.

A complete example

fixtures/shipping-rule.jsonata decides whether a store order gets free shipping. Its conditions carry annotations:

$approved := (
  /** @label "Authorized customer" @message "The customer login must be on the approved partners list." */
  customer.login in ["alice.walker99", "bob.harrison42"]
  and
  /** @label "Minimum order grand total" @message "The order grand total must exceed 50,000.00." */
  $total > 50000
  and …
);

When a order grand totals only 37,000, jlens eval --explain prints:

$ jlens eval -f fixtures/shipping-rule.jsonata -i order.json --explain
result: {"action":"REGULAR_FEE","code":"SHIP_FEE_402","delivery":"SLOW"}
• $total := $sum(items.(unitPrice * quantity)) = 37000
• $electronics := $sum(items[type = 'electronic'].(unitPrice * quantity)) = 17000
  • [type = 'electronic']: 2 of 3 items passed; rejected: {"code":"1","color":"blue",…}
• $blueTax := $sum(items[color = 'blue'].tax) = 1500
• $averageTax := $average(items.tax) = 2833.3333333333335
✘ $approved := ( customer.login in [...] and ... ) = false
  ✘ Minimum order grand total — $total > 50000 → false (37000 > 50000)
      The order grand total must exceed 50,000.00.
• $approved ? { ... } : { ... } → {"action":"REGULAR_FEE",…} [else]

The explanation follows only the decisions that mattered. The and chain failed, so it shows only the operand that failed, with its business message. The checks that passed are left out. For an approved order, every check is listed with ✔.

In Rust

#![allow(unused)]
fn main() {
use jlens_core::audit::{AuditOptions, explain};
use jlens_core::{Bindings, Expression, Options};

let rule = Expression::compile(&rule_source)?;
let (result, audit) = rule.evaluate_audited(
    Some(&document),
    &Bindings::new(),
    &Options::default(),
    &AuditOptions::decisions(),
);
println!("{}", explain(&audit));              // text
let json = serde_json::to_string(&audit)?;    // the full tree, e.g. to store with the decision
}

evaluate_audited returns the audit even when the evaluation fails. The error is in audit.header.error, and on the node that raised it.

What is recorded

The audit tree mirrors the evaluation, not the syntax: a node evaluated three times (once per item, or once per function call) appears three times. Each AuditNode has these fields:

The header records:

• the engine version;
• FNV-1a hashes of the expression and of the input document, so you can tell later exactly what was evaluated;
• the evaluation timestamp (the value of $now()) and its duration;
• the result, or the error.

Levels

At level decisions, nodes that decide nothing are collapsed: their recorded descendants move up to the nearest recorded ancestor. Inside a filter, the per-item evaluations are summarized into the filter’s statistics instead of being recorded one by one.

Bounds

Audits of large documents stay bounded. CaptureLimits sets:

Event logs and custom tracers

Auditing is built on a lower-level interface, the Tracer trait. The evaluator calls a tracer’s methods as it goes:

• enter/exit for each node;
• operands for each comparison;
• short_circuit, branch and bind;
• filter_begin, filter_item and filter_end for filters.

You can implement your own tracer, for example to collect metrics, and pass it to Expression::evaluate_traced. jlens provides three:

Performance

When no tracer is attached, auditing costs one check per evaluated node. Compared with the evaluator before tracing existed, interleaved benchmark runs measured between −5.7% and +2.7% across five workloads, within the noise of the machine. With a tracer attached, evaluation does more work in proportion to what is recorded. Level decisions is the cheapest useful level: for the shipping rule on a 200-item document, it takes about three times as long as an evaluation without a tracer, and full about fifteen times. cargo bench -p jlens-core -- audit measures this; see Performance.

Limitations

• Node spans refer to the expression’s own source. For expressions evaluated through $eval, and for the internal helper behind function composition ($f ~> $g), the reported source snippets refer to a different text.
• The audit captures values as JSON. Functions appear as "<function>", and undefined values are omitted.

Schema and autocomplete

jlens can learn the structure of your documents from samples and use it to help people write expressions. The language service then provides:

• completion of field names that knows which object you are in;
• warnings for fields that do not exist;
• hover documentation;
• signature help for functions.

The jlens-schema crate provides both parts, and the CLI and the WebAssembly package expose them.

1. Infer a schema from samples

$ jlens schema infer samples/*.json -o order.schema.json
inferred from 120 sample(s): order.schema.json

Glob patterns are expanded by jlens itself, so they also work in Windows shells. With --each, a sample file holding a top-level array counts as one sample per item.

Inference records, for every place in the documents:

The result is a standard JSON Schema 2020-12 document. Fields that are not required carry x-jlens-presence, the fraction of objects that had them. You can commit the file, review it, and edit it like any JSON Schema. jlens also reads JSON Schemas written by hand or by other tools: it supports $ref to local definitions, and it merges anyOf, oneOf and allOf.

2. Describe the fields

Samples tell jlens what fields exist, but not what they mean. Overrides add that information. An overrides file maps paths to settings; [] steps into array items:

{
  "items[].type": {
    "description": "Product category used for catalog classification",
    "enum": ["apparel", "electronic", "toy"]
  },
  "items[].tax": { "label": "Import tax" },
  "customer.login": { "description": "Login chosen at sign-up" },
  "internalRevision": { "hidden": true }
}

$ jlens schema infer samples/*.json --overrides overrides.json -o order.schema.json

A path that does not exist yet is created, so overrides can also declare fields that are absent from the samples.

3. Use it

In the terminal

$ jlens check --schema order.schema.json -f rule.jsonata
warning[W0201]: Field "unitprice" is not in the schema; did you mean `unitPrice`?
 --> rule.jsonata:4:22
  |
4 |   $sum(items[type = 'x'].unitprice)
  |                      ^^^^^^^^^

$ jlens schema complete --schema order.schema.json "items[type = 'electronic']."
code                     field     string
color                    field     string
type                     field     string
unitPrice                field     number
quantity                 field     integer
tax                      field     number

--cursor N sets the cursor position in characters, and --json prints the full completion items.

In Rust

#![allow(unused)]
fn main() {
use jlens_schema::{FunctionDecl, LanguageService, Schema};

let schema = Schema::from_json_schema(&serde_json::from_str(&schema_text)?)?;
let service = LanguageService::new(Some(schema))
    .with_variable("approved", None)         // variables the rule is evaluated with
    .with_function(                          // and its host functions
        FunctionDecl::new("shipping", "Shipping cost of a weight to a region")
            .signature("<ns?:n>")
            .params(["weight", "region"]),
    );

let completions = service.complete(text, cursor_byte_offset);
let diagnostics = service.diagnostics(text);
let hover = service.hover(text, cursor_byte_offset);
let help = service.signature_help(text, cursor_byte_offset);
}

Declare the host functions the expressions are evaluated with: they are completed first, and hover and signature help show their signature, parameters and summary. A declared function replaces a built-in function of the same name, as it does during evaluation.

All offsets and spans in the Rust API are byte offsets into the expression text. The WebAssembly package converts them to and from the UTF-16 offsets that browsers and editors use.

In the browser

The WebAssembly package exposes the same service for editors such as CodeMirror and Monaco; see the WebAssembly guide.

What completion understands

Completion knows the context at the cursor, that is, which object the next field name will be looked up in. It follows the evaluator’s rules:

Each field suggestion carries:

• its type (string, array<object>, integer | null, …);
• its label and description;
• its presence, for optional fields;
• its allowed values or examples.

Field names that are not plain identifiers are inserted with backquotes, and hidden fields are never suggested.

Completion works on incomplete expressions too, such as $count(items[ or items[type = 'x' and . That is the normal state of an expression while someone types it.

Diagnostics

LanguageService::diagnostics returns:

• syntax errors, from the recovering parser;
• annotation warnings;
• W0201 warnings for field names the schema does not have, with a “did you mean” suggestion when a field name is close;
• W0202 warnings for calls of unknown functions, with the same kind of suggestion, and W0203 warnings for calls with too few or too many arguments for the function’s signature. These need no schema. A function is known when it is built in, declared with LanguageService::with_function (or in jlens.json), or bound anywhere in the expression; a variable declared with with_variable may hold a function, so its calls are not checked;
• W0205 warnings for arguments whose type the function’s signature cannot accept, such as $sum(items.name) when name is a string. Only types that are known count: literals, the results of operators, and fields of the schema. A field that is absent passes any signature; the warning means that the call fails whenever the field is present;
• W0204 warnings for variables that are not declared, with a suggestion, when at least one variable is declared with with_variable (or in jlens.json): the declarations are then taken to be complete. Names bound in the expression, and functions, are known.

Missing fields are reported only where the context is fully known. Nothing is reported after a user-defined function, an object constructor or $lookup, so the warnings do not produce false alarms. Schemas inferred from samples can miss optional fields that the samples did not contain. Add those fields with overrides, or treat W0201 as advice.

Editors (language server)

jlens lsp is a language server: editors that speak the Language Server Protocol get, in JSONata files:

• completion of fields, variables, and built-in and host functions;
• diagnostics: syntax errors, annotation problems, and fields that the schema does not have, with a “did you mean”;
• hover, with the type and description of fields and the documentation of functions;
• signature help in function calls.

It is the language service of the web editor, behind a standard protocol. Install it with the CLI:

cargo install jlens-cli
jlens lsp          # what editors run; it speaks LSP on standard input and output

Configuration: jlens.json

A jlens.json at the root of a workspace folder says which schema, variables and host functions apply to which expression files:

{
  "rules/**/*.jsonata": {
    "schema": "schemas/order.schema.json",
    "variables": {"minimum": {"type": "number"}, "region": null},
    "functions": [
      {"name": "shipping", "signature": "<ns:n>", "params": ["weight", "region"],
       "summary": "Shipping cost of a weight to a region"}
    ]
  },
  "**/*.jsonata": {}
}

• Keys are glob patterns, matched against the file’s path relative to the folder of jlens.json. * stays within a directory and ** crosses directories. The first matching pattern applies; files that no pattern matches get completion and diagnostics without a schema.
• schema is a JSON Schema of the input documents, relative to the folder of jlens.json. jlens schema infer creates one from sample documents; see the schema guide.
• variables declares the variables the expressions are evaluated with, with a JSON Schema of each value, or null when the shape is unknown. Names can start with $.
• functions declares the host functions: name, and optionally signature, params and summary.

All settings are optional. When jlens.json or a JSON file changes, the server reloads the configuration and the schemas, and checks the open files again. It relies on the editor’s file watching, or on the file being saved in the editor. A configuration that cannot be read is reported as a message, and its files are checked without settings.

The client can also pass the same object as initializationOptions, relative to the first workspace folder. Its patterns come before those of jlens.json.

jlens check reads the same file, so that CI reports what the editor shows.

Neovim

With Neovim 0.11 or later, in init.lua:

vim.filetype.add({ extension = { jsonata = "jsonata" } })
vim.lsp.config("jlens", {
  cmd = { "jlens", "lsp" },
  filetypes = { "jsonata" },
  root_markers = { "jlens.json", ".git" },
})
vim.lsp.enable("jlens")

Helix

In languages.toml:

[[language]]
name = "jsonata"
scope = "source.jsonata"
file-types = ["jsonata"]
roots = ["jlens.json"]
language-servers = ["jlens"]

[language-server.jlens]
command = "jlens"
args = ["lsp"]

Emacs

With Eglot (built into Emacs 29):

(define-derived-mode jsonata-mode prog-mode "JSONata")
(add-to-list 'auto-mode-alist '("\.jsonata\'" . jsonata-mode))
(with-eval-after-load 'eglot
  (add-to-list 'eglot-server-programs '(jsonata-mode "jlens" "lsp")))

Visual Studio Code

The jlens extension (thkprado.jlens, from 0.15) highlights JSONata and starts jlens lsp for the jsonata language, in files ending in .jsonata. It looks for the server in this order:

1. the setting jlens.server.path, the path of a jlens program;
2. the jlens binary bundled with the extension, in its packages for Linux, macOS and Windows on x86-64, and Linux and macOS on ARM64;
3. jlens on the PATH, on other platforms.

When it finds none, it says so, and highlighting keeps working. jlens.server.path is read from user settings only, never from a workspace’s: a repository cannot choose which program runs. The command jlens: Restart Language Server starts the server again, after an update of jlens for example, and changing the setting does too.

The jlens output channel shows the server’s version and path, and its messages. The setting jlens.trace.server (messages or verbose) adds the protocol’s messages to it.

The extension also checks jlens.json as it is edited, with a JSON Schema: unknown settings, misspelled ones and wrong types are underlined, and hovers describe each setting.

If another JSONata extension is installed, VS Code uses both: disable the other’s language features, or you get the diagnostics of both. In the browser, the WebAssembly package offers the same service to CodeMirror and Monaco.

Positions

The server counts positions in UTF-16 code units, the protocol’s default, or in bytes when the client offers UTF-8 (general.positionEncodings). Documents are synchronized incrementally. Positions past the end of a line or of the document are clamped to it, as the protocol requires.

Errors

A bug in jlens that panics while the server handles a request does not stop the server: the request is answered with an InternalError, and a panic while it handles a notification (such as checking a document after a change) is logged with window/logMessage. The panic itself is printed on the standard error, which editors keep in the server’s log. Please report it, with the text of the document.

Architecture overview

                ┌───────────────────────── jlens-core ──────────────────────────┐
source ──► lexer ──► Pratt parser ──► raw tree ──► process ──► Ast (Expression)  │
                │      (recover mode, annotations)    (paths, stages, %,         │
                │                                      tail calls)       │      │
                │                                                        ▼      │
input JSON ──► Value / Document ───────────────────────────────────► evaluator  │
                │                                    function library ──┘  │    │
                │                                    Tracer events ◄───────┘    │
                │                                       │                       │
                │                     AuditCollector ◄──┴──► JsonlTracer, ...   │
                └──────────────────────────┬───────────────────────────────────┘
                                           │ result, AuditTree, Explanation
        jlens-schema: Schema (inference, JSON Schema) ─► typing ─► LanguageService
        jlens-cli: the `jlens` binary        jlens-wasm: the ES module for browsers

Crates

The core crate has no dependency on the others. Everything a tool needs from the language is public there:

• the syntax tree, with node ids and spans;
• recovering parsing;
• the function catalog;
• the tracer events.

From source to result

1. Lexing and parsing (Parsing). The lexer produces tokens with byte spans and collects documentation comments. A Pratt parser, ported from jsonata-js, builds a raw tree. In recover mode it records errors and keeps going, which editors need. Annotations are attached to the subexpression that follows them.
2. Processing. The raw tree becomes the Ast:
   • path steps and their stages (filters, sorting, #, @) are laid out;
   • parent references (%) are resolved to ancestor slots;
   • tail calls are marked. The result is immutable and shared by all evaluations of the expression.
3. Evaluation (Evaluation). A tree-walking interpreter over the Value model. Values are reference counted, with sequences flagged as in jsonata-js; see ADR 0003. Functions are values. Tail calls run in a trampoline. Guardrails bound the depth and the time.
4. Tracing. When a tracer is attached, the evaluator reports events: entering and leaving a node, operands, short circuits, branches, bindings and filters. The audit collector turns them into an AuditTree, and explain summarizes the tree. See ADR 0004.

Tooling

jlens-schema builds on recovering parsing (Schema and language service):

• Schema records the shape of sample documents and round-trips through JSON Schema.
• The typing walker follows an expression the way the evaluator would. It computes what each part of the expression refers to in the schema: paths, filters, variables and function results.
• The LanguageService answers completion, diagnostics, hover and signature help from that information. A placeholder is substituted at the cursor so that incomplete expressions parse.

The CLI and the WebAssembly package are thin layers over these crates. The WebAssembly package converts offsets between UTF-8 bytes (Rust) and UTF-16 code units (JavaScript) at the boundary.

Quality

• Conformance testing: the official JSONata test suite runs on every cargo test, against a baseline that may only improve. Property tests check that nothing panics.
• Performance: criterion benchmarks and a comparison with jsonata-js.
• Documentation checks: cargo xtask doc-check runs every example in this book. The generated pages (functions, errors) are checked against the code.

The design decisions are recorded in the decision records.

Parsing

Parsing turns source text into the tree the evaluator walks. It has three stages, each ported from src/parser.js in jsonata-js:

source ──► lexer ──tokens──► Pratt parser ──raw tree──► process ──► Ast
            (lexer.rs)         (parser.rs)               (process.rs)  (ast.rs)

Lexer

The lexer produces one token at a time, on demand, because one character is ambiguous: / starts a regular expression in operand position (/ab+c/i) and means division in operator position (a / b). The parser passes an infix flag with each request.

Token kinds are names, variables ($x), strings, numbers, the literals true, false and null, regexes and operators. and, or and in are lexed as operators; the parser turns them back into names when they appear where an operand is expected, as in and.or.

Differences from the reference lexer:

• Spans. Every token records its full byte range. The reference implementation only records the position just after a token.
• Doc-comments. /** ... */ comments are kept and attached to the next token, for annotations. Ordinary comments are skipped as before.
• Surrogates. 𝄞-style escape pairs are combined into one character. A lone surrogate becomes U+FFFD, because Rust strings cannot hold one.

Pratt parser

JSONata’s grammar is mostly operators, and a Pratt parser (top-down operator precedence) handles it with a table of binding powers:

Each token can define two parse functions:

• nud, which parses the token in prefix position, such as a literal, -x, (block), [array] or {object};
• led, which parses it in infix position, such as a + b, a[...], f(...) or a{...}.

The core loop is short:

expression(rbp):
    t = current token; advance
    left = t.nud()
    while rbp < binding power of current token:
        t = current token; advance
        left = t.led(left)
    return left

A few constructs are desugared while parsing:

• x ?? y and x ?: y become a fallback node (NodeKind::Fallback) that evaluates x once, then keeps its value if it exists (??) or is true as a Boolean (?:). jsonata-js desugars them into the conditions $exists(x) ? x : y and x ? x : y, with a copy of x: nesting them doubles the tree at each level;
• function($a){...} is recognised as a lambda when the callee of a call is the name function or λ.

Annotations and binding power

When the first token of an expression carries a doc-comment, the loop above runs in two phases. First it parses with binding power max(rbp, 30), which stops at and or or, and attaches the annotation to that result. Then it continues with the original rbp. Pratt parsing builds the same tree regardless of where the loop is split, so the annotation marks a subtree without changing the shape of the tree. The test annotations_never_change_the_tree checks this property.

Post-processing

process.rs is the port of processAST. It turns the raw tree into the evaluator’s tree:

• Paths. Nested . operators become one Path node with a flat list of steps. String steps become names, and number or boolean steps are rejected (S0213).
• Predicates. step[expr] becomes a filter stage on the step. On a non-path expression such as $var[0], it becomes a predicate that applies to the whole result.
• Bindings and clauses. @$v and #$i set focus and index on the step and switch it to tuple mode. ^(...) becomes a Sort step, and {...} after a step becomes its group.
• The parent operator %. Each % gets an ancestor slot. The processor walks back through the enclosing path to the step whose input is the wanted ancestor, and marks it to bind that slot’s label (!0, !1, …). That step and the steps after it run as a tuple stream. Slots live in Ast::slots, an arena, because the reference implementation shares and mutates slot objects across nodes.
• Tail calls. A function call in tail position of a lambda (either branch of a conditional, or the last expression of a block) is wrapped in a thunk. The evaluator’s trampoline runs thunks in a loop, so tail-recursive functions don’t grow the stack.

Finally, nodes receive sequential NodeIds in pre-order. Audit trees will refer to nodes by these ids.

Error recovery

In recover mode, an error does not abort parsing. The behaviour follows jsonata-js:

• the error is recorded;
• the rest of the input is consumed;
• the current token becomes an error marker with binding power 10, which wraps whatever was parsed before it.

So Account.Order[0;].Product yields path(Account, Order[0]) plus error S0202.

jlens recovers in more cases than jsonata-js:

• Lexer errors, such as an unterminated string while the user is typing, are recovered.
• Post-processing errors (S0209, S0210, S0213, S0215–S0217) become diagnostics and error nodes.
• An expression that starts at the error marker reuses the recorded error, instead of reporting an extra S0211 for an undefined token.

The seven recovery cases from jsonata-js’s test/parser-recovery.js are ported in crates/jlens-core/tests/parser.rs. They check the same error codes, positions and partial trees. The same test file also parses every expression of the official suite: all valid expressions must parse, and all invalid ones must fail with the expected S0xxx code.

Evaluation

The evaluator (crates/jlens-core/src/eval.rs) is a tree-walking interpreter over the processed syntax tree. It is a function-by-function port of src/jsonata.js in jsonata-js. That code effectively defines JSONata’s subtler behaviour, such as when a result is a sequence or an array, so jlens keeps its structure recognisable. Unlike jsonata-js, evaluation is synchronous.

Expression::evaluate(input, bindings)
  └─ eval::evaluate          converts input JSON, wraps an input array, binds $$ and bindings
       └─ Evaluator::eval    one call per node: dispatch, predicates, group-by, sequence unwrapping
            ├─ path / step / tuple_step / filter / stages      location paths
            ├─ binary / array_constructor / group / sort       operators and constructors
            └─ call / apply / partial / apply_expression       functions

Values

See ADR 0003 for the reasoning. The short version:

Sequences

A sequence is the result of navigating a path. After evaluating any node, a sequence of length 0 becomes undefined, and a sequence of length 1 becomes its only item, unless KEEP_SINGLETON is set by the [] modifier. Arrays that come from the input, and arrays built with [...], are not sequences, so they never collapse. This single rule, applied at the end of Evaluator::eval, is why Account.Order.Product.Price returns a number for one product and an array for several.

When the input document is itself an array, it is wrapped in a one-item sequence flagged OUTER_WRAPPER, so that the array counts as a single context value. $ unwraps it again.

Paths and tuple streams

path evaluates its steps left to right. Each step maps over the current sequence and flattens the results by one level. Array constructors flagged CONS are not flattened.

Steps that bind variables switch the path into tuple-stream mode from that step on. These are @$focus, #$index, and steps marked as the ancestor for a %. In this mode each item is carried as a tuple, an object holding @ (the current value) and the bound variables. Later steps, predicates, sort keys and group-by expressions are evaluated in a frame created from the tuple. At the end of the path the tuples are mapped back to their @ values, unless an enclosing path still needs the ancestry.

Functions and the trampoline

A lambda value captures its definition (Arc<LambdaDef>, shared with the tree), the environment frame in which it was created, and the context value. Applying it:

1. validates the arguments against the lambda’s signature, if it has one;
2. creates a child frame of the captured environment and binds the parameters (missing ones are bound to undefined, so they shadow outer variables);
3. evaluates the body.

When post-processing marked a call as being in tail position, the body evaluates to a thunk instead of making the call. Evaluator::apply runs thunks in a loop, so tail-recursive functions run in constant stack depth. $loop(100000, 0) in the test suite exercises this.

Built-in functions are Native entries in functions/mod.rs, each with its signature and declared parameters. Host functions (HostFunction, from the bindings) have an optional signature and an arity; their arguments and results are converted to and from JSON (ADR 0006). Partial application ($f(?, 2)) works for lambdas, built-ins and host functions.

Signature validation

Signatures such as <s-nn?:s> are checked by turning the arguments into a string of type symbols (s, n, b, l, a, o, f, or m for undefined). That string is matched against the parameter patterns by a small greedy backtracking matcher. The matcher is equivalent to the regular expression jsonata-js builds, but needs no regex engine. The match then:

• fills in the context value for - parameters (T0411 if its type is wrong);
• wraps single values for a parameters;
• checks array subtypes (T0412).

Environments and memory

Frames form a chain from the innermost scope to the root; the built-ins sit behind the root. A closure stored in the frame that it captured creates a reference cycle, as in $f := function(){ $f() }. Reference counting cannot reclaim such a cycle on its own. The evaluator therefore records every frame that receives a function, array or object binding, and clears those frames when the evaluation ends.

Guardrails

jsonata-js has no depth limit of its own; it stops when the JavaScript stack overflows. jlens needs limits so that it fails with an error instead of crashing the process.

• Native targets. The parser, the post-processing pass, the evaluator and the schema typing walker grow their stack on demand (with the stacker crate), so the default depth limit is generous. The CLI also runs on a thread with a 64 MiB stack, which covers the other recursive walks of the tree (printing it as JSON, for example).
• WebAssembly. The stack cannot grow, and an overflow traps, which leaves the module instance unusable. jlens links the module with a 4 MiB stack and checks the space left before each recursive step (jlens_core::stack), failing with D1011 (evaluation) or U1001 (parsing) instead. WebAssembly frames also use the JavaScript engine’s own stack, which is about 1 MiB in V8 and cannot be measured from inside the module; the lower default depth limit (500) keeps evaluations inside it. The WebAssembly guide has the measurements behind it. A larger maxDepth can raise a JavaScript RangeError and break the instance.
• Calls from native functions. A lambda whose body is a tail call leaves no expression on the stack, so a recursion through a higher-order function (function($n){ $map([$n], function($v){ $f($v) }) }) deepens only through the native function’s calls back. Each of those calls counts as a level of depth and checks the stack too.

The nesting limit bounds every recursive walk of the syntax tree, including the ones that do not grow the stack, such as cloning and dropping it. Both nested subexpressions and the operators of a chain count: 1 + 2 + 3 is a tree two levels deep, built by a loop rather than by recursion. 1,000 levels is far more than any hand-written expression uses.

The size limits stop an expression that doubles a string or an array on each step, or a path whose steps multiply the length of a sequence, before it runs out of memory, which would abort the process. They are checked by the operators that build strings and arrays (&, ranges, array constructors, path steps) and on the value that each built-in or host function returns; $join checks the length before joining. The operators check before they build the value; a function’s result is checked once it has returned. * and **, which flatten nested arrays, count the items as they go.

The string and array limits apply to each string and array on its own, and values are shared rather than copied: an object bound to a variable can be nested in a new object twice, then that object twice, and so on. Thirty such steps stand for about a billion copies of the first object while using little memory, until something writes them all out. So everything that writes a value out is limited too:

• The result, which becomes JSON when the evaluation ends, is measured first: its size is the length of its compact JSON text, counting a shared value each time it appears. The walk stops once the size passes Options::max_result_size, and the evaluation fails with U1001, before anything is converted. Escaped characters count as one byte, so the size is slightly below the length of the text that format_json prints.
• The arguments of a host function are measured together in the same way, and so is its context value (HostCall::input).
• $string, $clone (and the transform operator, which clones its input) and & write JSON text, which is a string: the text is checked against Options::max_string_length as it is written.
• The value of an error (T2001, T0410 and others) is left out when it is larger than the default size of a result.

Tracing hooks

The evaluator reports its work to an optional Tracer; ADR 0004 records why. The hooks sit at these points:

When no tracer is attached, eval calls the untraced code directly. The traced variant is a separate function marked cold, so the hot path does not change. The audit guide describes the tracers built on these hooks.

Schema inference and the language service

The jlens-schema crate has four modules:

Inference

A Shape describes all the values found at one place in the documents. Shape::observe folds one more value into it:

• Types. Each value adds its JSON type. A number written without a fraction is an integer; one with a fraction or an exponent turns the shape into number. The float_roundtrip feature of serde_json preserves that distinction.
• Objects. Every field gets a sub-shape, and every object observed increments a counter. required is field observations == objects observed, and presence is the ratio of the two.
• Arrays. All items of all arrays observed here fold into one items shape.
• Scalars:
  • numbers update the minimum and the maximum;
  • strings narrow a common format;
  • up to ten distinct scalar values are tracked. They become an enumeration once at least two distinct values exist and some value repeats, which keeps unique identifiers out of enumerations.

Inference never looks back at earlier samples, so Schema::observe can process a stream of documents in constant memory per distinct field.

Static typing

A type Ty is a set of shapes plus a partial flag. The flag means that something unknown contributed: a user function, an object constructor, or a missing schema. Walker::walk computes the type of each node and applies the evaluator’s context rules:

• A path evaluates each step against the elements of the previous step’s result. Elements are the shapes with arrays replaced by their items, recursively, which is the flattening the evaluator applies.
• Stages of a step (predicates and index bindings) see the step’s elements. A group-by sees the elements of the path’s result.
• @$var binds the step’s elements and leaves the context unchanged. #$i binds a number.
• $x := e binds x to the type of e until the end of the enclosing block.
• Lambdas are walked with their parameters bound to unknown types. When a lambda is the callback of $map, $filter, $single, $sort or $reduce, its parameters get the element type of the array argument.
• Calls to built-ins that return their first argument ($sort, $filter, $reverse, $distinct, $shuffle, $single, $clone, $sift) keep its type. $append returns the union of its arguments’ types. Other built-ins return scalars, except those whose structure is opaque, such as $map and $lookup, which return unknown types. A name bound outside the expression (a declared variable or host function) is not a built-in, and its calls return unknown types.

While walking, the walker reports events to a visitor:

• a field reference, with the context it is looked up in;
• a variable reference, with its type and the scope.

The services are all visitors over the same walk.

Completion

Completion has to work on text that does not parse yet, so it proceeds in four steps:

1. Lexical check. If the cursor is inside a string or a comment, nothing is suggested. If it is inside a backquoted name, a quoted field name is completed.
2. The word at the cursor becomes the replacement range. If the previous token ends an operand (a name, a literal, or a closing bracket) and is not and, or or in, and there is whitespace before the cursor, operators are suggested instead.
3. The placeholder trick. The word is replaced with a sentinel identifier (__jlens_cursor__, or $__jlens_cursor__ for variables), and the text is parsed with error recovery. The parse succeeds whenever the expression before the cursor is well formed, because the sentinel completes it. Anything broken after the cursor becomes an error node.
4. The walk finds the sentinel’s event. For a field, it yields the context type, whose fields are the suggestions. It also yields the scope, whose variables are suggested with the declared host functions and the built-in functions.

The two field paths that matter most, items. and items[, are therefore handled by the same machinery as $map(items, function($i){ $i..

Diagnostics, hover and signature help

• Diagnostics combine the recovering parser’s errors and annotation warnings with W0201 for every field reference whose context is fully known (Ty::is_known_objects) but lacks the field. A Levenshtein distance of at most 2 picks the “did you mean” suggestion. The walk also reports every call of a $name and every name it binds (:=, parameters, @, #); the calls are checked after the walk, because a function may call a name bound after it, or itself. Calls of unknown names get W0202, and calls whose argument count the signature rejects get W0203: parameters marked ? or - are optional, and + accepts any number.
• The call events carry the type of each argument. Literals and operators have a JSON type (Ty::literal), schema fields have theirs, and anything else is unknown. Ty::symbols turns a fully known type into signature symbols, keeping the items of arrays apart, and the signature is matched as jsonata-js matches it (greedy, with backtracking), except that an argument whose type is unknown matches any parameter. A path may yield a single item of an array, so an array also matches what its items match. When no match exists, W0205 names the argument jsonata-js would report in T0410 or T0412. Undefined is left out: a field that is absent passes any signature, but one that is present cannot.
• When variables are declared, the declarations are taken to be complete, and references to names that are neither declared, bound anywhere in the expression, nor functions get W0204.
• Hover keeps the innermost field or variable event whose span contains the offset, and renders its type and documentation as Markdown.
• Signature help scans backwards from the cursor for the unmatched ( of a call, ignoring strings and comments and balancing brackets. It counts the top-level commas to find the active parameter, and looks the function up among the declared host functions, then in the built-in catalog.

Roadmap

How 0.1 was built

jlens was built in phases. A phase was complete only when its documentation was written.

Releases

The changelog lists every change.

Priorities

Three principles order the work after 0.2:

1. The core is a library for custom integrations. Applications embed jlens-core in their own pipelines. It must stay small and decoupled, let the application set every limit, never panic, and change its API only on purpose.
2. The CLI is a tool for automation and AI agents. It never prompts, it gives the same output for the same input, and it has one documented machine-readable contract: what goes to standard output, what goes to standard error, and what each exit code means.
3. Editors come after. The language server is done; the VS Code extension is 0.15.

For ad-hoc JSON queries, jq remains the usual tool. jlens is for JSONata that will run somewhere else, such as a rules engine or a jsonata-js service. It checks those expressions, tests them and explains their results before they ship.

0.3: a machine contract and core guardrails

0.4: catching mistakes before they run

0.5: safe releases, enforced limits and proven compatibility

0.6: bounded regular expressions, wider differential tests, signed packages

0.8: proven robustness and fast single evaluations

0.9: numbers and dates at the edges, and tools for the next steps

0.10: performance and robustness debts

Explorations: each one may end in a fix, or in the cause written down and the debt accepted. Both outcomes close the item.

0.11: faster evaluation of common shapes

A comparison with jsonata-core 2.2.11, another Rust port, on the cases of benches/ (evaluate_document against its evaluate, both converting the result to serde_json::Value) found it 2 to 5 times faster on five cases: path (0.9 against 0.1 µs), filter + aggregate (164 against 93 µs), group-by (570 against 123 µs), sort + map (1.79 ms against 655 µs) and the shipping rule (433 against 254 µs). Both engines are about equal on the other cases, and jlens is faster on regex. Measuring parts of these expressions traced the gaps to a few places in eval.rs, where work is repeated or allocated that no result depends on. Each item removes one of them without changing any result, error or evaluation order that jsonata-js shows. jsonata-core’s other technique, a second evaluator for simple expressions, is left out: two evaluators of the same language can disagree.

Every item is measured with cargo xtask ab against the commit before it, and checked with the conformance suite and a cargo xtask differential run. When a tracer is attached, evaluation takes the general path, so audits keep reporting every node.

0.12: jlens online

Today the book and the web editor exist only in the repository: the README links no site, CI builds the book only to check it, and trying the editor takes a clone, a Rust toolchain and cargo xtask wasm. This release publishes both on GitHub Pages, at each release tag, as try.jsonata.org does for jsonata-js. The editor shows what sets jlens apart (completion from an inferred schema, diagnostics, the explanation and audit tree of every result), and everything runs in the browser: no expression or document leaves the visitor’s machine.

The site is the book at the root and the editor under editor/, built from the files of the tag, so the editor runs the engine that the book of that version describes. The first deployment, and enabling Pages in the repository settings, wait for the maintainer.

0.13: host functions that call back

A host function cannot call a JSONata function it receives as an argument: the argument arrives as None (undefined in WebAssembly), as ADR 0006 decided. Rule engines need it for helpers written in the host language that apply a function given to them ($retry($fetch, 3), $memoize(function($k) {...}), a lookup with a fallback function). This release adds a callable handle without changing the functions already written against ADR 0006.

0.14: more ways to install the CLI

Today the jlens command comes from the release archives, cargo binstall or cargo install. Tools like jq also come from the package managers people already use. Each channel adds a step to every release, and one more place where a release can fail. So they come in order of reach, each published by release.yml from the archives already built and attested, never rebuilt, and each one stays only if it updates without manual work.