Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Rust crate

jlens-core is the JSONata engine. jlens-schema adds schema inference and the language service; see the schema guide. This chapter covers jlens-core. Its API reference is on docs.rs.

[dependencies]
jlens-core = "0.2"
serde_json = "1"

The snippets in this chapter come from crates/jlens-core/tests/guide.rs, which runs them as tests.

Compile once, evaluate many times

    use jlens_core::{Bindings, Expression};
    use serde_json::json;

    let expr = Expression::compile("$sum(items[type = $kind].(price * qty))")?;
    let order = json!({"items": [
        {"type": "book", "price": 12.5, "qty": 2},
        {"type": "pen", "price": 1.5, "qty": 10}
    ]});
    let bindings = Bindings::new().var("kind", "book");
    let total = expr.evaluate(Some(&order), &bindings)?;
    assert_eq!(total, Some(json!(25)));

Expression::compile parses the expression and prepares it for evaluation. A compiled expression is immutable and cheap to clone. Keep it, and evaluate it as often as needed, from any thread (see Threads).

Documents and results are serde_json::Values. Object fields keep their order (jlens enables serde_json’s preserve_order). Variables come from Bindings: $kind above. Bindings can also be collected from (String, Value) pairs, such as the fields of a JSON object.

Undefined

JSONata distinguishes nothing (an empty sequence) from null. In the API, nothing is None, both for the input and for the result:

    let expr = Expression::compile("customer.email")?;
    // No input document: `None`. A result of `None`: the expression produced nothing.
    assert_eq!(expr.evaluate(None, &Bindings::new())?, None);
    assert_eq!(
        expr.evaluate(Some(&json!({"customer": {}})), &Bindings::new())?,
        None
    );

Guardrails

Expressions can recurse and loop. Bound their cost with Options, especially for expressions that users write:

    use jlens_core::Options;
    use std::time::Duration;

    let options = Options::new()
        .timeout(Duration::from_millis(100)) // D1012 when exceeded
        .max_depth(500); // D1011 when exceeded
    let runaway = Expression::compile("($f := function($n) { $f($n + 1) }; $f(0))")?;
    let err = runaway
        .evaluate_with(None, &Bindings::new(), &options)
        .unwrap_err();
    assert_eq!(err.code(), "D1012");

The default depth limit is 50,000 levels natively; the stack grows as needed. There is no default timeout. Expressions nested more than 1,000 levels deep (counting each operator of a chain such as 1 + 2 + 3) are rejected when compiled (U1001). A string longer than Options::max_string_length (256 MiB), an array longer than Options::max_array_length (16,777,216 items) or a result larger than Options::max_result_size (256 MiB of JSON text) stops the evaluation with U1001 instead of exhausting memory. See Guardrails.

Cancellation

A CancelToken stops an evaluation from another thread, for example when the request that started it goes away. Clones of a token share one flag:

    use jlens_core::CancelToken;
    use std::{thread, time::Duration};

    let token = CancelToken::new();
    let options = Options::new().cancel_token(token.clone());
    let forever = Expression::compile("($loop := function() { $loop() }; $loop())")?;
    let worker = thread::spawn(move || forever.evaluate_with(None, &Bindings::new(), &options));
    thread::sleep(Duration::from_millis(50));
    token.cancel(); // from any thread that holds a clone
    let err = worker.join().unwrap().unwrap_err();
    assert_eq!(err.code(), "U1003");

The evaluation checks the token between steps, as it checks the timeout, and fails with U1003. A host function that is running finishes first. A cancelled token stays cancelled, so create one per job.

Reproducible results

$now() and $millis() read the system clock once per evaluation, and $random() and $shuffle() draw from a generator seeded from it. Fix both in Options to get the same result from the same expression, input and options, for tests or for replaying a decision:

    let options = Options::new()
        .now(1_700_000_000_000) // $now(), $millis()
        .seed(42); // $random(), $shuffle()
    let expr = Expression::compile("{'at': $now(), 'order': $shuffle([1, 2, 3, 4, 5])}")?;
    let bindings = Bindings::new();
    let first = expr.evaluate_with(None, &bindings, &options)?.unwrap();
    assert_eq!(first["at"], json!("2023-11-14T22:13:20.000Z"));
    // Same expression, input and options: the same result.
    assert_eq!(expr.evaluate_with(None, &bindings, &options)?, Some(first));

A seed reproduces results with the same version of jlens, on any platform; it does not reproduce the numbers jsonata-js draws from Math.random().

Errors

Every error has a stable code: S0xxx for syntax errors, T for type errors, D for dynamic errors, and U1001 for limits. Match on the code, not on the message:

    let err = Expression::compile("Account.Order[0;]").unwrap_err();
    assert_eq!(err.code(), "S0202");
    assert_eq!(err.message(), r#"Expected "]", got ";""#);
    assert_eq!(err.position(), Some(16)); // UTF-16 offset, as in jsonata-js
    let span = err.span().unwrap(); // byte range of the offending token
    assert_eq!((span.start, span.end), (15, 16));
    assert_eq!(err.token().as_deref(), Some(";"));

    let expr = Expression::compile("name + 1").unwrap();
    let err = expr
        .evaluate(Some(&json!({"name": "x"})), &Bindings::new())
        .unwrap_err();
    assert_eq!(err.code(), "T2001");
    assert_eq!(err.value(), Some(&json!("x")));
    // `Display` gives the code, the message and the position.
    assert_eq!(
        err.to_string(),
        r#"T2001: The left side of the "+" operator must evaluate to a number (at position 6)"#
    );

position matches jsonata-js: a UTF-16 offset, usually just after the offending token. span is a byte range, for slicing the source in Rust. Errors serialize to JSON with their code, message and details.

Host functions

The application can provide its own functions, called from expressions like built-in ones. Bind a HostFunction with Bindings::function:

    use jlens_core::{Error, HostFunction};

    // `$shipping(weight, region?)`: the region defaults to the context value's `region`.
    let shipping = HostFunction::with_signature("<ns?:n>", |args, call| {
        let weight = args[0].as_ref().and_then(|w| w.as_f64()).unwrap_or(0.0);
        let region = match &args[1] {
            Some(region) => region.clone(),
            None => call
                .input()
                .map_or(json!(null), |order| order["region"].clone()),
        };
        let rate = match region.as_str() {
            Some("north") => 1.5,
            Some("south") => 2.0,
            _ => return Err(Error::new("SHIP01").with("message", format!("No rate for {region}"))),
        };
        Ok(Some(json!(weight * rate)))
    })?;
    let bindings = Bindings::new().function("shipping", shipping);

    let expr = Expression::compile("$shipping($sum(items.weight))")?;
    let order = json!({"region": "south", "items": [{"weight": 2}, {"weight": 1.5}]});
    assert_eq!(expr.evaluate(Some(&order), &bindings)?, Some(json!(7)));

    let err = Expression::compile("$shipping(3, 'west')")?
        .evaluate(None, &bindings)
        .unwrap_err();
    assert_eq!(err.code(), "SHIP01");
    assert_eq!(err.message(), r#"No rate for "west""#);
    assert_eq!(err.position(), Some(10)); // the call, as for built-in functions
  • Arguments and results are JSON. None stands for undefined. A function passed as an argument (such as a lambda) arrives as None; HostCall::function gives a handle to call it (below).
  • Signatures use the notation of the built-in functions. HostFunction::with_signature checks the arguments before each call, with the same errors (T0410, T0412). A parameter with - takes the context value when the argument is missing. HostFunction::new(arity, ...) skips the check and passes the arguments as given.
  • The arity tells higher-order functions how many arguments to pass: $map passes the index only to functions with two parameters or more. Partial application ($shipping(?, 'north')) uses it too.
  • The context value is available through HostCall::input, which converts it to JSON only when called.
  • Errors returned by the function stop the evaluation. Give them a code of your own and a message detail. Like other errors, they get the position and token of the call.
  • Bindings take precedence over built-in functions of the same name, as in jsonata-js. A name holds a value or a function: binding one replaces the other.

Host functions are Send + Sync, so bindings that hold them can be shared between threads. A host function runs to completion: a timeout is checked again only once it returns, or when it calls a function it received.

Functions as arguments

A host function can call a function it receives, such as a fallback or a step to retry. HostCall::function(i) gives a Callable for argument i when it is a function: a lambda, a built-in function, a partial application, a chain or another host function.

    // `$rate(region, fallback)`: the rate from the application's table, or `fallback(region)`.
    let rate = HostFunction::with_signature("<sf:n>", |args, call| {
        match args[0].as_ref().and_then(|r| r.as_str()) {
            Some("north") => Ok(Some(json!(1.5))),
            Some("south") => Ok(Some(json!(2))),
            _ => {
                let fallback = call.function(1).expect("the signature requires a function");
                fallback.call(&[args[0].clone()])
            }
        }
    })?;
    let bindings = Bindings::new().function("rate", rate);

    let expr = Expression::compile("$rate(region, function($r) { $length($r) / 10 })")?;
    let order = json!({"region": "west"});
    assert_eq!(expr.evaluate(Some(&order), &bindings)?, Some(json!(0.4)));

    // An error in the fallback comes back to the host function, which returns it here.
    let err = Expression::compile("$rate('west', function($r) { $error('No rate for ' & $r) })")?
        .evaluate(None, &bindings)
        .unwrap_err();
    assert_eq!(err.code(), "D3137");
    assert_eq!(err.message(), "No rate for west");
  • The call is part of the evaluation, as when $map calls its function. The context value is null, and a lambda keeps the variables of its definition. The call counts as a level of depth, so a recursion through a host function ends with D1011; it checks the timeout and the cancellation token; its result is limited by max_result_size. An audit shows its steps under the host function’s call.
  • Arguments and the result are JSON, as for the host function itself. A sequence comes back as an array or a single value; a function crosses as None, in both directions.
  • Errors come back as Err, with their codes and positions. The host function can handle them, or return them, which ends the evaluation.
  • The handle borrows the call: it cannot be kept and called after the host function returns.

Many expressions, one document

Each evaluate call converts the parts of its input document that the expression reads into jlens’s internal representation, one level at a time: a path costs the width of the objects it crosses. An expression that reads most of a document of a few dozen kilobytes (a filter over every item, say) spends more on that conversion than on the rule. To evaluate several rules against the same document, convert it once into a Document:

    use jlens_core::Document;

    let rules = [
        ("minimum total", "$sum(items.(price * qty)) >= 20"),
        ("at most 50 items", "$sum(items.qty) <= 50"),
        ("no pens", "$not('pen' in items.type)"),
    ];
    let compiled: Vec<_> = rules
        .iter()
        .map(|(name, source)| Ok((*name, Expression::compile(source)?)))
        .collect::<Result<_>>()?;

    let order = Document::parse(
        r#"{"items": [{"type": "book", "price": 12.5, "qty": 2},
                      {"type": "pen", "price": 1.5, "qty": 10}]}"#,
    )?;
    let failed: Vec<&str> = compiled
        .iter()
        .filter(|(_, rule)| {
            let outcome = rule.evaluate_document(&order, &Bindings::new(), &Options::default());
            outcome.ok().flatten() != Some(json!(true))
        })
        .map(|(name, _)| *name)
        .collect();
    assert_eq!(failed, ["no pens"]);

Document::parse reads JSON text straight into the internal representation, and Document::new converts a serde_json::Value. A Document is not Send: use it on the thread that created it. Performance has measurements.

Other input representations

An input held in another representation than serde_json (another JSON library, a database row, a host language’s objects) can be read the same way, one level at a time, without being converted first. Implement input::Source for it, and evaluate with Expression::evaluate_source. The documentation of the input module has an example. A level that the source fails to read stops the evaluation with U1004. The WebAssembly package reads JavaScript objects this way.

Explaining results

evaluate_audited records an audit tree of the evaluation: comparisons and their operands, and/or short circuits, conditionals, bindings, filters, and the labels of annotations. explain summarizes it:

    use jlens_core::audit::{AuditOptions, explain};

    let rule = Expression::compile(
        r#"/** @label "Big order" @message "Orders must total at least 100." */
           $sum(items.(price * qty)) >= 100"#,
    )?;
    let order = json!({"items": [{"price": 12.5, "qty": 2}]});
    let (result, audit) = rule.evaluate_audited(
        Some(&order),
        &Bindings::new(),
        &Options::default(),
        &AuditOptions::decisions(),
    );
    assert_eq!(result?, Some(json!(false)));
    let explanation = explain(&audit);
    assert_eq!(explanation.result, "false");
    assert_eq!(
        explanation.lines[0].text,
        "Big order — $sum(items.(price * qty)) >= 100 → false (25 >= 100)"
    );
    // The whole tree serializes to JSON, to store it with the decision.
    let stored = serde_json::to_string(&audit)?;
    assert!(stored.contains(r#""label":"Big order""#));

The audit guide describes the tree, the levels and the bounds that keep audits of large documents small.

Custom tracers

Audits are built by a tracer, an implementation of the Tracer trait that receives evaluation events. Every method has a default that does nothing, so a tracer implements only what it needs:

    use jlens_core::Node;
    use jlens_core::audit::{Tracer, ValueRef};

    /// Counts the comparisons an evaluation makes.
    #[derive(Default)]
    struct Comparisons(usize);

    impl Tracer for Comparisons {
        fn operands(&mut self, _node: &Node, _lhs: ValueRef<'_>, _rhs: ValueRef<'_>) {
            self.0 += 1;
        }
    }

    let expr = Expression::compile("items[price > 10 and qty < 5]")?;
    let mut counter = Comparisons::default();
    expr.evaluate_traced(
        Some(&json!({"items": [{"price": 12, "qty": 1}, {"price": 5, "qty": 9}]})),
        &Bindings::new(),
        &Options::default(),
        &mut counter,
    )?;
    assert_eq!(counter.0, 3); // the second item fails `price > 10`, so `qty < 5` is skipped

jlens provides three tracers:

  • AuditCollector builds audit trees;
  • JsonlTracer writes every event as a line of JSON;
  • TracingTracer, with the tracing feature, emits tracing events.

Without a tracer, the evaluator runs its fast path, so tracing costs nothing when unused.

Tools for editors

parse_recover keeps going after syntax errors. The syntax tree is public, with node ids, spans and annotations:

    use jlens_core::{NodeKind, Severity, parse_recover};

    // Recovering parsing never fails: it returns a partial tree and every problem found.
    let ast = parse_recover("Account.Order[0;].Product");
    let errors: Vec<_> = ast
        .diagnostics
        .iter()
        .filter(|d| d.severity == Severity::Error)
        .map(|d| (d.code.as_str(), d.span.start))
        .collect();
    assert_eq!(errors, [("S0202", 15)]);

    // Walk the tree, for example to list the variables an expression uses (functions are
    // variables too).
    let ast = jlens_core::parse("$a + $b * $sum(items)")?;
    let mut variables = Vec::new();
    ast.root.walk(&mut |node| {
        if let NodeKind::Variable { name } = &node.kind {
            variables.push(name.clone());
        }
    });
    assert_eq!(variables, ["a", "b", "sum"]);

The function catalog describes every built-in function, for completion lists and documentation:

    let catalog = jlens_core::function_catalog();
    let substring = catalog.iter().find(|f| f.name == "substring").unwrap();
    assert_eq!(substring.signature, "<s-nn?:s>");
    assert_eq!(substring.params, ["str", "start", "length"]);

For completion, diagnostics and hover against a schema, use jlens_schema::LanguageService; see the schema guide.

Threads

Expression is Send + Sync. Share one compiled expression between threads, for example behind an Arc:

    use std::sync::Arc;
    use std::thread;

    let expr = Arc::new(Expression::compile("$uppercase(name)")?);
    let workers: Vec<_> = ["ann", "bob"]
        .into_iter()
        .map(|name| {
            let expr = Arc::clone(&expr);
            thread::spawn(move || expr.evaluate(Some(&json!({"name": name})), &Bindings::new()))
        })
        .collect();
    for worker in workers {
        assert!(worker.join().unwrap()?.is_some());
    }

Each evaluation is independent and single-threaded. Values created during an evaluation never leave it, except as the serde_json::Value result.

Features

FeatureEffect
tracingAdds audit::TracingTracer, which reports evaluations as tracing spans and events

The minimum supported Rust version is 1.88. jlens-core compiles to wasm32-unknown-unknown; the WebAssembly guide covers the browser package built on it.