Python

The Python package jlens runs the jlens engine in CPython, through native extension modules built with PyO3 (ADR 0014). It covers evaluation, host functions written in Python, and audit trees with their explanations. Its API is the one of the WebAssembly package, in Python’s style: snake_case names, keyword options, a timeout in seconds, and offsets counted in code points.

pip install jlens

The package is in beta until jlens 1.0: its API may still change in a minor release, as the changelog will say. The installation page lists the platforms it has wheels for.

The snippets in this chapter come from the scripts in crates/jlens-python/tests/guide/, which the package’s tests run.

Compile once, evaluate many times

from jlens import Expression

expr = Expression("$sum(items[type = $kind].(price * qty))")
order = {
    "items": [
        {"type": "book", "price": 12.5, "qty": 2},
        {"type": "pen", "price": 1.5, "qty": 10},
    ]
}
assert expr.evaluate(order, {"kind": "book"}) == 25

An Expression is compiled once and evaluated any number of times, from any thread. Compiling raises a JlensError on syntax errors.

expr = Expression(source)                       # compile once
expr.evaluate(input, bindings, **options)        # Python values in, Python values out
expr.evaluate_json(text, bindings, **options)    # JSON text in, JSON text out
expr.evaluate_document(document, bindings, **options)
expr.source                                      # the source text

• input is the input document: dict, list, tuple, str, int, float, bool or None, nested in any way. None, or no argument, means there is no input document, so $ is undefined. See how the input is read.
• bindings is a dict of variables and functions: {"limit": 10} makes $limit available, and a callable becomes a host function.
• options are keyword arguments: the limits and the clock and seed that make an evaluation reproducible.
• The result is a plain Python value: dict, list, str, int, float, bool or None. A number that is integral and below 10²¹ in magnitude becomes an int, as json.loads would read it from jsonata-js’s output; other numbers become a float.

Undefined and null

JSONata tells an undefined result (the expression produced nothing) from null. evaluate returns None for both:

from jlens import Expression

expr = Expression("customer.email")
assert expr.evaluate({"customer": {}}) is None  # undefined
assert expr.evaluate({"customer": {"email": None}}) is None  # null
# To tell them apart, ask in the expression:
assert Expression("$exists(customer.email)").evaluate({"customer": {}}) is False

evaluate_json keeps them apart: it returns None for an undefined result, and the text "null" for null.

Options

from jlens import Expression, JlensError

clock = Expression("[$now(), $random()]")
first = clock.evaluate(now="2024-01-02T03:04:05Z", seed=7)
assert first[0] == "2024-01-02T03:04:05.000Z"
assert clock.evaluate(now="2024-01-02T03:04:05Z", seed=7) == first

loop = Expression("($f := function($n) { $f($n + 1) }; $f(0))")
try:
    loop.evaluate(timeout=0.05)  # seconds
except JlensError as error:
    assert error.code == "D1012"

• timeout (seconds, a float) stops the evaluation with D1012.
• max_depth, max_string_length, max_array_length and max_result_size bound the evaluation; see limits.
• now fixes $now() and $millis(): milliseconds since the epoch, a datetime, or an ISO 8601 timestamp. A datetime without a time zone is local time, as its timestamp() reads it.
• seed (an int) fixes $random() and $shuffle().

An unknown option raises a TypeError, and a value out of range a ValueError.

How the input is read

evaluate reads its input during the evaluation, one dict or list at a time, when the expression first reaches it, as the WebAssembly package does (ADR 0010). A path such as customer.name reads two small dictionaries, however large the rest of the document is.

Anything else, such as a set, a date, a key that is not a str, or a cycle, has no JSON form. Two consequences follow from reading the input as the expression goes:

• Errors appear when their part is read. A value that cannot be read stops the evaluation with U1004, with the message json.dumps would give, when the expression reads it. A path step reads one field of a dictionary; other operations (*, $keys, returning the dictionary) read the whole dictionary or list, and a key that is not a str fails then. An expression that never reaches the value succeeds (below).
• The input must not change during the evaluation. A host function, or another thread, that modifies the input can change what the rest of the evaluation reads. In free-threaded Python, that gives a result mixing both states, or U1004, never a crash. Copy the input first if something must modify it.

from jlens import Expression, JlensError

order = {"customer": {"name": "Acme"}, "log": {"seen": {1, 2}}}  # a set is not JSON
assert Expression("customer.name").evaluate(order) == "Acme"  # `log` is never read
try:
    Expression("log.seen").evaluate(order)
except JlensError as error:
    assert error.code == "U1004"

Evaluations never modify their input. A lone surrogate in a str, which UTF-8 cannot hold, becomes U+FFFD.

JSON text

evaluate_json takes the input as JSON text and returns the result as JSON text. For a document you already have as text, such as an HTTP response body, it avoids building Python objects only to read them. While it parses, it skips the fields the expression does not read, checking them as JSON syntax only:

from jlens import Expression

expr = Expression("items.sku")
assert expr.evaluate_json('{"items": [{"sku": "A1"}, {"sku": "B2"}]}') == '["A1","B2"]'
assert expr.evaluate_json("{}") is None  # undefined

Invalid JSON raises a ValueError.

Reusing a document

evaluate reads the parts of the input that the expression reaches, on every call. To evaluate several expressions against one document (a set of rules, say), or one expression several times, convert the document once:

from jlens import Document

rules = [Expression("$count(items) > 1"), Expression("$sum(items.(price * qty)) > 25")]
document = Document(order)  # or Document.from_json(text)
assert [rule.evaluate_document(document) for rule in rules] == [True, True]

Building a Document converts the whole value at once, with the rules above, and raises a ValueError or a TypeError for what it cannot convert. Evaluations never modify a document.

Threads

Expression and Document objects can be shared between threads. evaluate_json and evaluate_document let other Python threads run while they evaluate: they release the GIL, and attach again only to call a host function written in Python. evaluate reads Python objects as it goes, so it holds the GIL throughout.

from concurrent.futures import ThreadPoolExecutor

from jlens import Document, Expression

expr = Expression("$sum(items)")
documents = [Document.from_json(f'{{"items": [{i}, {i}]}}') for i in range(100)]
with ThreadPoolExecutor(8) as pool:
    totals = list(pool.map(expr.evaluate_document, documents))
assert totals[:3] == [0, 2, 4]

The package supports free-threaded Python (3.14t and later), where the threads of a pool evaluate in parallel. A Document used by several threads is converted once more on each thread, the first time that thread evaluates against it.

On Linux distributions built on musl, such as Alpine, give threads a larger stack before you create them, if they evaluate expressions that call Python code, such as a host function:

threading.stack_size(8 << 20)  # 8 MiB, the default with glibc
with ThreadPoolExecutor(4) as pool:
    taxes = list(pool.map(lambda p: expr.evaluate({"price": p}), [8, 12]))

musl starts threads with a 128 KiB stack, so jlens evaluates on a larger stack of its own. CPython 3.14 and later cannot ask musl where a thread’s stack ends: they assume it is 4 MB long. When the stack jlens allocated happens to lie at that assumed end, the Python code called from it raises RecursionError or aborts the process (Unrecoverable stack overflow). This is rare, depends on where memory is mapped, and does not affect the main thread, whose stack is larger. With a stack larger than 4 MB, a thread evaluates on its own stack.

Host functions

Python callables, in the bindings or registered on an expression, are called from the expression like built-in functions:

def shipping(weight, region):
    return weight * rates[region]


expr = Expression("$shipping($sum(items.weight), region)")
assert expr.evaluate(order, {"shipping": shipping}) == 10

# As in jsonata-js: available to every evaluation, with an optional signature.
expr.register_function("shipping", shipping, "<ns:n>")
assert expr.evaluate(order) == 10

• Arguments and the result are converted like the input. None stands for undefined, in both directions: a missing argument arrives as None, and a function that returns None gives an undefined result. A function argument arrives as a jlens.Function (below).
• A signature (register_function’s third argument) checks the arguments before each call, with the errors of the built-in functions (T0410, T0412). A parameter with - receives the context value when the argument is missing: <n-:n> makes price.$double() work. An invalid signature raises a JlensError.
• The number of positional parameters before the first with a default value tells higher-order functions how many arguments to pass: $map passes the index only to functions with two or more. A callable whose signature inspect cannot read counts as having none.
• Any callable works: a function, a lambda, a method, a class, or a built-in such as len.
• Asynchronous functions are not supported. A coroutine function fails with U1002. Fetch what the expression needs before evaluating it.
• Precedence: a function in the bindings replaces a registered function of the same name, and both replace a built-in function.

Exceptions

An exception raised by a host function is raised again by the evaluation as is, the same object, with position and token attributes added when it has none:

from jlens import Expression


class RateError(Exception):
    code = "RATE01"  # how evaluate_with_audit reports it


def rate(region):
    raise RateError(f"no rate for {region}")


expr = Expression("$rate(region) * 2")
try:
    expr.evaluate({"region": "west"}, {"rate": rate})
except RateError as error:
    assert (error.position, error.token) == (6, "rate")  # added by jlens

evaluate_with_audit reports it in its error, with its code attribute (or the name of its type) and its text. Inside $eval, it becomes the error of $eval (D3121).

Functions as arguments

A host function can call a JSONata function it receives, such as a step to retry or a fallback: a lambda, a built-in function, a partial application, a chain or another host function. It arrives as a jlens.Function:

def retry(step, times):
    for attempt in range(1, times + 1):
        try:
            return step(attempt)
        except Exception:
            if attempt == times:
                raise


expr = Expression("$retry(function($i) { $lookup(sku, $i) }, 3)")
assert expr.evaluate({"sku": "A1"}, {"retry": retry, "lookup": lookup}) == 7

• The call is part of the evaluation, as when $map calls its function. The context value is null, and a lambda keeps the variables of its definition. The call counts as a level of depth, so a recursion through a host function ends with D1011; it checks the timeout; its result is limited by max_result_size.
• Arguments and the result are converted as for a host function. Its arity attribute is the number of parameters the function declares.
• Errors are raised as a JlensError with their codes, or as the exception a host function raised. The host function can catch them, or let them through, which ends the evaluation with that error.
• It is valid until the host function returns. Called later, or from another thread, it raises a JlensError with code U1005.

Errors

Errors are jlens.JlensError exceptions, a subclass of Exception. They carry the JSONata error fields; every offset counts code points, the unit of str, so source[error.start:error.end] is the part of the source the error is about:

try:
    Expression("Account.Order[0;]")
except JlensError as error:
    assert error.code == "S0202"
    assert str(error) == 'Expected "]", got ";"'
    assert error.position == 16  # where it was detected, as in jsonata-js
    assert (error.start, error.end) == (15, 16)  # the offending part of the source
    assert error.token == ";"

Evaluation errors carry value too, when there is one (for T2001, the value that is not a number):

try:
    Expression("customer.name + 1").evaluate({"customer": {"name": "Acme"}})
except JlensError as error:
    assert error.code == "T2001"
    assert error.value == "Acme"

str(error) is the message. Match on code, never on the message. Arguments of the wrong type, such as bindings that are not a dict, raise a TypeError instead, and a value that cannot be converted, such as a cyclic binding or invalid JSON text, a ValueError. The error reference lists every code.

The fields an expression reads

reads tells which fields of its input and of each binding an expression may read, without evaluating it (ADR 0016): an application can fetch only those from a database, or know which rules depend on a field. union_reads combines the answers of several expressions.

expr = Expression("$sum(items[type = $kind].(price * quantity)) * $rate(country)")
reads = expr.reads({"rate": lambda country: RATES[country]})
assert reads == {
    "input": {"country": True, "items": {"price": True, "quantity": True, "type": True}},
    "bindings": {"kind": True, "rate": True},
}

union = union_reads([reads, Expression("customer.name").reads()])
assert union["input"]["customer"] == {"name": True}

• The answer is an upper bound. True reads the value whole; a dict reads only the fields it lists, of an object, or of each item of a list; an empty dict needs the value but none of its fields, as $count does. On its input and bindings cut down that way, an expression gives the same result, or fails with the same error code. Where the analysis cannot follow a value, under * or **, in a constructed object or in the result, it reads the value whole.
• Pass the bindings the expression will be evaluated with. A Python function does not receive the context of its call, so a call to one in the bindings, or registered on the expression, reads only its arguments, and the context a - parameter may take. A call to a function the analysis does not know may read its context: the whole input, at the top of an expression.
• "everyBinding": True says the expression may read any binding whole, as $eval can: keep them all. The keys are those of the JSON form, which the other packages and the CLI share.
• A binding named after a built-in function shadows it during the evaluation, but the analysis takes the name as the built-in’s: such an application cannot cut its input by it.

Document.from_json(text, read) parses JSON text cut by a Read, the "input" of reads() or of union_reads(). It skips the fields the Read leaves out, which on a wide document is several times faster (Performance), and the document serves each expression that went into the union as the whole one would:

rules = [Expression("total > 100"), Expression("customer.tier = 'gold'")]
document = Document.from_json(text, union_reads([rule.reads() for rule in rules])["input"])
assert [rule.evaluate_document(document) for rule in rules] == [True, True]

The fields skipped are checked as JSON syntax only: a number too large for a float, which a full parse rejects, passes there.

Explaining results

evaluate_with_audit evaluates and records how the result was reached; see the audit guide. Evaluation errors are reported in the result, not raised:

from jlens import Expression

expr = Expression('/** @label "Big order" */ $sum(items.(price * qty)) > $limit')
audited = expr.evaluate_with_audit(
    {"items": [{"price": 2.5, "qty": 4}, {"price": 1, "qty": 3}]},
    {"limit": 20},
    level="decisions",
    limits={"max_nodes": 1000},
)
assert audited["result"] is False
assert audited["error"] is None
assert audited["explanation_text"] == (
    "result: false\n✘ Big order — $sum(items.(price * qty)) > $limit → false (13 > 20)\n"
)
root = audited["audit"]["root"]
assert (root["label"], root["operands"], root["outcome"]) == ("Big order", [13, 20], "false")

The options are those of evaluate, plus two more:

• level is "off", "decisions" (the default) or "full".
• limits sets the capture limits: max_items, max_string, max_depth, max_rejected, max_nodes and max_source.

The result is a dict (typed as jlens.AuditResult) with result, error, audit, explanation and explanation_text. audit is the audit tree in the JSON format of jlens eval --audit json, with its camelCase field names, and its spans count code points. explanation["lines"] holds structured lines (depth, passed, text, message, node) for rendering in a user interface. The input is converted whole first, as for a Document.

Limits

The size of the result counts a value each time it appears, as the JSON text does, and also limits the arguments passed to a host function. The stack grows as needed, so the depth limit is that of the native targets; see the evaluator chapter. There is no cancellation from another thread: use timeout. The performance chapter compares the package’s speed and its behaviour at these limits with the other JSONata packages for Python.

Typing

The package ships type stubs and py.typed. Expression, Document, Function and JlensError are typed, and so are the values evaluate_with_audit takes and returns: AuditResult, ErrorInfo and CaptureLimits, which are TypedDicts, and those reads and union_reads return: Reads, a TypedDict, and Read. Results of evaluate are Any.

What the package leaves out

The language service and schema inference are not in the Python package: editors reach them through the language server, and jlens schema infer infers a schema from the command line. They can be added when a Python application asks for them.

Building from source

The source distribution builds on any platform with Rust 1.88 or later; pip builds it when no wheel fits, such as for free-threaded Python 3.14. From a checkout, with maturin in a virtual environment:

cd crates/jlens-python
pip install maturin pytest
maturin develop                          # builds the package into the active environment
python -m pytest -p no:cacheprovider tests