Groovy closures in queries

You can use Groovy closures in Deephaven queries. This guide explains:

  • What a closure is
  • How to call one in a query string
  • When to use a built-in function or a direct Java method call instead
  • How to pass tables to a closure
  • How to import closures from Groovy files on the classpath

A closure is Groovy's way to write a user-defined function. It lets you package a calculation once and reuse it. In a query string, Deephaven calls the closure for every row. In a script, you can call it on whole tables.

On a ticking table, the column stays current as new data arrives. In each update cycle, the engine calls the closure again for added rows and for rows whose input columns changed.

In Groovy, you write a closure inside curly braces ({}). The following closure, f, adds two integer values:

A closure can take any number of parameters but returns only one value. To return several values, wrap them in a single object, such as a list or an array.

Call a closure in a query string

You can call closures in query strings. A query string can call f, the closure we defined above, because f is assigned at the top level of the script without def. That puts f in the query scope. A method declared with def myMethod(int a) { ... } is not in the query scope, so assign a closure to a variable instead.

The query below calls f twice. It computes Z with a plain call and ZInt with an (int) cast:

A closure called in a query string returns Object, whatever types the closure declares. The reason is how the engine makes the call:

  • In a query string, f(X, Y) is shorthand for f.call(X, Y).
  • call is the generic method every closure has, and it returns Object.

In the example above, sourceMeta shows that Z is an Object column, even though f adds two integers. The (int) cast makes ZInt an int column.

Cast the result whenever you need a typed column, for example to do arithmetic on it or aggregate it. For more about casts, see Casting.

Tip

Declare the expected argument types when defining a closure that query strings call. This documents the input the closure expects. For instance, f declares int a, int b rather than a, b.

Prefer built-in functions when available

One thing to consider before you write a closure to call from a query string is:

Can I perform the same operation using a built-in query language function?

For instance, the following closure calls Math.sin from Java's java.lang.Math class. The first row of X holds a null value, NULL_DOUBLE:

Deephaven has a built-in sin function that you can call from query strings with no imports. It handles null values. For example, sin(NULL_DOUBLE) returns NULL_DOUBLE. By contrast, Math.sin returns -0.004961954789184062 for the same input, which is not what we're looking for.

The following query adds a column that uses the built-in sin. In the first row, YBuiltIn is null, while Y holds the value that Math.sin returned:

Note

Closures used in query strings should perform null checks on their arguments, for example, with isNull. A closure that doesn't check for nulls treats a Deephaven null as an ordinary value and returns a meaningless result, as the Math.sin example above shows.

The following closure checks for nulls first and returns NULL_DOUBLE for a null input:

In the first row, YSafe is null, like YBuiltIn.

Familiarity with Deephaven's built-in functions can speed up your workflow. You can call a large number of them from query strings without adding imports or changing the classpath. For more information and a complete list of what's available, see auto-imported functions in Deephaven and built-in query language functions.

Call Java methods without a closure

When no built-in function fits, a query string can still call a public Java method directly, without wrapping it in a closure.

For example, the sine closure above only wraps Math.sin, so a query string could call Math.sin(X) directly. The direct call has one advantage and one limit:

  • The result is a double column with no cast, because the engine sees the declared return type of Math.sin.
  • Null inputs still give the same meaningless value, so the built-in sin is still the better choice there.

How a query string names the class depends on whether it is auto-imported:

  • For an auto-imported class, such as Math from the java.lang package, use the class's simple name.
  • For any other class, use its fully qualified name, or first add it to the query library.

Java objects and classes in query strings describes which classes are auto-imported and shows how to add others.

Neither java.net.URLEncoder nor java.nio.charset.StandardCharsets is auto-imported. The following example therefore writes out both class names in full:

Both rows of Encoded hold Deephaven+docs, the URL-encoded form of the string.

To call methods from a Java library that Deephaven does not include, add its JAR to the classpath first. For details, see Install and use Java packages.

Pass tables to a closure

Sometimes you need to apply the same operations to many tables, for example the same sums and averages grouped by a key column. Groovy closures can take tables as input and return tables as output, so you can define those operations once and call the closure on each table. Some Deephaven operations, such as the partitioned table operations described at the end of this section, require this pattern.

The following example uses the closure doAgg to perform multiple aggregations on a table. The same closure can be called on any table that has a Letter column and numeric X and Y columns.

doAgg runs the same aggregations on source and source2, so result and result2 have the same columns.

Note

Table operations return new tables rather than modifying the source table. A closure that applies operations to a table has no effect unless it returns the resulting table. A Groovy closure returns the value of its last expression, so doAgg returns the result of aggBy without a return statement.

Partitioned table operations such as transform and partitionedTransform take a closure that receives tables and returns a table. When that closure uses operations that compile formulas or use the query scope, it must run them inside an execution context. The Transform section of the partitioned tables guide shows how.

Import closures from Groovy files

It's common in Groovy to keep custom closures in separate Groovy files, organized in package folders. To use them in Deephaven, put the top folder of that package tree on the server's classpath. You do that with the EXTRA_CLASSPATH environment variable in your docker-compose.yml file.

The classpath entry must be a path inside the container, so the compose file below mounts the host's ./data directory at /data. For more about mounts, see Docker bind mounts.

For example, say we want to use closures from a class in /data/groovyscripts/com/example/MyClass.groovy:

A Groovy file loaded from the classpath doesn't get the imports that the Deephaven console adds automatically. Import every Deephaven class or static method the file uses, as the doAgg example does with AggSum and AggAvg.

To make that class available:

The resulting docker-compose.yml looks like this:

Now, from the IDE, we can import the class that contains the closure we want to call:

The import statement makes MyClass available to your Groovy script, but not to query strings, because the query engine compiles them separately.

To call one of its closures in a query string, assign the closure to a top-level variable without def. The variable is then in the query scope, as f is in Call a closure in a query string:

To use MyClass itself in a query string, rather than one of its closures, see Groovy classes and objects in query strings.