Execution Context

An ExecutionContext is the environment the engine evaluates a query in. It bundles four things:

• The query library: the imports that query strings can use.
• The query compiler: it turns query strings into code.
• The query scope: the variables that query strings can use.
• The update graph: it schedules when ticking tables refresh.

Most of the time you never see it. When a Deephaven script session starts, it creates a default context and runs every query you enter in the console inside it. That default context is systemic: the system supplies it, not your code. A systemic context is unrelated to systemic object marking.

You need a context of your own in two situations: when a table operation runs on a thread you start, and when an operation does deferred work that the engine runs later. This page covers both, then shows when and how to create additional contexts.

Contexts are also a tool in their own right. Because each context can carry its own query scope and update graph, you can use contexts to keep independent real-time workloads from stepping on each other's variables or update cycles.

When an ExecutionContext is needed

The script session supplies a context for the code you run in the console. You must supply one yourself when:

• A table operation runs in a thread other than the script session's thread.
• A table operation does deferred work: the engine runs part of it later, outside the script session, such as when a new partition arrives in a partitioned table.

Table operations in a separate thread

Each thread has its own current context. A thread you start has no usable context until you open one in it.

Take, for instance, the following code, which attempts to use a TablePublisher to write data to a blink table in a separate thread once per second:

The table operation in the new thread fails, and the server log shows the following error:

This occurs because the addTables closure runs in a separate thread, and the table operations it calls (emptyTable and update) don't run inside an open ExecutionContext.

A simple fix lets the code run. Get the script session's context with getContext, then call its open method in a try-with-resources block around the table operation. open returns a SafeCloseable that restores the previous context when the block ends:

The table blink table now receives a batch of 5 to 10 random rows about once per second, five times. Several threads can use the same execution context, so any other thread you start can open defaultCtx the same way.

Table operations that do deferred work

A partitioned table transform can do deferred work: if a new partition arrives later, the engine re-evaluates the transform closure for that partition.

Operations that record a context for later work don't record a systemic one. Because the script session's context is systemic, the transform overload that takes no ExecutionContext argument doesn't record it for that deferred work. The closure then runs without access to the query library, query scope, or query compiler unless it opens an execution context itself.

A timeTable starts empty, so in the following code every partition arrives after transform is set up, and each call to maxDate is deferred work:

• source is a time table that adds a row every second. Its Key column is A for the first five rows and B after that.
• partitionBy splits source into a partitioned table with one partition per key.
• transform calls maxDate on each partition. maxDate opens the script session's context, defaultCtx, around its table operations.

A usable context can come from any of three places:

• The closure opens one, as maxDate does above.
• You pass one to transform. The transform(ExecutionContext, UnaryOperator, boolean, Dependency...) overload runs the closure in the context you pass. Its expectRefreshingResults flag must be true when the underlying table of the partitioned table is static but the closure can return refreshing tables. The last example on the transform reference page shows this overload.
• You open a non-systemic context around the transform call. makeExecutionContext(false) returns a copy of the current context that isn't systemic. Its argument says whether the new context is systemic. It shares the current query scope, query library, query compiler, and update graph. The transform overload that takes no ExecutionContext argument records it, so maxDate runs in it without its own try-with-resources block:

When and how to create additional execution contexts

For most applications where a single user runs Deephaven, the systemic execution context is enough. Each ExecutionContext can have its own update graph, query library, and query scope, which lets you compartmentalize units of code so they work independently of one another. That matters in these more complex use cases:

• Freezing query-scope variables: Some work must keep the values its variables had when it was set up. For example, transforms set up in a loop each need their own iteration's value, and a thread might need to be shielded from changes that other threads make to the query scope. Use captureQueryScopeVars, described in Create a context from the current one.
• Isolating update processing: A secondary update graph runs its own update cycles, so an expensive operation on it doesn't delay the update cycles of the main graph. Separate update graphs refresh independently, so their update cycles can run in parallel on different cores. Build a context with its own update graph, described in Create an execution context from scratch.
• Separating workloads: When multiple users share a Deephaven server, or a single user wants to keep each workload's variables, imports, and update processing separate from the others, each workload can run in a context with its own query scope, query library, and update graph. Build each context from scratch, described in Create an execution context from scratch.

Create a context from the current one

The thread example and the first partitioned table transform example above reuse the script session's own context, which getContext returns. You can create additional execution contexts, which aren't systemic, with makeExecutionContext(false) or the ExecutionContext builder.

• makeExecutionContext(false) returns a context that shares the current query scope, query library, query compiler, and update graph, as in the deferred-work example.
• A context built with newBuilder and captureQueryScopeVars has its own query scope (shown below).

To give the new execution context its own query scope that holds only selected variables from the current one, build it with captureQueryScopeVars. The new query scope gets those variables with their current values, and later changes to them aren't visible in the new context.

newBuilder starts from the current thread's update graph, operation initializer, and AuthContext, with no query library, query scope, or query compiler. The following code adds them:

• captureQueryCompiler reuses the current context's query compiler.
• captureQueryLibrary reuses the current context's query library.
• captureQueryScopeVars copies value1, value2, and value3 into a new query scope.

The code then sets value1 to 100 and runs the same update on an emptyTable inside and outside the new context:

inside shows 1, the value value1 had when the context was created, and outside shows 100.

Neither form gives the context its own update graph. For that, create an execution context from scratch.

Create an execution context from scratch

Create an ExecutionContext from scratch, instead of capturing components from an existing context, when:

• You embed the Deephaven engine in a standalone Java application, where no script session provides a context.
• You need a context with its own update graph, such as a separate PeriodicUpdateGraph or an EventDrivenUpdateGraph that refreshes only on request.

The following example demonstrates how to build an ExecutionContext from scratch using the builder pattern:

A context also has an operation initializer, which controls whether operations parallelize their initial work. It also carries the AuthContext that was current when the context was built. Deephaven records that AuthContext with the work done in that context, for example in the performance logs.

The builder lets you specify each component:

• Query library: A new query library with only the default imports, via newQueryLibrary.
• Query scope: A new, empty query scope, via newQueryScope.
• Operation initializer: Whether and how the engine parallelizes the initial work of table operations in this context, via setOperationInitializer. The example passes OperationInitializer.NON_PARALLELIZABLE, which runs that work on the calling thread.
• Update graph: A custom update graph, such as a PeriodicUpdateGraph or an EventDrivenUpdateGraph, via setUpdateGraph.
• Query compiler: The compiler that turns query strings into code, via setQueryCompiler. QueryCompilerImpl.create also accepts an optional directory to add to the compiler's classpath and an optional parent class loader.

As described in Create a context from the current one, newBuilder starts from the current thread's update graph, operation initializer, and AuthContext. The example replaces the update graph and operation initializer and keeps the AuthContext.

A new PeriodicUpdateGraph doesn't refresh until you call its start method, as the example does.

Open the new context around the work that should run in it:

Refreshing tables that belong to different update graphs can't be used together in one table operation. For example, a join between a table that ticks on MyCustomGraph and a table that ticks on the server's main update graph fails.

To refresh only on demand, build an EventDrivenUpdateGraph instead of the PeriodicUpdateGraph and pass it to setUpdateGraph in the builder chain above. An event-driven graph has no start method, so leave out the start call. Instead, it runs one update cycle each time you call its requestRefresh method:

Related documentation

• Live DAG
• Table types
• Create an empty table
• Partitioned tables
• Query scope
• Time tables
• CumMax
• dropColumns
• emptyTable
• partitionBy
• TablePublisher
• timeTable
• transform
• update
• updateBy
• updateView
• ExecutionContext Javadoc
• PartitionedTable Javadoc
• PeriodicUpdateGraph Javadoc
• EventDrivenUpdateGraph Javadoc