Systemic Object Marking
In Deephaven, a systemic object is an object whose failure the engine reports to the server, in addition to logging it. The engine logs every failure, but it reports only failures of systemic objects.
What happens after a systemic failure depends on the server:
- Deephaven Community Core writes an extra
Error in table updateentry to the server log and keeps running. - Deephaven Core+ (Enterprise) terminates the worker.
Reporting is separate from failure propagation. When a table fails, the tables that depend on it also fail, whether or not they are systemic.
Systemic object marking lets you decide which objects are systemic:
- When marking is disabled (the default), the engine treats every table and single-table listener as systemic. Listeners registered with
merged_listenare the exception. - When marking is enabled, a new table, or a table listener that listens to a single table, takes its status from the thread that creates it. It is systemic if that thread is marked systemic, and non-systemic otherwise.
- Deephaven marks its own update graph threads systemic. Other threads, including the one that runs your console code, are non-systemic unless you mark them.
When the engine creates tables later for an operation you ran, such as new constituents of a partitioned table, they take the status of the thread that ran the operation, not that of the update graph thread.
Systemic object marking is most useful on Core+, where it lets a worker keep running when a non-critical table fails, such as one created while exploring data in the console. On Community Core, it decides only whether a failure also produces that Error in table update log entry.
Systemic objects are unrelated to the systemic execution context that a script session owns.
This guide first shows what happens when a table fails. It then explains how to enable marking, how to control which objects are systemic with the deephaven.systemic_obj_tracker module or by marking an existing table, and how to set the systemic status of a script you run from the Python client.
What happens when a table fails
The difference between systemic and non-systemic objects is most visible when a ticking table fails. In this example, the formula Z = Y.toString() throws a NullPointerException once X exceeds 10 and Y becomes null:
With marking enabled, console code runs on a non-systemic thread, so time_bomb is non-systemic unless you mark it systemic. With marking disabled, time_bomb is systemic.
After about 11 seconds, the table fails. Whether or not time_bomb is systemic, it and the tables that depend on it fail, and the engine logs the error. The difference is whether the engine also reports the error:
- If
time_bombis systemic, the engine reports the error. A Community Core server writes theError in table updatelog entry and keeps running, and a Core+ worker terminates. - If
time_bombis non-systemic, the engine doesn't report its failure. A Community Core server writes noError in table updatelog entry, and a Core+ worker keeps running.
A non-systemic table's failure can still be reported through an object that depends on it. If a systemic table or listener depends on time_bomb, that object also fails, the engine reports its failure, and a Core+ worker terminates.
Enable systemic object marking
Systemic object marking is disabled by default, and you can't turn it on or off while the server is running. To enable it, set the SystemicObjectTracker.enabled property to true when you start the server, for example with the JVM option:
See how to set configuration properties for the ways to pass a JVM option or set a property in a configuration file.
Check whether systemic object marking is enabled with is_systemic_object_marking_enabled:
Manage thread systemic status
Because systemic status comes from the creating thread, you control which tables and table listeners are systemic by setting the status of the thread that creates them. The exception is a listener registered with merged_listen. It is never systemic, whether or not marking is enabled, so the engine never reports its failures.
Use is_systemic to check the current thread's status and set_systemic to change it. If systemic object marking is not enabled, is_systemic always returns False, even though every table is then systemic, and set_systemic raises a DHError.
This example checks is_systemic_object_marking_enabled first. When marking is enabled, it marks the current thread systemic and prints the thread's new status. Otherwise, it prints a message:
The new status lasts until you change it again, so console code you run later on the same thread also uses it. The server runs console code from every session on the same thread, so the new status also applies to code that other sessions and clients run later.
To change the status only for a few lines, use a context manager instead.
Set systemic status for a block of code
The deephaven.systemic_obj_tracker module provides two context managers that set the current thread's systemic status for the code inside a with block:
systemic_object_markingmakes objects created in the block systemic.no_systemic_object_markingmakes objects created in the block non-systemic.
On exit, each context manager restores the thread's previous status.
Like set_systemic, both context managers raise a DHError if systemic object marking is not enabled.
Mark an existing table systemic
To make an existing table systemic, call its with_attributes method with {"SystemicTable": True}. It returns a new, systemic table that shares the original's data. The original table stays as it was.
In this example, systemic_t is systemic, and t keeps the status it had when it was created:
Tables you create from systemic_t don't inherit its mark. They take their status from the thread that creates them.
Systemic scripts from the Python client
When you run a script on the server from the Python client, the systemic argument of Session.run_script sets the systemic status of the thread that runs it:
Truemakes the objects the script creates systemic.Falsemakes them non-systemic.None, the default, runs the script without changing the thread's status. When marking is enabled, its objects are non-systemic unless earlier console code marked that thread systemic.
When marking is disabled, the server ignores the argument, and every table and single-table listener the script creates is systemic.