Handle PyObjects in tables

This guide shows how to recognize org.jpy.PyObject columns in your tables, what limits them, when to use them, and how to avoid them. The rest of this guide calls them PyObject columns.

This data type most commonly arises from Python functions called in query strings without type hints or type casts. Two other sources also produce PyObject values:

Methods called through a class or object produce java.lang.Object columns instead. These columns have the same limits as PyObject columns and also need a type cast.

Because a PyObject can hold any Python object, the engine can infer very little about it. As a result, PyObject columns support fewer operations and are less performant than Java primitive columns. They are still useful in some situations. See When to use PyObject columns.

What is a PyObject?

A PyObject is a Java wrapper around an arbitrary Python object. It comes from jpy, the bidirectional Python-Java bridge that Deephaven uses to call Python from query strings. It is a flexible data type because it can hold any Python object. This flexibility comes at the cost of compatibility and speed.

Consider the following example, which calls three Python functions in query strings without any type hints. They return an int, list, and dict, respectively.

Recognize a PyObject column

source_meta holds the meta_table of source. Its DataType column shows org.jpy.PyObject for all three columns. Because all three functions lack return type hints, the engine has no Java type to assign, so it stores each result as a PyObject.

To recognize a PyObject column in any table, check its meta_table for org.jpy.PyObject in the DataType column.

Without a return type hint or a cast, the result column isn't typed. The column type depends on the function's arguments:

  • If every argument is a column, a variable, or a constant, as in f(X), the column is a PyObject column.
  • If an argument is an expression, as in f(X + 1), the column is a java.lang.Object column.

A java.lang.Object column has the same limits as a PyObject column and takes the same remedies.

Tip

Add type hints to Python functions that you call in query strings. See How to avoid PyObject columns for this and other remedies.

Limitations of PyObject columns

Compatibility

Many of Deephaven's built-in query language functions do not support PyObject columns. Built-in functions that take java.lang.Object as input support PyObject columns because PyObject extends Object.

The following code calls the built-in sin function on a PyObject column, and the query fails:

The stack trace contains the following line:

No sin overload accepts a PyObject argument. To call sin on these values, cast them in the formula, as in Y = sin((double)X). You can also add a type hint to the function. See How to avoid PyObject columns.

Performance

PyObject columns are less performant than Java primitive columns for two reasons:

  • Each cell is a Java object that references a separate Python object. These columns use more memory than primitive columns, which store their values in primitive arrays.
  • Almost every operation on these values, such as passing them to a Python function or casting them, crosses the Python-Java boundary.

When to use PyObject columns

Despite these limitations, PyObject columns are useful when:

The following example shows all three cases. get_player returns a dict, and a second Python function named best_score reads that dict. get_reading returns an int, a str, or a float, depending on the row:

source_meta shows that Player and Reading are PyObject columns. BestScore and ReadingType are long and java.lang.String columns, respectively, because best_score and type_name have return type hints. The ReadingType column shows that the Reading column holds a different Python type in each row.

How to avoid PyObject columns

For better performance and type safety, minimize the use of PyObject columns in your tables. Keep them only for the cases listed in When to use PyObject columns.

Use whichever of these remedies matches where the PyObject values come from:

  • Add type hints to your functions: Deephaven uses the return type hint to convert return values to the matching Java type. A hint on a tuple or list works too when it names a supported element type. For example, tuple[int, int, int] produces a Java long[] column. See Array return values and Multiple return values.

    Deephaven has no Java type for the following hints, so they still produce a PyObject column:

    • -> dict
    • -> object
    • -> MyClass, where MyClass is a custom class
    • -> list or -> tuple with no element type
    • -> list[dict], -> list[MyClass], or another sequence of dict or custom-class elements
  • Use explicit type casts: Cast the value with a Java-style cast, such as (int) or (String), when a type hint can't help:

    • The value is an attribute or method of a Python class or object. Deephaven ignores type hints on these.
    • The function is one you can't edit.
    • The value is a NumPy scalar variable, such as a numpy.int64. You can also convert it with Python's int, float, or bool before you use it.
  • Convert tuple and NumPy array variables to Java arrays: Use jpy.array or deephaven.dtypes.array to convert the variable before you use it in a query string. See Sequences and Work with arrays.

The following code block uses the first two remedies: a type hint on my_func and a (double) cast on the result of my_func_without_type_hint. The resulting Y and Z columns are both double columns: