Data types in Deephaven and Python

For performance reasons, the Deephaven engine is implemented in Java. As such, Deephaven tables use Java data types for columns, both Java primitive types and Java objects.

Deephaven Python queries combine the power of Python with Deephaven's tables. Table columns hold Java types, so every value a query puts in a column has a Java type. Knowing how Python and Java types map to each other helps you write queries that are fast and produce the column types you expect.

The Java data types section maps Python and NumPy types to the Java types that columns hold. Data type conversions then shows how to make sure a column gets the type you want with type hints or type casts.

Java data types

Primitive types

In Python, everything is an object, including scalar types like int and float. A Python int carries methods and attributes that a Java primitive value does not. Java, by contrast, has primitive types as well as objects.

Java primitive types map directly to hardware representations and have fixed sizes, unlike Python's arbitrary-precision int. Deephaven stores primitive values directly rather than as objects, so they are faster and use less memory. That speed and memory saving is the main reason to use them for table columns.

The following table shows the mapping between Java primitive types, Java primitive type sizes, and the Python and NumPy types that map to them in a function's type hints:

Java primitive typeJava primitive type sizePython typeNumPy type
byte1 byteN/Anp.int8
short2 bytesN/Anp.int16
int4 bytesN/Anp.int32
long8 bytesintnp.int64
float4 bytesN/Anp.float32
double8 bytesfloatnp.float64
char2 bytesN/Anp.uint16

Deephaven tables have no primitive boolean column type. True/false values go in java.lang.Boolean columns, which Deephaven stores as one byte per value:

  • A formula that returns a Java boolean produces a java.lang.Boolean column.
  • In a function's type hints, Python bool and np.bool_ map to java.lang.Boolean, as the object types table shows.

Variables used directly in a query string follow different rules from type hints. For example, a Python int variable becomes the smallest Java integer type that holds its value. Most NumPy scalar variables become PyObjects. Python variables in query strings explains how each kind of Python variable converts.

Object types

Object types are Java classes whose values are objects rather than raw values. They consume more memory and are generally slower than their primitive counterparts.

Some object types are commonly used despite the overhead. For instance, java.lang.String is the best way to store text in tables.

The following table shows the Python and NumPy types that map to common Java object types in a function's type hints. The last row, org.jpy.PyObject, is the Java wrapper for a Python value that has no Java mapping:

Java object typePython typeNumPy type
java.lang.Stringstrnp.str_
java.lang.Booleanboolnp.bool_
java.time.Instantdatetime.datetimenp.datetime64
ArraySequence[T], such as Sequence[int]np.ndarray with an element type, such as npt.NDArray[np.int64]
org.jpy.PyObjectobjectnp.object_

An array type hint's element type follows the same mapping, so Sequence[int] and npt.NDArray[np.int64] both produce a long[] column.

For every type Deephaven defines in Python, see deephaven.dtypes.

Array types

Deephaven tables commonly hold two kinds of array column:

Arrays vs vectors compares the two.

A Python list used in a query string without type information becomes an org.jpy.PyListWrapper column, not a Java array. Tuples, NumPy arrays, and other sequences that are not lists or strings become PyObjects.

To get a Java array column, convert the sequence first with jpy.array or deephaven.dtypes.array. Work with arrays shows how.

PyObjects

org.jpy.PyObject is the Java wrapper that the jpy Java-Python bridge puts around arbitrary Python objects.

A PyObject column typically appears when a query string uses one of these Python values:

  • Most NumPy scalar variables, such as np.int64, np.float32, or np.bool_. The exceptions are np.float64 and np.str_, which become double and java.lang.String values.
  • A tuple or NumPy array.
  • A class or instance attribute, such as obj.x. See Python classes and objects in query strings.
  • The result of a Python function whose return type has no Java mapping. This happens when the function has no return type hint, is annotated -> object, or uses a hint such as -> dict. A type cast on the call avoids this.

In these cases, the column holds PyObjects, not plain java.lang.Object values.

PyObjects can represent any Python object, such as a dictionary or a custom class instance returned from a function that has no type hint. That flexibility comes with significant limitations:

  • Limited compatibility with Deephaven's built-in functions
  • Reduced performance compared to Java primitive types
  • Higher memory usage, because each value is a separate object rather than a primitive

Use PyObjects only when working with complex Python objects that have no Java equivalent. Otherwise, use type hints or explicit type casts so the column gets a Java type.

Data type conversions

There are two ways to make sure a column computed by a Python function has the type you want: type hints and type casts. The following code block uses both, plus a column that uses neither, for comparison:

In source_meta, Typecast and TypeHint are double columns, while NoTypeHint is an org.jpy.PyObject column because its function has no type hint and its formula has no type cast.