Data types in Deephaven and Python
For performance reasons, the Deephaven engine is implemented in Java. As such, Deephaven tables use Java data types for columns, both Java primitive types and Java objects.
Deephaven Python queries combine the power of Python with Deephaven's tables. Table columns hold Java types, so every value a query puts in a column has a Java type. Knowing how Python and Java types map to each other helps you write queries that are fast and produce the column types you expect.
The Java data types section maps Python and NumPy types to the Java types that columns hold. Data type conversions then shows how to make sure a column gets the type you want with type hints or type casts.
Java data types
Primitive types
In Python, everything is an object, including scalar types like int and float. A Python int carries methods and attributes that a Java primitive value does not. Java, by contrast, has primitive types as well as objects.
Java primitive types map directly to hardware representations and have fixed sizes, unlike Python's arbitrary-precision int. Deephaven stores primitive values directly rather than as objects, so they are faster and use less memory. That speed and memory saving is the main reason to use them for table columns.
The following table shows the mapping between Java primitive types, Java primitive type sizes, and the Python and NumPy types that map to them in a function's type hints:
| Java primitive type | Java primitive type size | Python type | NumPy type |
|---|---|---|---|
byte | 1 byte | N/A | np.int8 |
short | 2 bytes | N/A | np.int16 |
int | 4 bytes | N/A | np.int32 |
long | 8 bytes | int | np.int64 |
float | 4 bytes | N/A | np.float32 |
double | 8 bytes | float | np.float64 |
char | 2 bytes | N/A | np.uint16 |
Deephaven tables have no primitive boolean column type. True/false values go in java.lang.Boolean columns, which Deephaven stores as one byte per value:
- A formula that returns a Java
booleanproduces ajava.lang.Booleancolumn. - In a function's type hints, Python
boolandnp.bool_map tojava.lang.Boolean, as the object types table shows.
Variables used directly in a query string follow different rules from type hints. For example, a Python int variable becomes the smallest Java integer type that holds its value. Most NumPy scalar variables become PyObjects. Python variables in query strings explains how each kind of Python variable converts.
Object types
Object types are Java classes whose values are objects rather than raw values. They consume more memory and are generally slower than their primitive counterparts.
Some object types are commonly used despite the overhead. For instance, java.lang.String is the best way to store text in tables.
The following table shows the Python and NumPy types that map to common Java object types in a function's type hints. The last row, org.jpy.PyObject, is the Java wrapper for a Python value that has no Java mapping:
| Java object type | Python type | NumPy type |
|---|---|---|
java.lang.String | str | np.str_ |
java.lang.Boolean | bool | np.bool_ |
java.time.Instant | datetime.datetime | np.datetime64 |
| Array | Sequence[T], such as Sequence[int] | np.ndarray with an element type, such as npt.NDArray[np.int64] |
org.jpy.PyObject | object | np.object_ |
An array type hint's element type follows the same mapping, so Sequence[int] and npt.NDArray[np.int64] both produce a long[] column.
For every type Deephaven defines in Python, see deephaven.dtypes.
Array types
Deephaven tables commonly hold two kinds of array column:
- Java arrays, most often of primitive types such as
int[]ordouble[]. - Deephaven vectors, which table operations such as grouping produce.
Arrays vs vectors compares the two.
A Python list used in a query string without type information becomes an org.jpy.PyListWrapper column, not a Java array. Tuples, NumPy arrays, and other sequences that are not lists or strings become PyObjects.
To get a Java array column, convert the sequence first with jpy.array or deephaven.dtypes.array. Work with arrays shows how.
PyObjects
org.jpy.PyObject is the Java wrapper that the jpy Java-Python bridge puts around arbitrary Python objects.
A PyObject column typically appears when a query string uses one of these Python values:
- Most NumPy scalar variables, such as
np.int64,np.float32, ornp.bool_. The exceptions arenp.float64andnp.str_, which becomedoubleandjava.lang.Stringvalues. - A tuple or NumPy array.
- A class or instance attribute, such as
obj.x. See Python classes and objects in query strings. - The result of a Python function whose return type has no Java mapping. This happens when the function has no return type hint, is annotated
-> object, or uses a hint such as-> dict. A type cast on the call avoids this.
In these cases, the column holds PyObjects, not plain java.lang.Object values.
PyObjects can represent any Python object, such as a dictionary or a custom class instance returned from a function that has no type hint. That flexibility comes with significant limitations:
- Limited compatibility with Deephaven's built-in functions
- Reduced performance compared to Java primitive types
- Higher memory usage, because each value is a separate object rather than a primitive
Use PyObjects only when working with complex Python objects that have no Java equivalent. Otherwise, use type hints or explicit type casts so the column gets a Java type.
Data type conversions
There are two ways to make sure a column computed by a Python function has the type you want: type hints and type casts. The following code block uses both, plus a column that uses neither, for comparison:
In source_meta, Typecast and TypeHint are double columns, while NoTypeHint is an org.jpy.PyObject column because its function has no type hint and its formula has no type cast.
Related documentation
- Arrays vs vectors
- Query string overview
- Formulas
- Python variables in query strings
- Python functions in query strings
- Python classes and objects in query strings
- Java objects in query strings
- Work with arrays
- Group and ungroup data
- Casting
- Handle PyObjects in tables
- Use jpy
- Built-in functions
- Deephaven Python data types
- Arrays
- Date-time
- Strings
empty_tablemeta_tableupdatedeephaven.dtypesPydoc