Deephaven Community Core version 0.31.0 was recently released. Deephaven's final release of 2023 brings several bug fixes and quality-of-life improvements. Let's talk about some noteworthy updates from the release.
Python type hints and query strings
Deephaven's Python API now supports type hints for functions that:
- Return arrays, lists, or other sequences of data.
- Have optional returns.
This means that the query engine fully supports your functions that return lists, arrays, or (optionally) nothing. The best way to show what this means is through code. Keep reading to see these new features in example queries.
Arrays, lists, and sequences
Users often work with data that's grouped into arrays in tables. Python functions that return these data types are now much easier to use within tables via type hints. The Python modules typing and numpy.typing were built for these use cases. These features work with partitioned tables as well.
from deephaven import empty_table
import numpy.typing as npt
from typing import List
import numpy as np
def list_func(val) -> List[np.double]:
return [val, val * np.pi]
def np_array_func(val) -> npt.NDArray[np.int_]:
return np.array([val, val + 1, val + 2])
source = empty_table(20).update(["X = i"])
result = source.update(["ListColumn = list_func(X)", "NDArrayColumn = np_array_func(X)"])
result_meta = result.meta_table
- result
- result_meta
- source
Optional returns
Sometimes functions don't return anything at all. These are best handled by the Optional keyword provided by the typing module in a type hint. The optional annotation supports both scalar and array return types.
from deephaven import empty_table
from typing import Optional
import numpy.typing as npt
import numpy as np
def optional_func(val) -> Optional[np.double]:
if val % 2 == 0:
return None
else:
return val * np.pi
def optional_list_func(val) -> Optional[npt.NDArray[np.int_]]:
if val % 2 == 0:
return np.array([val, val + 1])
else:
return None
source = empty_table(20).update(["X = i"])
result = source.update(["ColWithNulls = optional_func(X)", "ListColWithNulls = optional_list_func(X)"])
result_meta = result.meta_table
- result
- result_meta
- source
Blink input tables
The Groovy API now supports blink input tables. This enables client APIs to add data into blink tables, which keep only the most recent update cycle of data in memory.
Parquet
Deephaven's Parquet integration has significantly expanded and improved over the last few releases. 0.31 is no different. Users can now create a refreshing, hive partitioned empty Parquet table. Previously, the backing directory structures needed to be in place beforehand. This change greatly reduces the work needed to produce tables for these types of workflows.
Looking ahead
With 2023 coming to a close, it's as good a time as ever to look ahead to what's in store for 2024. We have some big plans for the upcoming year. A new feature we're particularly excited about is deephaven.ui. deephaven.ui is a new framework for building reactive UI components for real-time data applications, entirely in Python. No front-end engineering needed. Users can turn Kafka feeds, event streams, and data lakes into live BI dashboards in minutes, not days.
deephaven.ui has been under active development and testing for some time now, and we're very excited for its upcoming release. If you develop applications with Deephaven, we have no doubt that you'll benefit from it. Stay tuned for future updates.
Reach out
Our Slack community continues to grow! Join us there for updates and help with your queries.