pandas holds data projects together like glue
The pandas DataFrame
Python pandas is the second-most popular Python framework, according to Stack Overflow's 2021 survey. It presents a versatile table construct -- the pandas DataFrame -- that is a higher-order sibling to NumPy's array object.
Deephaven is a compelling complement to pandas, because it shares a fundamental "table-first" development style. Deephaven's "StreamingTables", however, are designed to be dynamic (e.g., changing in real time). Since both are powerful Python libraries, users migrate seamlessly between pandas and Deephaven, using DataFrames as static sources for StreamingTables, or snapshotting the latter for export to DataFrames on a one-time or periodic basis.
pandas and NumPy
pandas DataFrames are backed by NumPy arrays, where users can access rows and columns by referring to their names. Thus, DataFrames in particular make accessing data easier than alternatives such as NumPy ndarrays, lists, or other objects. Additionally, DataFrames have a number of built-in methods for moving window operations, finding specific data, separating columns and rows from one another, and many others.
pandas DataFrames, Deephaven Tables and AI/ML
The pandas Python library supports operations on data structures that map directly to Deephaven data tables. Furthermore, pandas can be used in conjunction with Deephaven tables for queries in artificial intelligence and machine learning (AI/ML). Models can be trained using pandas DataFrames. Then, the trained models can be leveraged in real time on Deephaven tables. The latter is usually accomplished by running functions for converting pandas DataFrames to and from Deephaven tables (see supporting documentation below).
And to further streamline real-time AI calculations within Python, converting pandas DataFrames to NumPy ndarrays is lightning fast, due to the fact that the structures themselves are backed directly by NumPy ndarrays.
Real-time calculations in Python
pandas allows the Deephaven Core query engine to train AI/ML models, among many other practical data applications. Particularly, pandas and Deephaven play well together along:
- Training an AI/ML model on a DataFrame, which the user can then test on a real-time Deephaven table
- Reordering and splitting table data with a pandas DataFrame
- Moving window operations in pandas and Deephaven
Video: Interop with Python
Starter projects with pandas
Cool things you can do with pandas + Deephaven in real time:
Develop with Deephaven Core
- VIDEO: Interop with Python
- How to use pandas in Python queries
- Export data to popular formats
- Use deephaven.learn
- What can you use Deephaven for?
Python-related blog posts:
- Performance vs. pandas
- Jump right into Python machine learning with Deephaven Learn
- Introducing Deephaven Learn: Your new favorite tool for real-time AI in Python
- Real-time classification with Deephaven and SciKit-Learn