Import and Export Data

Data I/O is mission-critical for any real-time data analysis platform. Deephaven supports a wide variety of data sources and formats, including CSV, Parquet, Kafka, and more. This chapter shows how to:

  • Read and write CSV and Parquet data.
  • Read Parquet files from S3.
  • Consume and publish Kafka streams.
  • Read and write Iceberg tables.
  • Convert tables to HTML.
  • Use pandas DataFrames as a bridge to other Python libraries.
  • Use function-generated tables to pull in data from other sources.

Note

The first CSV and Parquet examples read files from Deephaven's example data. A container has those files at /data/examples only when you start it from a deployment that includes example data. The docker run command in Get started does not include them. To run these examples, start one of those deployments, or download the files into the data/examples directory that command mounts, keeping the same paths.

CSV

Deephaven can read CSV files that exist locally or remotely.

Tip

To import a CSV without writing code, use Upload Table from File in the Console Options menu (â‹®). See Upload a table from a file.

This example reads a local CSV file.

Deephaven can also write data to CSV. The code below writes that same table back to a CSV file.

Just to show that it's there, the code below reads the new file back.

Parquet

Apache Parquet is a columnar storage format that supports compression to store more data in less space. Deephaven reads and writes single Parquet files as well as partitioned Parquet directories, where one table's data is split across many Parquet files in a directory. Deephaven reads Parquet data stored locally or in cloud object storage, such as Amazon S3, any S3-compatible object store, and Google Cloud Storage.

The example below reads from a local Parquet file.

The code below writes that same table back to a Parquet file.

Just to show that it worked, the code below reads the new file back.

The example below reads a Parquet file from a RustFS server at rustfs.example.com. RustFS is an S3-compatible object store. The S3Instructions class from the deephaven.experimental.s3 module specifies how to connect to the S3 instance. To run the example, replace the endpoint, file path, and credentials with those of your own S3 instance. See Read Parquet from S3 for more.

Kafka

Apache Kafka is a distributed event streaming platform for publishing and subscribing to streams of records. Deephaven can consume data from Kafka streams and publish data to them. See the Kafka guide for more.

The examples below connect to a Kafka broker at redpanda:9092. Redpanda is a Kafka-compatible broker. To run the examples, start Deephaven alongside Redpanda as described in Launching Kafka with Deephaven. Run the docker compose commands below from the directory that holds the docker-compose.yml file from that guide.

The code below consumes a stream.

The result_append table starts empty. Rows appear as records arrive on test.topic. To send some, run docker compose exec redpanda rpk topic produce test.topic in a terminal and type a few lines. rpk is Redpanda's command-line client.

Similarly, this code publishes the data in a Deephaven table to a Kafka stream.

To see the records arrive, run docker compose exec redpanda rpk topic consume time-topic in a terminal.

Iceberg

Apache Iceberg is a high-performance, open table format for huge analytic datasets. Deephaven's deephaven.experimental.iceberg module reads and writes Iceberg tables. For more on Deephaven's Iceberg integration, see the Iceberg user guide.

Both Iceberg examples need the custom Docker deployment described in A Deephaven deployment for Iceberg.

An Iceberg catalog adapter connects Deephaven to an Iceberg catalog. The following example creates one with iceberg.adapter and uses it to read an existing Iceberg table into a Deephaven table.

Using the same iceberg_catalog_adapter, this code creates a new Iceberg table and writes a Deephaven table to it. create_table fails if the table already exists. To write to an existing table, use load_table instead.

HTML

The to_html function from the deephaven.html module converts a Deephaven table into an HTML string. This is useful for displaying small tables in web pages or for creating simple HTML reports. to_html builds the whole string in memory, so avoid converting large tables.

pandas DataFrames

The Python integrations chapter showed how to_pandas and to_table from the deephaven.pandas module convert between Deephaven tables and pandas DataFrames. For I/O, that makes pandas a bridge. Convert a table with to_pandas to leverage pandas' extensive data manipulation and analysis capabilities, or to hand it to any other Python library that works on DataFrames. Convert a DataFrame with to_table to bring a library's results back into Deephaven.

Note

Converting a table to a pandas DataFrame copies the entire table into memory. For large tables, filter or aggregate the data in Deephaven first.

Function-generated tables

Function-generated tables are a powerful tool for ingesting data from external sources such as WebSockets, databases, and much more. A Python function populates a function-generated table. Deephaven reruns the function at a regular interval, or whenever one of the tables you pass in the source_tables argument of function_generated_table changes. The table does no data I/O on its own, but your function can read from an external source each time it runs.

The following example regenerates data in a table once per second. To keep the example self-contained, the function generates random data instead of reading from an external source.

For an example that polls a web API for weather data, see the function-generated tables blog post.