Overview of the Deephaven Core R Client

The Deephaven Core R Client provides an R interface to Deephaven’s powerful real-time data engine, Deephaven Core. To use this package, you must have a Deephaven server running and be able to connect to it. For more information on how to set up a Deephaven server, see the documentation here.

Building blocks of the Deephaven R Client

There are two primary R classes that make up the Deephaven R Client, the Client class and the TableHandle class. See ?Client and ?TableHandle for more information. The Client class is used to establish a connection to the Deephaven server with its constructor Client$new(), and to send server requests, such as running a script via run_script(), or pushing local data to the server via import_table(). Basic usage of the Client class may look something like this:

library(rdeephaven)
client <- Client$new("localhost:10000", auth_type = "psk", auth_token = "my_secret_token")

Many of these server requests end up creating or modifying tables that live on the server. To keep track of these tables, the R client retrieves references to them, and wraps these references in TableHandle objects. These TableHandles have a host of methods that mirror server-side table operations, such as head(), tail(), update_by(), and so on. So, you can typically use TableHandles as if they are tables themselves, and all of the corresponding methods that you call on them will be executed on the server. Here is a simple example of pushing data to the server, retrieving a TableHandle to the resulting table, and applying some basic table operations to the table:

df1 <- data.frame(x = 1:10, y = 11:20)
th1 <- client$import_table(df1)

th2 <- th1$
  update("z = x + y")$
  where("z % 4 == 0")

TableHandles also support common functional methods for converting server-side Deephaven tables to R objects stored in local memory such as as.data.frame(), as_tibble(), and as_arrow_table(). Here’s an example of converting the table created above to an R data frame and verifying that other functional methods work as expected:

df2 <- as.data.frame(th2)

print(nrow(th2) == nrow(df2))
print(ncol(th2) == ncol(df2))
print(dim(th2) == dim(df2))
print(all(as.data.frame(head(th2, 2)) == head(df2, 2)))
print(all(as.data.frame(tail(th2, 2)) == tail(df2, 2)))

For more information on these classes and all of their methods, see the reference documentation by running ?Client or ?TableHandle.

Real-time data analysis

Since TableHandles are references to tables living on the Deephaven server, they may refer to streaming tables, or tables that are receiving new data periodically(typically once per second). Here’s a simple example of creating a table that adds a new row every second:

th3 <- client$time_table("PT1s")$
  update(c("X = ii", "Y = sin(X)"))

R objects like data frames or Dplyr tibbles do not have this streaming property - they are always static objects stored in memory. However, a TableHandle referring to a streaming table may be converted to a data frame or tibble at any time, and the resulting object will be a snapshot of the table at the time of conversion. This means that you can use the Deephaven R Client to perform real-time data analysis on streaming data! Here, we make a simple plot of the ticking table, and call it three times to demonstrate the dynamic nature of the table:

plot(as.data.frame(th3)$X, as.data.frame(th3)$Y, type = "l")
Sys.sleep(5)
plot(as.data.frame(th3)$X, as.data.frame(th3)$Y, type = "l")
Sys.sleep(5)
plot(as.data.frame(th3)$X, as.data.frame(th3)$Y, type = "l")

There are performance and memory considerations when pulling data from the server, so it is best to use the provided TableHandle methods to perform as much of your analysis as possible on the server, and to only pull the data when something must be done in R, like plotting or writing to a local file.

Powerful table operations

Much of the power of Deephaven’s suite of table operations is achieved through the use of the update_by() and agg_by() methods. These table methods are important enough to warrant their own documentation pages, accessible by running vignette("update_by") or vignette("agg_by"). These methods come with their own suites of functions, prefixed with agg_ and uby_ respectively, that are listed in their documentation pages. Running ls("package:rdeephaven") will reveal that most of the functions included in this package are for these methods, so it is important to get acquainted with them.

Getting help

While we’ve done our best to provide good documentation for this package, you may find you need more help than what this documentation has to offer. Please visit the official Deephaven Community Core documentation to learn more about Deephaven and to find comprehensive examples. Additionally, feel free to reach out to us on the Deephaven Community Slack channel with any questions. We hope you find real-time data analysis in R to be as easy as possible.