Open API overview

Deephaven's Open API provides an interface for external applications to access and present data, both through creating new queries or using existing ones. This section of the documentation describes the basic functionality provided through the Open API access, and the aspects of execution taking place on the server. More specific documentation is provided for each of the languages and runtimes supported. (Note: Javascript is the only language currently supported. However, support for other languages (e.g., Java, Python, etc.) is anticipated.)

Deephaven Servers and Concepts

An installation of Deephaven consists of several servers:

  • One or more Authentication servers, handling authentication and authorization for the other servers.
  • One or more Query Configuration servers, keeping track of persistent queries.
  • One or more Remote Query Dispatcher servers, managing workers as needed.
  • Remote Query Processor (worker) servers, performing queries as requested by persistent queries or client consoles.
  • A Web server, providing the ability for a client to authenticate and gain access to a worker to make queries.

Authentication can take many forms, implemented per installation, and from the perspective of the client require a username and some secret to connect, such as a SSO token/nonce or a password. Once connected and authenticated, the user can request Auth Tokens to connect to another server, or a Reconnection Token to reconnect to the Authentication server in the event of a disconnect. The Reconnection Token must be periodically refreshed to allow reconnecting without storing credentials in the client. The Auth Token expires after one minute and can only be used once, so it should only be requested when the client wants to start a connection to a new server.

The Web server can be connected to by a browser or other websocket client. After connecting, the client must authenticate to obtain an Auth Token, and generally should keep that Auth Token up to date. This server will provide access to available Query Configs, each detailing the worker on which they are running and the tables available within that Query Config. In a later revision of the Deephaven Open API, the Remote Query Dispatcher will also be available, allowing for direct console access to the Deephaven system.

From a Query Config, the client will have the details required to initiate another websocket connection to that worker. There, they must register with their active Auth Token to gain access to the tables on that server. To interact with a table, the client should first request the details about the table. With those details, the client may request the the contents of the table and subscribe to any updates that occur.

When it comes to requesting data and updates, tables come in two basic forms: preemptive, and non-preemptive. A preemptive table will always send all of its contents to the client, and does not support specific viewport subscriptions, while a non-preemptive table will allow the client to specify that they only want to be informed about updates in a specific set of rows and columns.

Once subscribed, the worker will first send a snapshot of results the client is interested in, and then will send periodic updates as needed. If too many changes have taken place in a given timeframe to handle an update, a new snapshot may be sent instead. The client can cancel their subscription to a table, and can also request a snapshot of the table without any update subscription.

The configuration of a Deephaven table is immutable, but new tables can be easily created. Sorting, filtering, or adjusting the columns of a table are all performed by passing the original table back to the server with one of those operations, and a new table handle will be sent back informing the client that they can now subscribe to or request data for that new table. The new table may or may not share a definition with the previous table - changes such as sorting and filtering do not affect the definition. Subscriptions or references can be maintained to intermediate tables, to modify sort or filter operations, or to see the previous results alongside the new ones.

Order of operations on the server

Each operation changing what data is visible (sort, filter, setViewport) is understood to be performed serially - developers can still call other methods, and their work will be queued up but not performed until the previous operations' changes have been applied.

Additionally, when changing the sort or filter of a table, the current viewport is removed, allowing the API consumer to define the correct behavior for their application, either to move to the top of the new data, or to try to keep the same range of rows visible (as opposed to keeping the same rows visible, which may now be scattered and have other rows between them, or may not exist at all). This means that a new viewport should be applied every time the sort and/or filter has been changed.

In this example, a new sort is being applied, rowCount is the number of visible rows, and visibleColumns is the list of columns to display:

// First, change the sort
table.applySort([nameColumn.sort().desc()]);
// Then, specify the viewport to be used after the sort is complete
table.setViewport(0, rowCount, visibleColumns);

In contrast, here we are applying both a sort and a filter at the same time. The viewport should only be set after both the new sort and new filter has been applied because otherwise it would be cleared right away.

// Change the filter
table.applyFilter([nameColumn.filter().eqIgnoreCase("FOO").not()]);

// table.setViewport <-- Do not change viewport yet, since we are about to make another change

// Change the sort
table.applySort([nameColumn.sort().asc()]);

// Finally, apply the viewport now that both changes are made
table.setViewport(0, rowCount, visibleColumns);

Operations performed will be queued on the server to avoid latency.

Deephaven Glossary

  • Auth Token - An identifier created by the Deephaven Authentication server, allowing a client to connect to a server and make requests to it.
  • Open API - Allows access to Deephaven data without using our own client.
  • Persistent Query - A scripted query set up to run on a worker, available to multiple users at a time, discoverable from the Query Configuration server.
  • Query Config - Describes a persistent query, the script running on it, and the tables it makes available to users.
  • Reconnection Token - A signed identifier created by the Deephaven Authentication server, allowing a client to reconnect without providing credentials again.
  • Table Definition - Metadata for a table on the Deephaven server, describing the columns in the table and their types.
  • Web API - A specialized version of Open API access, intended for use when building a web application.