JavaScript

Deephaven's Open API provides an interface for external applications to access and present data, both through creating new queries or using existing ones. This section of the documentation describes the basic functionality provided through the Open API access, and the aspects of execution taking place on the server. JavaScript is the only language currently supported.

JavaScript examples

In this section, we provide a high-level overview of how a JavaScript client might interact with a Deephaven installation.

This will be done with pseudo-code and simple explanations to give a complete picture of how the system can be used, and a brief introduction into Deephaven terminology.

For implementation details, see the JavaScript API docs.

To get started, we need to load the JS library in a script tag. This file is tied to the version of the server that it is connecting to, and so is typically loaded from the server when the application starts. The file resides at /irisapi/irisapi.nocache.js on the web server that the web IDE is loaded from, and what the client will connect to. In this example, we'll assume that the server is hosted at https://myserver/, and that this URL correctly resolves for the development environment.

<script
  src="https://myserver/irisapi/irisapi.nocache.js"
  type="text/javascript"
></script>

Example 1: Logging in

class MyApp {
  constructor() {
    // Connect to the server we previously loaded the library from. We use the path
    // "/socket", and protocol "wss" to connect to an https webserver.
    this.client = new dh.Client('wss://myserver/socket');
    // The actual token here can be a password, or any other supported auth token:
    this.client
      .login({ username: 'bob', token: '$₪£€₽$€₡₪₽€', type: 'password' })
      .then(() => this.startApp(), console.error);
    // .then() is used to add callbacks to a Promise,
    // which will be used anywhere data is asynchronously loaded
  }
  startApp() {
    // Now that the client is authenticated, all Deephaven functions,
    // such as opening persistent queries and opening tables, are available
  }
}
new MyApp();

In Example 1 above, we are sending the username, the token, and the type of auth to use. You can configure your own auth handler to integrate with any external system, and Deephaven will simply check with your server to verify user identities.

The underlying network transport mechanism is not yet available to clients as we will manage web socket connections and the auth token refreshes automatically; all the client needs to do is add callbacks to Promises as needed.

Example 2: Discovering database tables

startApp() {
    // Subscribe to updates on all persistent queries
    this.client.addEventListener('configadded', e => addQuery(e.detail));
    this.client.addEventListener('configremoved', (e) => removeQuery(e.detail));
    this.client.addEventListener('configupdated', (e) => updateQuery(e.detail));

    // Fetch all known persistent queries (collections of tables)
    this.client.getKnownConfigs().forEach(queryInfo => addQuery(queryInfo));
}

addQuery(queryInfo) {
    console.log(`Loaded query ${queryInfo.name} [${queryInfo.serial}]`);
    for ( name in queryInfo.tableNames) {
        this.addTable(queryInfo, name);
    }
}
removeQuery(queryInfo) { ... }
updateQuery(queryInfo) { ... }
addTable(queryInfo, name) {
    // Render a list of queries and tables to choose from
}

In Example 2 above, we are retrieving and subscribing to changes in Persistent Queries, which include collections of tables and the configuration for running those tables; you can read more about Persistent Query settings in the technical specification.

Example 3: Selecting a table

addTable(queryInfo, name) {
    // Create an element to click to select table.
    show(new TableSelector(queryInfo, name);
}

class TableSelector {
  constructor(queryInfo, name) {
      // Create a function that returns a promise to load the table.
      // Connecting to a table to read information can be expensive, so we avoid doing it.
      this._table = () => queryInfo.getTable(name);
      this._name = name;
      someElement.innerText = name;
  }

  onClick() {
    this._table().then(table =>
        show(new TableView(this._name, table))
    );
  }
}

class TableView {
    constructor(name, table) {
        this._table = table;
        // Draw a placeholder
        renderFrame(name, table);

        // Subscribe to the updated event to get table data to render
        // addEventListener returns a function to easily perform cleanup
        this._cleanup = table.addEventListener('updated', e => this.renderTable(e));
        // This remote call does not return data directly;
        // instead, we use our subscription to the `updated` event, above.
        table.setViewport(startRow, endRow, table.columns);
    }

    renderFrame(name, table) {
        someHeader.innerText = name;
    }

    renderTable(event) {
        this._viewport = event.detail;
        drawViewport();
    }

    drawViewport() {
        var cnt = this._viewport.offset;
        this._viewport.rows.forEach(row => this.drawRow(cnt++, row));
    }

    drawRow(rowNum, row) {
        // Render a row of data using this._table.columns for type information.
        this._table._columns.forEach(column=> this.drawCell(rowNum, row, column) );
    }

    drawCell(rowNum, row, column) {
        doHtmlThings(rowNum, row.get(column));
    }

    dispose() {
        // Stop subscribing to the events from this table
        this._cleanup();

        // Additionally, it may make sense to close the Table entirely, if it is not to be used again
        this._table.close();
    }

}

In Example 3 above, we are loading the actual table data for rendering; metadata such as table size, columns, sorts and filters will be kept up to date for you, but the actual table data will not load until you call table.setViewport(), and receive updated events to trigger drawing.

This allows both the client and the server to decide to trigger a redraw, without burdening the client with extra state or callback management.

Example 4: Sorting and filtering

class TableView {

    // Given a table with columns: type(String), index(Number), created(Date), modified(Date),
    // setSort(columns.type.asc(), columns.created.desc())
    setSort() {
        // all arguments should be iris.Sort objects.
        this._sorts = Array.prototype.slice.apply(arguments);
        var promise = this._table.applySort(this._sorts);
        return promise;
    }

    addSort(sort) {
        this._sorts.push(sort);
        return this._table.applySort(this._sorts);
    }

    setFilter() {
        this._filters = Array.prototype.slice.apply(arguments);
        return this._table.applyFilter(this._filters);
    }

    addFilter(filter) {
        this._filters.push(filter);
        return this._table.applyFilter(this._filters);
    }

    // Use cloning when you want to create a new table to apply sorts and filters without modifying the existing table.
    // Note that each the original and the clone will fire their own events, maintain their own viewport, and
    // individual close() calls will need to be applied individually as appropriate.
    clone(name) {
        if (!name) {
            name = `${this._name}Clone`;
        }
        return this._table.copy()
            .then(newTable=>return new TableView(name, newTable));
    }

}

Example 5: Add support for Core+ queries

class TableSelector {
  constructor(legacyClient, queryInfo, name) {
    // ...
    this._table = () => this.getTable(legacyClient, queryInfo, name);
    // ...
  }

  isCorePlusQuery(queryInfo, workerKinds) {
    // Get the worker kind object matching the query engine
    // If the worker supports the Community protocol, it's a Core+ query
    const workerKind = workerKinds?.find(
      ({ name }) => name === queryInfo.workerKind
    );
    console.log('Query worker kind', workerKind);
    return workerKind?.protocols?.includes('Community') ?? false;
  }

  async getCorePlusApi(jsApiUrl) {
    // Dynamically load the API instance from the given URL
    console.log('Import API', jsApiUrl);
    try {
      const api = (await import(jsApiUrl)).default;
      console.log('API bootstrapped from', jsApiUrl);
      return api;
    } catch (e) {
      console.error('Unable to bootstrap API', e);
      throw new Error('Unable to bootstrap API');
    }
  }

  async getCorePlusClient(api, token, grpcUrl, envoyPrefix) {
    // Create a Core+ client instance and authenticate
    const clientOptions = envoyPrefix
      ? {
          headers: { 'envoy-prefix': envoyPrefix },
        }
      : undefined;
    console.log('Init Core+ client', grpcUrl);
    const corePlusClient = new api.CoreClient(grpcUrl, clientOptions);
    console.log('Core+ client', corePlusClient);
    const loginOptions = {
      type: 'io.deephaven.proto.auth.Token',
      token,
    };
    console.log('Log in with', loginOptions);
    await corePlusClient.login(loginOptions);
    console.log('Log in success');
    return corePlusClient;
  }

  async getCorePlusConnection(api, token, queryInfo) {
    const { serial, grpcUrl, envoyPrefix } = queryInfo;
    console.log('Get Core+ Client for query', serial);
    const corePlusClient = await this.getCorePlusClient(
      api,
      token,
      grpcUrl,
      envoyPrefix
    );
    return corePlusClient.getAsIdeConnection();
  }

  async getTable(legacyClient, queryInfo, name) {
    const { workerKinds } = await legacyClient.getServerConfigValues();
    if (this.isCorePlusQuery(queryInfo, workerKinds)) {
      // Getting the table from the Core+ query requires a Core+ API instance
      // and an authenticated Core+ connection
      const api = await this.getCorePlusApi(queryInfo.jsApiUrl);
      const token = await legacyClient.createAuthToken('RemoteQueryProcessor');
      const connection = await this.getCorePlusConnection(
        api,
        token,
        queryInfo
      );
      const objectDefinition = {
        name,
        type: 'Table',
      };
      return connection.getObject(objectDefinition);
    }
    // Get the table from the legacy query
    return queryInfo.getTable(name);
  }

  onClick() {
    this._table().then((table) => show(new TableView(this._name, table)));
  }
}

In example 5 above, we dynamically import the API instance from the Core+ worker, and use it to create a table. This builds on the earlier examples, and will initially use the Enterprise library to discover and authenticate to the Core+ workers.

JS API Design

The Deephaven high-level Javascript API is written to handle the details discussed above, and give a web developer a familiar starting point when writing their application. This section will discuss the abstractions provided to the web developer to avoid the need to understand the comprehensive workings of the Deephaven platform.

The connection is established by specifying the URL of the Web server to the Client object and logging in. From there, Auth Tokens are automatically maintained, and any call that requires connecting to another server will be done automatically, including exchange of tokens.

While client tables could still be used as if they were immutable by cloning before making any changes to sort, filter, or columns, this API also allows their use as mutable objects, and internally tracks the server-side tables, creating new ones as is necessary. Additionally, the distinction is not made between table handles and table definitions - the JS type Table will wrap all of the relevant details.

Total table size is exposed through a method instead of requiring client code to keep track of changes to the size across messages.

Format info is not exposed as a distinct, hidden column, but as additional detail on each cell.

Most operations will likely be handled "off-thread" through the use of a web worker to actually handle messages from the server, and to pass changes back to the server again, so processing sets of changes will not impede the UI from being responsive. Combined with the async nature of well-behaved JS communicating with the server, most methods that change the content to be rendered or ask for details not yet rendered will themselves be async, generally through the use of Promises or events.

Deephaven servers and concepts

An installation of Deephaven consists of several servers:

• One or more Authentication servers, handling authentication and authorization for the other servers.
• One or more Query Configuration servers, keeping track of persistent queries.
• One or more Remote Query Dispatcher servers, managing workers as needed.
• Remote Query Processor (worker) servers, performing queries as requested by persistent queries or client consoles.
• A Web server, providing the ability for a client to authenticate and gain access to a worker to make queries.

Authentication can take many forms, implemented per installation, and from the perspective of the client require a username and some secret to connect, such as a SSO token/nonce or a password. Once connected and authenticated, the user can request Auth Tokens to connect to another server, or a Reconnection Token to reconnect to the Authentication server in the event of a disconnect. The Reconnection Token must be periodically refreshed to allow reconnecting without storing credentials in the client. The Auth Token expires after one minute and can only be used once, so it should only be requested when the client wants to start a connection to a new server.

The Web server can be connected to by a browser or other websocket client. After connecting, the client must authenticate to obtain an Auth Token, and generally should keep that Auth Token up to date. This server will provide access to available Query Configs, each detailing the worker on which they are running and the tables available within that Query Config. In a later revision of the Deephaven Open API, the Remote Query Dispatcher will also be available, allowing for direct console access to the Deephaven system.

From a Query Config, the client will have the details required to initiate another websocket connection to that worker. There, they must register with their active Auth Token to gain access to the tables on that server. To interact with a table, the client should first request the details about the table. With those details, the client may request the the contents of the table and subscribe to any updates that occur.

When it comes to requesting data and updates, tables come in two basic forms: preemptive, and non-preemptive. A preemptive table will always send all of its contents to the client, and does not support specific viewport subscriptions, while a non-preemptive table will allow the client to specify that they only want to be informed about updates in a specific set of rows and columns.

Once subscribed, the worker will first send a snapshot of results the client is interested in, and then will send periodic updates as needed. If too many changes have taken place in a given timeframe to handle an update, a new snapshot may be sent instead. The client can cancel their subscription to a table, and can also request a snapshot of the table without any update subscription.

The configuration of a Deephaven table is immutable, but new tables can be easily created. Sorting, filtering, or adjusting the columns of a table are all performed by passing the original table back to the server with one of those operations, and a new table handle will be sent back informing the client that they can now subscribe to or request data for that new table. The new table may or may not share a definition with the previous table - changes such as sorting and filtering do not affect the definition. Subscriptions or references can be maintained to intermediate tables, to modify sort or filter operations, or to see the previous results alongside the new ones.

Order of operations on the server

Each operation changing what data is visible (sort, filter, setViewport) is understood to be performed serially - developers can still call other methods, and their work will be queued up but not performed until the previous operations' changes have been applied.

Additionally, when changing the sort or filter of a table, the current viewport is removed, allowing the API consumer to define the correct behavior for their application, either to move to the top of the new data, or to try to keep the same range of rows visible (as opposed to keeping the same rows visible, which may now be scattered and have other rows between them, or may not exist at all). This means that a new viewport should be applied every time the sort and/or filter has been changed.

In this example, a new sort is being applied, rowCount is the number of visible rows, and visibleColumns is the list of columns to display:

// First, change the sort
table.applySort([nameColumn.sort().desc()]);
// Then, specify the viewport to be used after the sort is complete
table.setViewport(0, rowCount, visibleColumns);

In contrast, here we are applying both a sort and a filter at the same time. The viewport should only be set after both the new sort and new filter has been applied because otherwise it would be cleared right away.

// Change the filter
table.applyFilter([nameColumn.filter().eqIgnoreCase("FOO").not()]);

// table.setViewport <-- Do not change viewport yet, since we are about to make another change

// Change the sort
table.applySort([nameColumn.sort().asc()]);

// Finally, apply the viewport now that both changes are made
table.setViewport(0, rowCount, visibleColumns);

Operations performed will be queued on the server to avoid latency.

Deephaven Glossary

• Auth Token - An identifier created by the Deephaven Authentication server, allowing a client to connect to a server and make requests to it.
• Open API - Allows access to Deephaven data without using our own client.
• Persistent Query - A scripted query set up to run on a worker, available to multiple users at a time, discoverable from the Query Configuration server.
• Query Config - Describes a persistent query, the script running on it, and the tables it makes available to users.
• Reconnection Token - A signed identifier created by the Deephaven Authentication server, allowing a client to reconnect without providing credentials again.
• Table Definition - Metadata for a table on the Deephaven server, describing the columns in the table and their types.
• Web API - A specialized version of Open API access, intended for use when building a web application.

Web Development Glossary

• Promise - A fluent callback API in modern browsers.
• Web Socket - An outgoing TCP socket connection with a few additional constraints (HTTP CONNECT handshake, XOR of frame contents) that can be created by JS code in modern browsers.
• Web Worker - A distinct JS process without access to the DOM, events, or other UI details, allowing for work to be done in the background of modern browsers.

Related documentation

• Javascript API docs