Code Studio

When using the Deephaven Web interface, the Code Studio is used to write console-based queries for analyzing data. The tables, charts, and widgets generated from those queries (as well as other tables, charts, and widgets) can then be arranged to create customized workspaces in the Deephaven interface.

Connect

When you open a new Code Studio, the New Console Connection screen provides options to:

• specify the Server you want to use,
• the Heap Size (memory size) requested to run your queries,
• and the language you plan on using for the query (Groovy or Python).

Server

Available servers are presented in alphabetical order in the drop-down menu. Because every installation is unique, please consult your system administrator for additional information about available servers.

Heap Size

This setting refers to the dynamic memory (RAM) you want Deephaven to use to process the instructions and the data needed to run this specific query. Memory listed in this setting is noted in gigabytes (GB). Note: memory is a finite resource on the system. It is usually best to start small when estimating the amount of memory needed to run the query, and then adjust it accordingly as needed.

Language

Deephaven scripts can be written in Groovy or Python. This value defaults to Groovy. If you prefer to write your scripts in Python, select Python from the drop-down menu.

Show Advanced

Additional connection options are available by selecting Show Advanced.

Connect to Server

After adjusting the applicable settings in the dialog box, click Connect.

After establishing a connection to the server, the Code Studio window will look similar to the image below. Two primary sections will fill the screen:

On the left, the Console and Log panels are nested together. The Console is for writing and executing queries. After the query executes, any tables, plots and/or widgets generated by the query will appear in the workspace as a new panel.

To the right, the Command History, File Explorer, and PQ Explorer panels are nested together. Any of the panels can be rearranged in the workspace by dragging and dropping the panel tabs, or maximized to full screen. (See: Customizing Your Layout.) Note that if your broswer window width cannot accommodate all three tabs, an arrow will appear that provides access to the additional tabs. In the example below, the arrow can be used to bring the PQ Explorer into the foreground:

Additional Code Studios can be added to the interface by using the New button in the Navigation bar at the top. Consider creating separate code studios for each console and server configuration you commonly access. All Code Studios (and Dashboards) can be renamed by right-clicking the tab, as shown below:

Console

The Console window is used for writing and executing queries. The tables, plots, and widgets generated by those queries can then be used to build your workspace.

Queries

Simply speaking, a query is a way to ask the data system for information. For example, you can write a query to filter, group, and aggregate data; to join different tables; to create different views; or to build plots and other widgets.

There are two types of queries:

• Console-based queries are typed in the console or accessed via Git or other types of repository. Typically, console-based queries are used for one-off questions or data extractions and the development of new Persistent Queries. Tables and charts generated by console-based queries are added to the Code Studio window as new panels.
• Persistent queries use the same language as console-based queries. However, persistent queries can be saved, thereby enabling the automation of repetitive and/or timed operations. Persistent queries can also be shared with other individuals or groups. Tables and charts generated by persistent queries are accessible from the Panels menu. To learn more, see Panels and Creating Persistent Queries on the Web.

The following code block is an example of a query using the Deephaven Query Language. When executed in the console, it will open a table of stock market data and then filter that table to a specific date and Sym (AAPL):

newTable=db.t("LearnDeephaven", "StockTrades")
// This tells the system Look in the namespace called "LearnDeephaven",
// open a table called "StockTrades",
// and store the contents of the table in a variable named "newTable"
    .where("Date=`2017-08-25`") // This line filters the data to include only those rows on a given date.
    .where("Sym=`AAPL`") // This line filters the data to include only those rows that have the value AAPL in the Sym column.

newTable=db.t("LearnDeephaven", "StockTrades")\
    .where("Date=`2017-08-25`")\
    .where("Sym=`AAPL`")

# This tells the system Look in the namespace called "LearnDeephaven",
# open a table called "StockTrades",
# store the contents of the table in a variable named "newTable",
# filter the data to include only those rows on a given date,
# then filter the data to include only those rows that have the value AAPL in the Sym column.

The specifics about writing queries are covered elsewhere. However, as you can see, the language is very intuitive and easy to follow.

After the query runs, the generated table will appear in a new panel in the Code Studio window.

As shown above, the new table holds information pertaining only to trades of Apple (AAPL) securities on August 25, 2017.

After the query executes, a blue button will appear in the Console to show the name of the table that was just created. Clicking this button will refresh the table, or reopen the table if it had been subsequently closed.

Now that we have the table created, let's add another query to create a simple line plot using data from the previous table.

newPlot = plot("AAPL", newTable, "Timestamp", "Last")
     .xBusinessTime()
     .show()

from deephaven import Plot

newPlot = Plot.plot("AAPL", newTable, "Timestamp", "Last")\
     .xBusinessTime()\
     .show()

After the query executes, the plot is added as a new panel in the Code Studio window. As before, the blue button in the Console panel can be used to refresh or reopen the plot.

This is a very simple example. However, Deephaven is capable of much more complex analyses using much larger datasets.

Console Menu

The menu at the top of the Console window provides the following information and options:

Worker status

This shows the current state of the worker: Idle (hollow circle), Working (solid circle), or Disconnected (hollow red circle).

Query worker name

This is the name of the Deephaven "worker" assigned to this console.

Server IP Address

This is the server assigned to this process.

Script type

This tells users which programming language is being used. Options include Groovy and Python.

Memory (Heap)

This is the amount of memory assigned to process queries in this console session.

Show Table

This option becomes available if any console-based queries have generated a table. Selecting an item in the drop-down menu will bring it into focus. If a table has been subsequently closed, you can re-open the table by selecting its name from the drop-down list.

Show Widget

This option becomes available if any console-based queries have generated a plot or other widget. Selecting an item in the drop-down menu will bring it into focus. If a plot or widget has been subsequently closed, you can re-open it by selecting its name from the drop-down list.

More Actions

This More Actions includes the following options:

• Print Stdout - If selection, "Stdout" (standard output) will be included in the console history. If your script generates a large amount of logging to stdout, you may wish to turn this off.
• Auto Launch Panels - If selected, any console-based queries will automatically open in new panels in the Code Studio window. If unselected, tables or widgets produced from console-based queries will not automatically open as new panels. However, these can be opened by clicking the blue button under the query or from the Show Table/Show Widget menus.
• Close Panels on Disconnect - If selected, all open panels will be closed when the console disconnects, or when you exit the web browser. If unselected, panels will remain open upon disconnect or a new session.
• Upload Table from file - Opens the wizard to import table data from a file, such as a CSV. See below for more details.
• Disconnect - Disconnects the console from the query server and will return you to the New Console connection screen.

Upload Table from file

Tables from files can be imported in the UI, in one of two ways:

1. Selecting Upload Table from file from the console's More Actions menu:

A table name will be generated for you, but you can enter your own. You can also choose an alternative file format:

2. Drag a CSV file onto the console.

This feature supports the following file extensions:

• csv
• tsv (tab separated)
• tab (also tab separate)
• zip (containing one of the above)

Log

The Log panel tab is located beside the Console panel and presents a record (log) of what your query programming is doing, including data input/output, and processing. This feature is primarily designed for more advanced programmers, but the log can help all users find and resolve errors that may occur.

There are eight different logging levels in Deephaven:

1. FATAL - Details about severe events that will likely cause the query to abort.
2. STDERR - Presents details about events occurring in the stream where a program writes its own error data.
3. ERROR - Presents details about more harmful errors that are occurring even though the query is still running.
4. WARN - Presents details about potentially harmful events that are occurring.
5. STDOUT - Presents details about events occurring in the stream where a program writes its output data.
6. INFO - Presents general information showing the progress of the query
7. DEBUG - Presents more specific information that should be useful for debugging
8. TRACE - Presents very fine-grained details about events as they occur.

Log Search

The Log search allows you to search the logs for specific keywords/phrases. Search will find partial matches and is case-insensitive by default. However, you can choose alternative search options in the search field, including:

• Match Case
• Match Whole Word
• Use Regular Expression

If the search term is present in the log, the keyword will be highlighted (Note: this can be turned off and on with the Find in selection button in the search toolbar). If there are multiple instances of the keyword, you can either scroll through the log window or click on the left and right arrows in the search toolbar to jump directly to the next instance of the search term. The log scrollbar will also indicate each instance of your search term, as shown below:

Log Settings

Clicking the "gear" icon at the right of the Log panel will open the Settings menu:

The Display Log Levels drop-down menu allows you to choose the level of details presented in the log window when a query is running.

• By default, only five levels are shown, including FATAL, STDERR, ERROR, WARN, and STDOUT.
• Each log level may be turned on or off individually.
• Selecting Toggle All will show every detail of logging presented in Deephaven.
• Select Reset to restore the default levels.

Log Context Menu

Right-clicking within the log panel opens a context menu with the following options:

Copy

Copies either the line from where the right-click was generated, or selected lines, to the clipboard.

Command Palette

Opens the Command Palette, which provides an exhaustive list of commands for editing, including keyboard shortcuts for the most common operations.

Command History

The Deephaven Command History can be used to review, retrieve and rerun queries that were used in previous queries. By default, the Command History appears to the right of the console window, as shown below. However, you can drag and drop its tab to another location in your workspace.

The most recently used commands are shown at the bottom of the list. When the panel is full of previous commands, scrolling enables users to find older commands.

The history will show as much of the line as fits the window width. Hovering over that line will open a flyout menu to the left with the full query and its runtime.

You can highlight one or more commands in the History panel by clicking or shift clicking on the commands. If you double-click on a command in the panel or hit enter, that command will be copied to the console.

At the top of the Command History are three options:

• The Search field can be used to find a specific keyword in a previous command.
• Open selected commands in the console copies the selected commands into the console.
• Open selected commands in the notebook copies the selected commands into the notebook panel, or if none are present, creates a new notebook panel with the commands.

Command History Context Menu

If you right-click on a command in the Command History panel, the following options are presented:

• Copy - Copies the selected command(s) to the clipboard.
• Run in Console - Runs the selected command(s) in the console.
• Send to Console - Copies the selected command(s) into the console, where they can be edited before the user runs the query.
• Send to Notebook - Copies the selected command(s) to the most recently opened notebook panel in the console.
• New Notebook... - Copies the selected command(s) into a new notebook panel.

Notebooks can be added to your workspace using the File Explorer or the PQ Explorer.

File Explorer

The File Explorer tab at the top right of the Code Studio provides options to create and open Deephaven notebooks.

PQ Explorer

The PQ Explorer tab at the right of the Code Studio lists all the persistent queries available to you, as well as some status information.

As shown above, the green circle indicates that the persistent query named "DemoQuery" is active, while "DemoQuery1" is not. Selecting any query automatically opens its script in a new notebook panel. When a notebook generated from the PQ Explorer is saved, any edits will be automatically applied to that persistent query script and reflected in the Script tab of the Persistent Query Configuration Editor.

Clicking the drop-down arrow within the Save button provides options to either restart the query immediately, or apply the changes upon the next scheduled restart:

If the query is Git-hosted, the script will open as a read-only Notebook. The overflow menu provides the option to copy the script into a new editable notebook.

Autocomplete

As you write queries in the Deephaven web console, the autocomplete feature speeds up your query writing by suggesting possible values. As you type in the Console window, a drop-down list will appear with values relevant to your query, such as available namespaces, table names, and table operations.

For example, the animation below shows Deephaven's autocomplete feature generating suggestions (in drop-down menus) as portions of the query are typed.

To select one of the suggested values, scroll to the desired value, and click or press Return/Enter on the highlighted item.