Console

The Deephaven Console is one of the options used to write queries for data analyses. Console-based queries differ from persistent queries, which can be saved, thereby enabling the automation of repetitive and/or timed operations. Also, persistent queries can be easily shared with other individuals or teams.

Click Create Console. You will then be given the opportunity to rename the new console panel. After clicking OK, a new panel will open with a Console window at the top and the Log window at the bottom:

Console

Settings

When creating a new console, the following configuration options are presented:

• Query Server: This setting will default to the primary server in your installation of Deephaven, or the previously used server. If other servers are available and you are authorized to access them, they will be presented in the drop-down menu. Because every installation is unique, please consult your system administrator for additional information about the query server.

• Heap Size : This setting refers to the dynamic memory (RAM) you want Deephaven to use to process the instructions and the data needed to run this specific query. Memory listed in this setting is noted in gigabytes (GB). Because memory is a finite resource on the system, it usually best to start small when estimating the amount of memory needed to run the query, and then add more via this setting if needed. Please contact your system administrator for further information about the memory capacity of your installation.

• Session Type : Deephaven scripts can be written in Groovy or Python. When Pythoh is chosen from the drop-down menu, the default System Python environment will be used, unless otherwise specified. However, a given Python environment can be specified for use by providing extra JVM arguments, as laid out in Installing Python > JPY JVM Flags.

• Show Advanced Options: If you select the Show Advanced Options check box, additional setting options will appear in the console panel, as shown in the following image:

  • Extra JVM Arguments - This field allows users to access different Java utilities that are not included in core Deephaven installation. For example, one may want to run a different profiler or debugging processor. These items can be included in the Deephaven configuration by typing the extra JVM arguments in this field. For another example, you can set certain scripts to be loaded every time you start your console by adding new values to IrisDbGroovySession.initScripts with the following argument:

    IrisDbGroovySession.initScripts=ClientDbStrategies/groovy/core/core_utils.groovy:Research/groovy/user/my_analysis_utils.groovy

    Note: clicking the right-facing arrow will open the field in a pop-out for easier editing.

  • JVM Profile - This drop-down field allows users to select an alternative profile for new persistent queries. Remote processing profiles are used to specify JVM parameters and environment variables for persistent queries and interactive consoles. Several predefined profiles are available, such as CMS GC or G1 GC, and new ones can be defined using controller properties. To learn more, see Remote Processing Profiles in the System Administration Guide.

    

  • Enabled detailed GC logging - Garbage Collection (GC) is a JVM memory management program that frees unreachable memory by getting rid of objects not being used by a Java application. When the check box is selected, Garbage Collection information will be included in the server-side dispatcher logs.

  • Remote Debug of Worker - A debugger is a program that lets you step through your code one line at a time so you can detect errors. Options to implement the debugging program are provided as radio buttons below this heading, and include the following:

    • Disabled
    • Allow worker to use any free port
    • Use fixed port:
    • Connect to: When debugging is enabled, you can connect to the worker with a debugger. Alternatively, the worker can be connected to your IDE to initiate debugging. You may use a JDWP debugger of your choice, such as JetBrains IntelliJ.

  • Extra Environment Variables - Extra Environment Variables are used to pass additional configuration information to Deephaven. For example, the following variables direct Java to load shared libraries from /usr/lib/my-native-library, and change the query worker's log level to DEBUG:

    JAVA_LIBRARY_PATH=/usr/lib/my-native-library REMOTE_QUERY_PROCESSOR_LOGLEVEL=DEBUG

    Note: clicking the right-facing arrow will open the field in a pop-out for easier editing.

  • Operate As - If the check box is selected, this allows system administrators (and other authorized users) to run the console as a specific user.

  • Close tables on disconnect - If the check box is selected, all tables created in this console will be closed when the console is disconnected. If the check box is not selected, tables and charts created by that instance of the console will not be closed. However, there will not be any data displayed in the tables/widgets.

Console Menus

After adjusting the applicable settings, the console window will be ready for your queries.

The three available buttons open the menus discussed below.

Show Table

Clicking Show Table enables users to open any tables associated with this instance of the console.

When a query executed in the console creates a new table, that table is immediately presented in the lower panel. The name of that table will also be added to the Show Table drop-down menu. Use the Show Table menu to reopen any table or to create multiple copies of the table in the console.

Show Widget

If a query creates a widget in the console, the Show Widget button becomes enabled.

Use the Show Widget menu to reopen any table from this console instance, or to create multiple copies of the plot in the console. (Note: "Widget" is a term used to describe various user interface components. In Deephaven, a plot is a common form of widget.)

More Actions

• Upload Delimited File... enables the user to upload a CSV to Deephaven. Multiple delimited-type files can be imported, including Comma, Tab, and Pipe delimited options: However, any single character can actually be used by typing it into the "Delimiter" box.
• Show (or Close) History Panel reveals a partition on the right side of the console window, which show a list of commands used previously in the console. See Command History.
• Add Notebook opens a Notebook panel in the console. See Notebook.
• Open Notebook enables users to select and open previously saved notebooks. See Notebook.
• Add Log Panel will create a new log panel if you have closed the log panel for this console.
• Disconnect enables the user to disconnect from the server without quitting Deephaven altogether.
• Reconnect enables the user to reconnect to the server without having to restart the entire application.
• Close Tables on Disconnect closes all tables when Deephaven becomes disconnected from the server.
• Close Tables/Widgets closes all tables and widgets generated by this instance of the console.
• Thread Dump is an option most often used by Deephaven power users and programmers, as well as support. Similar to a stack trace, the thread dump shows a snapshot of the current state of threads running in the application, and can be useful when troubleshooting unexpected behavior.
• Send Thread Dump to Support provides users with the option of sending the thread dump to Deephaven Support. See Help

Console Keyboard Shortcuts

When writing queries in a Deephaven console window, it is common to reuse previously entered commands. For example, the "up arrow" key can be used to retrieve and reuse commands entered in the past. Pressing the Ctrl-R / Control-R key combo enables a search for a previously used command. The console and Notebook panels support the following keyboard shortcuts to make editing a bit faster, including a mix of Emacs keys and others. Default Java keys may also work.

Windows	MacOS	Action
Alt+Backspace	Control+Backspace	Deletes the word before the cursor
Alt+B	Control+B	Moves back a word, starting at your cursor
Alt+D	Control+D	Deletes the word starting at your cursor
Alt+F	Control+F	Moves forward a word, starting at your cursor
Ctrl+A	⌘+A	Selects the entire command. On a Window-based workstation, if your command is already selected, Ctrl-A selects the entire console. On a MacOS-based workstation, Command-A always selects the entire console.
Ctrl+C	⌘+C	Copies commands only from the Console. To copy all text (including prompts), right-click the selection and select Copy.
Ctrl+E / End	Control+E / End	Goes to end of the line (as with END)
Ctrl+F	⌘+F	Opens the search bar in any table
Ctrl+G / ESC	Control+G / ESC	Kills the entire line
Ctrl+K	Control+K	Deletes all text from the cursor to the end of the line
Ctrl+R	Control+R	Starts the reverse command history search, and moves back one command
Ctrl+S	Control+S	When used in reverse command history search, this moves forward one command
Ctrl+W	⌘+W	Selects the current word
Ctrl+Y	⌘+Y	Redoes the undone previous action(s)
Ctrl+Z	⌘+Z	Undoes the previous action(s)
Ctrl+Shift+Z	⌘+Shft+Z	Redoes the undone previous action(s)

Command History

The Deephaven History panel is used to review, retrieve and rerun queries that were used in previous queries. The History panel can be accessed from the More Actions menu or by right-clicking inside the console window and selecting Show History Panel from the context menu.

When visible, the History panel always appears to the right of the console window, as shown below:

• The most recently used commands are shown at the bottom of the list. When the panel is full of previous commands, scrolling enables users to find older commands.
• The "Search" field can be used to find a specific keyword in a previous command.
• When a multi-line command is used in the console window, only the first line of the command is presented in the History panel. The key combinations of Ctrl+Up/Down (Windows) or ⌘+Up Arrow/Down Arrow (macOS) can be used to cycle through multi-line commands.
• You can highlight a command by clicking once on the command. If you double-click on a command in the panel, that command will be copied to the console.
• If you highlight a command and then click Execute Command, that command will be copied to the console panel and immediately executed.
• If you highlight a command and then click Edit Command, that command will be copied to the console window, where you can further edit the command as needed.
• If you right-click on a command, the following options are presented:
  • Close History Panel: Closes the command history.
  • Show Selected Command Details...: Opens the entire command in a new window. This is especially useful for reviewing a multi-line command.
  • Copy Selected Command to Clipboard: Copies the highlighted command to the clipboard.
  • Create Notebook from Selected Command: Copies the highlighted command to a new Notebook panel. See below.

Notebook

The Notebook is a feature in which users can develop query components, stage draft queries, or simply store notes. The Notebook can be used as a "sandbox" in which queries can be developed and refined, whereas the Deephaven Console is ultimately where queries are executed.

The Notebook can be added to the user interface from the More Actions menu:

Notebook Toolbar

The Notebook toolbar presents several options, discussed below. Hovering over the buttons in the Notebook toolbar will show the name of the function and the corresponding keyboard shortcut to execute the function (when available).

• Run Notebook - Automatically copies all of the content in the Notebook window, and then executes the content in the Console.
• Run Selected Lines - Copies the content you have highlighted in the Notebook window and then executes those selected lines in the Console. This option runs the entire selected line (or lines), even if you have highlighted only a portion of the line.
• Save - Saves all of the content of the Notebook to a file on the user's system. When Save is clicked for the first time, you will be prompted to enter a filename and select the file location. Depending upon the content in the Notebook, the file is saved as .txt, Groovy or Python filetype. Clicking Save again overwrites the existing file with the current Notebook content. After the file is saved, the name shown on the Notebook tab changes to match the filename used during the save process. Note: Notebooks with unsaved changes include an asterisk (*) in the panel tab.
• Save As - Enables the user to save all of the content of a Notebook into a new file and/or to a new location on the user's system. This can be used to save previously saved Notebooks with a different name or to a different location.
• Open - This option enables the user to open a .txt, Groovy or Python file on their system. Once a file is selected, the content of that file is copied into the Notebook panel, and will replace the previous content (if any) in the Notebook. If there is unsaved content in the existing notebook, the following dialog box will prompt you for a decision:
• Save to Persistent Query - Enables the user to save the content of the Notebook to a persistent query. Select whether to save the content to a new Persistent Query or to update an existing Persistent Query.
  • If Create New Persistent Query is selected, the Persistent Query Configuration Editor window will open and prompt the user for additional information. The entire contents of the Notebook will be copied to the Script Editor for this persistent query, and can be further edited. Once the persistent query is saved, the name shown on the Notebook tab will change to match the name of the persistent query. Moreover, if the script was edited further in the Script Editor, the most recent version of the script will be shown in the Notebook. However, the original Notebook may be reopened to retrieve any of the original content that may have been overwritten.
  • If Update Existing Persistent Query is selected, the user will then need to select the name of the existing persistent query in Deephaven that should be updated. When OK is selected, the content of the Notebook will completely replace the content found in the existing persistent query selected. After the file is saved, the name shown on the Notebook tab will change to match the name of the persistent query used during the Save process. This feature works with persistent query scripts that are sourced from within Deephaven. Scripts sourced outside of Deephaven (e.g., Git, etc.) will not be updated.
• Load from Persistent Query - Enables the user to extract the contents of an existing persistent query in Deephaven and load it into the Notebook window. The Load from Persistent Query dialog window will show a list of the persistent queries for which you are an authorized editor. After you select the desired persistent query, the previous contents of the Notebook (if any) will be replaced with the script from the selected persistent query. The name of the Notebook tab will also change to match the name of the persistent query selected. If the Notebook contains unsaved content when you select Load from Persistent Query, you will be prompted for a decision to handle the unsaved content. This feature works with persistent query scripts that are sourced from within Deephaven. Scripts sourced outside of Deephaven (e.g., Git, etc.) will not be loaded.
• Search - Entering a value in the Search field will enable you to search for that value in the Notebook. When a searched value is found in the Notebook, the value will be highlighted. The up and down arrows to the right of the Search field enable users to move forward and backward in Search history.

Notebook Right-click Options

Additional options are presented when you right-click in the Notebook window. The options presented will depend upon where you right click in the window and if you highlight any content.

The provides shortcuts for quick code execution and testing, and options related to saving, opening, loading and creating external files and persistent query scripts:

• Go to Line… - This option opens the Go to Line Number dialog window, which allows you to go to a line or a column (optional) in the Notebook to easily jump to any location.
• Execute Selected Lines - Copies the entire line(s) of content you have highlighted in the Notebook window, and then executes that content in the Console.
• Execute Selection - Copies only the content you have highlighted in the Notebook window, and then executes that content in the Console.
• Execute Notebook - Copies all of the content in the Notebook window, and then executes the content in the Console.
• Execute Below Cursor - Copies all of the content in the Notebook window that is below the location of your cursor, and then executes that content in the Console.
• Execute Above Cursor - Copies all of the content in the Notebook window that is above the location of your cursor, and then executes that content in the Console.
• Save <filepath/filename> - This option is shown when the content in the Notebook was opened from a previously saved Notebook file on your PC.
• Save As - Saves the entire notebook to a new Groovy or text file on your PC.
• Open file - Enables user to open an external text or Groovy file from your PC.
• Update script for <script name> - This option is presented when the content of the notebook was derived from a previously existing persistent query. Updates the original persistent query with any changes made in the Notebook.
• Load script from <script name>- This option is presented when the content of the notebook was derived from a previously existing persistent query. Provides users the option of loading the content from a specific existing persistent query.
• Create New Persistent Query - Provides users the option of saving the existing content in the notebook to a new persistent query.
• Load from Persistent Query - Provides users the option of loading the content of an existing persistent query into the Notebook.
• Auto save (files only) - When a Notebook has been saved to a file, the Auto save (files only) option is activated. When the check box is selected, Deephaven will automatically save the file 3-4 seconds after a subsequent change is made.
• Read only - When this checkbox is selected, the Notebook cannot be edited.

Comment/Uncomment Content in a Notebook

Content entered in a Notebook (or the Persistent Query Configuration Script Editor) can be converted into Comments (or reverted from Comments) with the following key commands:

• Ctrl + / (Windows) or Command + / (macOS): Comment out the selected lines. It will uncomment them if every line starts with a comment "//".
• Ctrl + Shift + / (Windows) or Command + Shift + / (macOS): Create a block comment around the selected text. If a block comment is exactly selected from /_ to _/ then it will uncomment the selected text.

Autocomplete

In addition to console hot keys and the History panel, Deephaven offers an "autocomplete" feature for the following aspects in a query:

• Namespaces, Table Names
• Column Names for view and select
• Method Names
• Previous commands

For example, if you type t2=db.t(" and then press the Tab key, the console will first provide a list of installed namespaces. In the first example below, there are 12 namespaces listed.

When you click on a namespace value, it will be added to the query. Click Tab again and you will see the table names listed in that namespace. In the example below, the namespace SalesGlobal was selected. After pressing the Tab key, five table names appeared.

Click on the table name you want to use and it automatically gets added to the query, as shown below.

If there is only one option that fits the autocompletion, or if the possible values for autocompletion all have a common prefix, the console automatically fills in that option for you. If you don't type anything else, you can click on the completion (which turns blue upon mouse over) and it will perform that completion.

If there are more than five lines of available options, you can see all of the autocompletion options by clicking the "N more..." link, or you can type additional text to use for the autocompletion and then press the Tab key again.

Log

The Log window is located below the Console window,and presents a record (log) of what your query programming is doing, including data input/output, and processing. This feature is primarily designed for more advanced programmers, but the log can help all users find and resolve errors that may occur.

There are eight different logging levels in Deephaven:

1. FATAL - Details about severe events that will likely cause the query to abort.
2. STDERR - Presents details about events occurring in the stream where a program writes its own error data.
3. ERROR - Presents details about more harmful errors that are occurring even though the query is still running.
4. WARN - Presents details about potentially harmful events that are occurring.
5. STDOUT - Presents details about events occurring in the stream where a program writes its output data.
6. INFO - Presents general information showing the progress of the query
7. DEBUG - Presents more specific information that should be useful for debugging
8. TRACE - Presents very fine-grained details about events as they occur.

Log Search

Log Search allows you to search the logs for specific keywords/phrases. If present, the keyword will be highlighted in the log. If there are multiple instances of the keyword, you can either scroll through the log window or click on the up and down arrows to the right of the search field to jump directly to the next instance of the search term. You can also select RegEx if your search includes "regular expressions."

Foreground Levels

The Foreground Levels drop-down menu allows you to choose which level of logging detail brings the log panel to the foreground of the interface (assuming it is not already). For example, a user may want a FATAL or ERROR log notation to be shown immediately, whereas a TRACE notation would not require such attention.

Selecting All Levels will bring the Log panel to the front every time a log notation is presented in Deephaven. Selecting Default Levels will bring the log panel to the front only when the first five levels of log notations are posted, including FATAL, STDERR, ERROR, WARN, and STDOUT.

Display Log Levels

The Display Log Levels drop-down menu allows you to choose the level of details presented in the log window when a query is running.

Selecting All Levels will show every detail of logging presented in Deephaven. Selecting Default Levels will show only the first five levels, including FATAL, STDERR, ERROR, WARN, and STDOUT.

Text/Tabular

The Text/Tabular setting enables the user to set the style of the log to either text or tabular. See samples below.

Example of a Text Log

Example of a Tabular Log