Run scripts on Deephaven Enterprise workers

The run_local_script command runs a local Groovy or Python script on a remote Deephaven Enterprise worker. It sends the script text to the worker for evaluation, using a temporary Persistent Query.

Use run_local_script to run one-off or scheduled scripts, such as merge scripts, from a shell or a scheduler without opening a Code Studio. The command exits when the script finishes, so it fits cron jobs and other automation. It returns a nonzero exit code when the script fails, so a scheduler can detect the failure.

Note

This page describes /usr/illumon/latest/bin/run_local_script, which runs scripts on Core+ workers by default. The Legacy engine has a separate iris_exec run_local_script utility with different options. See Running local command line scripts (Legacy) and How the two utilities differ.

Usage

Prerequisites

  • A python executable (Python 3) is on the PATH of the host that runs the command. If the host has only python3, run the command as python3 /usr/illumon/latest/bin/run_local_script.
  • The authenticated user is permitted to start a worker on the target query server. See private key login for key-based authentication.
  • The Deephaven Python wheel (deephaven_coreplus_client) is in /usr/illumon/coreplus/latest/py/wheel, which is where Core+ installations place it. The command reads the wheel only when it creates a temporary virtual environment.

Python environment

run_local_script requires the Core+ Python client. When the current Python environment does not include the client, the command creates a temporary virtual environment, installs the client wheel from /usr/illumon/coreplus/latest/py/wheel, and deletes the environment when the script completes, unless you set DEEPHAVEN_CLIENT_VENV_DIR.

To reuse one virtual environment across runs, set DEEPHAVEN_CLIENT_VENV_DIR to a directory that the user running the command can write to. The first run creates the virtual environment in that directory, and later runs reuse it instead of rebuilding it:

Examples

The examples use the authentication options and options that follow.

Run a Python script

Run a Groovy script on a specific query server

Connect to a remote server

Use --connection-json to specify a remote server:

Increase heap size and pass JVM arguments

Write logs to a file and extend the worker timeout

A scheduler such as cron can run this form of the command for a nightly job:

Authentication

Exactly one authentication option is required:

OptionDescription
-u / --user <username>Authenticate with a username. Prompts for a password.
-k / --key-file <path>Authenticate with a private key file.
-a / --samlAuthenticate using SAML.

Use a private key file for unattended runs, because the --user option prompts for a password.

Options

OptionDescriptionDefault
-s / --script <path>Path to the local script to run. Required.—
-c / --connection-json <url>URL to the server's connection.json file.Local server configuration
-q / --query-server <name>Name of the target query server.First server in the Controller's DB server list
-m / --heap-size <GB>JVM heap size in GB for the worker. Must be a whole number.4
-e / --worker-kind <kind>Worker engine type. Use DeephavenCommunity for Core+ workers, or a custom WorkerKind configured by your system administrator.DeephavenCommunity
-j / --jvm-arg <arg>Additional JVM argument for the worker. May be repeated. Use the = form, as in --jvm-arg=-Xss4m, because the value starts with a dash.—
-t / --timeout <seconds>Worker creation timeout in seconds.60
-l / --log-file <path>File to which the command appends log messages from the worker.stdout
-n / --no-create-venvDo not create a temporary virtual environment. Fails if the Deephaven client is not already installed.Off

Limitations

  • The script runs on the worker, not on the machine that runs the command, so file paths in the script resolve on the server's file system.
  • The command sends only the text of the one script file. It does not send other local files that the script might reference.
  • Script paths and other arguments that contain spaces or shell metacharacters fail when the command creates a temporary virtual environment, because it passes its arguments to the second run through a shell without quoting them. Use --no-create-venv or avoid such characters.
  • The command does not accept script arguments. To parameterize a script, generate the script file before each run.

Output and exit codes

The command streams the worker's log messages to standard output while the script runs, one message per line in the form <timestamp> | <level> | <message>. Use --log-file to append the messages to a file instead. If the script fails, the command also prints a Python traceback.

The command exits with 0 on success and 1 if the script throws an exception or fails to execute.

How the command runs a script

  1. The command reads the script file on the machine where you run it. It infers the script language from the file extension: files ending in .groovy run as Groovy, and all other files run as Python.
  2. The command connects to the server that connection.json describes and authenticates.
  3. It starts a new worker for a temporary Persistent Query named Run-Local-Script-Query. The worker uses the requested heap size, worker kind, JVM arguments, and query server.
  4. It sends the script text to the worker and waits for the script to finish.
  5. It closes the session. The temporary query ends when the script completes.

Temporary virtual environment

If the Python environment that runs the command does not already have the deephaven_enterprise.client package, the command creates a temporary virtual environment, installs the client wheel into it, and runs itself again inside that environment. The command deletes the environment when it finishes, unless DEEPHAVEN_CLIENT_VENV_DIR is set.

Installing the wheel also installs its dependencies, so the host needs access to a Python package index or a mirror.

Pass --no-create-venv to skip this step, for example when you run the command from an environment that already has the client installed.

How the two utilities differ

Deephaven has two utilities named run_local_script. They share a purpose but not their options.

Featurerun_local_script (this page)iris_exec run_local_script (Legacy)
Command/usr/illumon/latest/bin/run_local_script/usr/illumon/latest/bin/iris_exec run_local_script
Worker engineCore+ (DeephavenCommunity) by defaultLegacy
Script option-s / --script-s, placed after --
AuthenticationUsername, private key, or SAMLPrivate key (-k or IRIS_KEYFILE)
Server selection--connection-json and --query-server-h and -p (query server host and port)
Script variablesNot supportedPositional name value pairs bind global variables
Table data service, JVM profileNot supported-tds and -jp