Skip to main content

How to install Python packages

This guide discusses how to install Python packages for use with Deephaven. Packages can either be installed for one-time use in a Deephaven instance, or packages can be added to the Deephaven Docker images so that they are available every time Deephaven is launched.

Once a package is installed, it can be imported and used like any other Python package. For an index of available Python packages and for more information on Python packages, visit the Python Package Index.

List all available Python packages#

The packages available to Python can be listed by using Python's help function. To see a list of all available Python packages, run the following command from a Python session.

help("modules")
Please wait a moment while I gather a list of all available modules... __future__ _weakrefset heapq runpy _abc _xxtestfuzz hmac sched _ast abc html secrets _asyncio aifc http select _bisect antigravity idna selectors _blake2 argparse imaplib setuptools _bootlocale array imghdr shelve _bz2 ast imp shlex _codecs asynchat importlib shutil _codecs_cn asyncio inspect signal _codecs_hk asyncore io site _codecs_iso2022 atexit ipaddress sitecustomize _codecs_jp audioop itertools six _codecs_kr base64 jdl smtpd _codecs_tw bdb jpy smtplib _collections binascii jpyutil sndhdr _collections_abc binhex json socket _compat_pickle bisect keyword socketserver _compression builtins lib2to3 spwd _contextvars bz2 linecache sqlite3 _crypt cProfile llvmlite sre_compile _csv calendar locale sre_constants _ctypes certifi logging sre_parse _ctypes_test cgi lzma ssl _curses cgitb macpath stat _curses_panel charset_normalizer mailbox statistics _datetime chunk mailcap string _dbm cmath marshal stringprep _decimal cmd math struct _distutils_hack code mimetypes subprocess _dummy_thread codecs mmap sunau _elementtree codeop modulefinder symbol _functools collections multiprocessing symtable _hashlib colorsys netrc sys _heapq compileall nis sysconfig _imp concurrent nntplib syslog _io configparser ntpath tabnanny _json contextlib nturl2path tarfile _locale contextvars numba telnetlib _lsprof copy numbers tempfile _lzma copyreg numpy termios _markupbase crypt opcode test _md5 cryptocompare operator textwrap _multibytecodec csv optparse this _multiprocessing ctypes os threading _opcode curses ossaudiodev time _operator dataclasses pandas timeit _osx_support datetime parser token _pickle dateutil pathlib tokenize _posixsubprocess dbm pdb trace _py_abc decimal pickle traceback _pydecimal deephaven pickletools tracemalloc _pyio difflib pip tty _queue dill pipes turtle _random dis pkg_resources types _sha1 distutils pkgutil typing _sha256 doctest platform unicodedata _sha3 dummy_threading plistlib unittest _sha512 email poplib urllib _signal encodings posix urllib3 _sitebuiltins enum posixpath uu _socket errno pprint uuid _sqlite3 faulthandler profile venv _sre fcntl pstats warnings _ssl filecmp pty wave _stat fileinput pwd weakref _string fnmatch py_compile webbrowser _strptime formatter pyclbr wheel _struct fractions pydoc wrapt _symtable ftplib pydoc_data wsgiref _sysconfigdata_m_linux_x86_64-linux-gnu functional-tests pyexpat xdrlib _testbuffer functools pytz xml _testcapi gc queue xmlrpc _testimportmultiple genericpath quopri xxlimited _testmultiphase getopt random xxsubtype _thread getpass re zipapp _threading_local gettext readline zipfile _tracemalloc glob reprlib zipimport _uuid grp requests zlib _warnings gzip resource _weakref hashlib rlcompleter Enter any module name to get more help. Or, type "modules spam" to search for modules whose name or summary contain the string "spam".

Install packages from within a Python script#

Python packages can be installed using pip from within a Python script or from within the Deephaven Python console.

warning

If this method is used with Deephaven Docker images, Python package installs do not persist after the Docker container exits. The package installation must be repeated each time the container is started.

Run the following command:

import os
os.system("pip install package_name")

In this example, we install the Python Pendulum package, import the package, and use it to print the current time in Paris.

import os
os.system("pip install Pendulum")
import pendulum
now = pendulum.now("Europe/Paris")
print(now)
2021-10-01T05:58:56.333757+02:00

Now you can use any pip installed Python package with Deephaven.

Install packages in a running Docker container from the command line#

Python packages can be installed using pip via docker exec.

warning

If this method is used with Deephaven Docker images, Python package installs do not persist after the Docker container exits. The package installation must be repeated each time the container is started.

Here docker exec is used to run a pip install on the running Deephaven Docker image, which is named core_grpc-api_1.

docker-compose exec grpc-api pip install Pendulum

After installing the Pendulum package, we can use it within our script to print the current time in Paris.

import pendulum
now = pendulum.now("Europe/Paris")
print(now)
2021-10-01T05:58:59.386858+02:00

Add packages to a custom Docker image#

In this section, we add the same Pendulum package from the previous section to a custom Dockerfile and reference it from Deephaven so that we can use it in more than one session.

In order to use packages more than once, you can create a custom Docker image, and then use that image in Deephaven. The steps for accomplishing this differ slightly depending on how you launch Deephaven. Let's start with the steps that are common between both.

Prerequisites#

Before a custom Docker image can be built, you must acquire the necessary base images. This process differs based upon how you launch Deephaven:

docker-compose pull

Create a custom Dockerfile#

To begin with, create a new directory. This directory should not be in a Deephaven deployment directory. You can name it whatever you'd like. For this guide, we'll name ours deephaven-custom.

mkdir deephaven-customcd deephaven-custom

Now, in this directory, create a file called Dockerfile. Dockerfile should use ghcr.io/deephaven/grpc-api as the base image and should contain a recipe for installing the new package. When adding Pendulum, it looks like this:

FROM ghcr.io/deephaven/grpc-apiRUN pip3 install pendulum

Create a custom Docker image#

Now that we have the Dockerfile in place, we need to create the custom Docker image. To do so, run a command from the directory with Dockerfile that looks like:

docker build --tag <user>/grpc-api-<custom> .

This will create a new Docker image called <user>/grpc-api-<custom>. For this guide, we will call the image guide/grpc-api-pendulum:

docker build --tag guide/grpc-api-pendulum .

When the command finishes running, you can see the new image in your system:

docker image ls

Reference the new image#

To put it all together, we now need to reference this new image in the docker-compose file we use to launch Deephaven. The file name depends on how you build and launch Deephaven:

In The Docker Compose file, there are three lines of text that look like:

services:  grpc-api:    image: <IMAGE_NAME>

The image used by default depends on how you build and launch Deephaven. Regardless, this line is where you need to insert your custom image name. Modify the image line to use your new image:

services:  grpc-api:    image: guide/grpc-api-pendulum:latest

Now, when you launch Deephaven again, you can use the package!

caution

When base images are updated by rebuilding source code or redownloading pre-built images, custom images must be rebuilt to incorporate the base image changes.

Related documentation#