Admin glossary

This page defines common terms used throughout the security, TLS, and other system administration docs, and explains how those terms relate to Deephaven installation and security.

For the current certificate, trust, routing, and certificate-rotation model, use TLS certificates as the primary operational reference.

DNS

The Domain Name System is what maps a hostname like localhost or www.example.com to an IP address.

On Deephaven servers, it is important that all members of the Deephaven cluster agree about what IP belongs to what hostname.

FQDN

A fully qualified domain name (FQDN) refers to a complete.hostname.address, and can be as simple as localhost.

In Deephaven, all Server processes in a cluster will communicate directly with other server processes using TLS connections to each machine's FQDN:<process port> machine identity.

For all Kubernetes installations and any Podman or native installation that opts in to using Envoy, the only FQDN and port your client machines will connect to is Envoy (default port 8000).

When Envoy is not used, a Deephaven client connects directly to each server's FQDN and configured port.

For how FQDNs affect certificate SAN requirements and routing, see Deephaven machine identity and the SAN and hostname verification section of the TLS certificates guide.

Machine identity

A Deephaven machine's machine identity is the FQDN (hostname) that Deephaven clients or other Deephaven services use to connect to a specific Deephaven endpoint.

For TLS verification, the machine identity is the FQDN used for hostname verification and must appear in the leaf certificate's Subject Alternative Name entries for all server endpoints.

All machines in a Deephaven cluster must resolve a given machine identity consistently; inconsistent DNS resolution can cause internal TLS and authentication flows to fail.

See Deephaven machine identity for details about how machine identity is assigned.

Deephaven client

A Deephaven client process is any Java, Python, JavaScript, C++ or R program that connects to a Deephaven server to access data, execute code or perform some administrative operation.

When Envoy is used, all Deephaven client connections to the server will pass through Envoy's FQDN:<envoy port> (default 8000).

Kubernetes installations always use Envoy, while Podman and native installations support opting in to using Envoy, but also allow direct connection from Deephaven clients to Deephaven services.

A Deephaven client should not be confused with a TLS client, which can be any Deephaven client or server process that communicates using TLS.

Deephaven services

Within a Deephaven cluster, there are one or more machines running one or more "Deephaven services" or "Deephaven processes".

Deephaven services all run as one of three Deephaven service accounts, by default irisadmin (admin processes), dbmerge (data writing processes) and dbquery (data reading processes).

Whenever a document refers to a "Deephaven service", it specifically refers to a Deephaven-authored Java process which is started and monitored by the Monit process manager. This includes all Administrative services, Data-accessing services, and Utility services.

Third-party Infrastructure services like Etcd and Envoy may run on Deephaven servers, but they are referred to by their own names rather than as "Deephaven services."

Deephaven service accounts

Deephaven services run as dedicated Operating System service accounts rather than as end-user accounts.

The three primary Deephaven service accounts are used in all installation types:

These accounts are collectively the main Deephaven service accounts and apply to Kubernetes, Podman, and native installations.

The following shared groups also apply across installation methods:

Note that all three of the primary Deephaven service accounts will have primary membership in DH_SHARED_GROUP, with secondary membership in DH_SHARED_QUERY_GROUP and their own "private group", defined above.

Some additional service accounts are specific to native installations:

For property definitions, defaults, and sudo/group requirements, see Custom user accounts.

Deephaven user

A Deephaven user is defined by user and group records stored and managed using Deephaven ACLs, and is not related to Operating System user accounts or Deephaven service accounts.

A Deephaven user logs in to Deephaven through the Authentication server, which is often accessed through the Web API service. Once the user has proven their identity, they are issued an authentication token which defines their logged in username, their operational username, and a secret token with a short expiration (default of 10 minutes).

Using their authentication token, the Deephaven user can make a wide variety of requests to the server, with each request authorized based on ACL permissions granted to the user, or any special groups the user is a member of.

When a Deephaven user requests to execute code, that code is executed on the host system by a Deephaven service account (typically dbquery, maybe dbmerge and never as irisadmin), with all authorization provided by the Deephaven user's authentication token.

Deephaven property

The primary means to configure a Deephaven cluster is through property files.

Property files use a robust format for defining scoped key-value properties. The files are stored in Etcd and are parsed and exposed to all Deephaven Services via the Configuration Server.

Whenever any document says to "set a property", unless otherwise specified, this means to edit a property file.

Administrative services

In every Deephaven installation, all authentication, authorization, system configuration, and Persistent Query execution is controlled by a set of "Administrative Processes":

• Configuration server
  • All processes retrieve cluster configuration from this process.
  • Most system configuration is changed by communicating with the "config server" via the dhconfig command.
• Authentication server
  • All authentication requests and login flows route through the Authentication Server.
  • All Deephaven processes must be able to reach an Authentication Server to obtain and renew credentials.
• Persistent Query Controller
  • Monitors Persistent Queries and directs Query and Merge dispatchers to run worker processes.
• ACL Write Server.
  • Enables authenticated clients to update Deephaven ACLs if the requesting user has sufficient permission.
• Web API Service
  • Serves the Deephaven Web IDE static files and Deephaven OpenAPI endpoint.
  • Serves the Core+ connection.json file which describes the Deephaven cluster topology.
  • Serves the Legacy Client Update Service for deprecated Swing clients and Legacy Python clients.
  • Negotiates client user session logins, auth token refreshes, and client queries about system configuration.

Data-accessing services

The finer nuances of controlling read and write access to user data are better covered at a high level in the Data lifecycle guide, and in verbose detail in the ACL guide.

With respect to security, we only need to consider three specific "data-accessing processes":

• Query dispatchers (dbquery process)
  • Read-intensive Persistent Queries and console workers which can execute user Groovy or Python code.
  • Allows the least-trusted users access to historical and live data sources.
  • Run query dispatchers in isolated environments with read-only access to filesystem data.
  • You may choose to give some or all query users access to writable user tables using Deephaven group ACLs.
• Merge dispatchers (dbmerge process)
  • Merge workers run with filesystem permission to write to shared data directories.
  • Starts Persistent Queries and console workers that can execute user Groovy or Python.
  • May import very many rows of user data in a short period of time (typically via CSV or JDBC).
  • May steadily write very many rows of data over a long period of time.
  • Restrict Deephaven user access to trusted users who are members of the iris-data* ACL "special groups".
• Data Import Servers (db_dis process)
  • Listens for data sent by Log tailer processes to write data in "near realtime" to "Intraday" or "live" data locations (typically /db/Intraday).
  • Writes very many rows of data from multiple sources over a long period of time.
  • When using multiple Data Import Services or merge dispatchers to distribute load, it is important to share network filesystems securely.

For secure, high-performance systems you may want to run one or more of each data-accessing process on a different machine, while more economical clusters often run db_dis and dbmerge on the same server. For simple test machines, all processes can run on a single machine, with Deephaven ACLs and OS file system permissions used to prevent unauthorized access to data.

Utility services

Some Deephaven processes exist solely to support collecting and distributing data in the system.

• Log aggregator service
  • Listens for events logged by all other processes and writes data as a "binary log file".
• Log tailer
  • Watches for "binary log files" to be written, to forward that data to a Data Import Server so it can be written to Intraday/live data locations on disk (typically /db/Intraday).
• Table Data Cache Proxy
  • Caches table data locally for faster data access.
• Status dashboard
  • Exposes Deephaven metrics to Prometheus, for external monitoring.

All installation methods, by default, run a Log Tailer process on all machines and in all containers to ensure that all binary log files written to disk are forwarded to the Data Import Server.

For Kubernetes, all processes are managed automatically, and run in their own pod(s). The log tailer runs in a sidecar container.

For native installations, use cluster.cnf to assign processes to specific machines. Native installations automatically add a Log Tailer, a Log Aggregator Service, and a Table Data Cache Proxy by default, but this can be manually disabled via node roles in cluster.cnf. The only exception to this rule is for Etcd-only or Envoy-only nodes; if no other Administrative or Data-accessing services are running, no log tailer or log aggregator will be added by default.

Infrastructure services

Deephaven relies on some third-party binary processes which may or may not run on your Deephaven servers.

• Etcd
  • Stores all global cluster configuration data, ACLs and runtime system state.
  • Must be accessible to all other Deephaven services.
  • Need not be accessible to any Deephaven client processes.
  • For optimal performance and security, run Etcd on its own machine (with a fast-writing disk) and no other processes.
• Envoy
  • Simplifies firewalls by exposing a single hostname and port to all client processes.
  • For optimal performance, co-locate Envoy with the Administrative services.
• MySQL
  • The legacy datastore used only for table and data ACLs.
  • Superseded by Etcd-based ACLs since 2023.
  • No longer supported after the Grizzly+ release.

For Kubernetes, all services run in their own pod. Envoy and etcd are always used, MySQL is never used. For Podman, Etcd and, optionally, Envoy run within the single dh-infra pod. MySQL is never used. For native installations, the placement of every process is configurable via cluster.cnf. MySQL and Envoy are unused by default.

Deephaven installation type

Deephaven can be installed on Kubernetes, Podman or natively installed on a Linux host.

Choosing an appropriate installation method depends on many factors that do not belong in a glossary. Contact a Deephaven representative to plan an installation method that works for you.

TLS concepts

These entries define the TLS terms used throughout the docs. For Deephaven-specific certificate requirements, trust boundaries, routing models, and certificate rotation workflows, see TLS certificates.

Signing data

Signing proves that data came from the holder of a particular private key and that the data was not modified after it was signed.

In TLS, signatures are used so a server can prove it owns the private key corresponding to the public key in its certificate.

Encrypting data

Encryption protects data in transit from eavesdropping.

In TLS, the connection is encrypted so credentials, tokens, and other sensitive application data can be sent safely.

Deephaven provides multiple TLS boundaries between servers and clients, and encrypts sensitive data in transit.

TLS authentication versus application authentication

Authentication can happen at different layers. In TLS, it usually means the client verifies the identity of the server. At the application layer, it means a user or client proves its identity to obtain an authentication token.

In a typical HTTPS/TLS connection:

• The server presents a certificate to prove its identity.
• The client verifies that certificate against its trusted CAs and checks that the hostname matches.
• Once the TLS handshake completes, the connection is encrypted using negotiated session keys.
• Credentials, tokens, and other sensitive application data can then be sent over that encrypted connection.

Private key

A private key is secret cryptographic material that must be protected.

In TLS and certificate-based identity, private keys are used to:

• sign data so others can verify it came from the key holder (for example, proving the server owns the certificate it presents);
• decrypt data in some TLS configurations (depending on the negotiated key exchange); and
• act as the secret half of a public/private key pair.

A server’s public signing certificate is only useful when paired with its corresponding private key.

Public key

A public key is the shareable half of a public/private key pair.

Public keys are used to:

• verify signatures produced by the matching private key; and
• encrypt data intended for the private-key holder in some cryptographic schemes.

In TLS, the public key is embedded in the server's certificate so clients can verify signatures and validate server identity.

TLS server

TLS servers are any process that listen for HTTPS connection on a given URL and port.

The server process signs all requests with its configured TLS signing certificate and private key.

During the TLS handshake, the TLS server provides the public certificate the TLS client must use to validate the connection, and proves the server's identity using the private key, so the client and server can securely exchange asymmetric encryption keys to encrypt data in and out of the server.

TLS client

A TLS client is any application that communicates with another process (typically using https) over a given hostname and port.

Both Deephaven clients and Deephaven services act as TLS clients when communicating with Deephaven services, or the etcd backend datastore.

A TLS client will reject any https connection when the TLS handshake is signed by an invalid or untrusted certificate.

Certificate (X.509)

An X.509 certificate is a signed document that binds:

• an identity (the subject), and
• a public key,

plus metadata and constraints (validity period, key usage, etc.).

Certificates are commonly used in TLS for:

• server authentication (the TLS client confirms it is talking to the intended server), and optionally
• client authentication (mutual TLS).

PKCS#12

PKCS#12 (also known as PFX or P12) is a binary archive file format for storing cryptographic objects such as:

• private keys,
• certificates (including certificate chains), and
• trusted CA certificates.

PKCS#12 files are commonly used for:

• keystores (storing private keys with their associated certificates),
• truststores (storing trusted CA certificates), and
• importing/exporting certificates and keys between systems.

In Deephaven:

• All Deephaven Java services sign all TLS requests using PKCS#12 keystores.
• Generated truststores may be provided in PKCS#12 format (for example, truststore-iris.p12).
• PKCS#12 files are password-protected and should be treated as sensitive credentials.
• Common file extensions are .p12 and .pfx.

See File/container formats: PEM, DER, PKCS#12 for related formats and Keystore (private keys + certificates) for usage details.

SAN

SAN (Subject Alternative Name) is the certificate extension that carries DNS names (and/or IP addresses) the certificate is valid for.

In Deephaven, all hostname verification uses SAN. Common Name is no longer used.

Practically speaking, this means all TLS certificates for a given machine must contain a DNS:your.fqdn.here entry for the hostname used to reach that machine. See SAN and hostname verification for the current Deephaven rules.

Hostname verification

Hostname verification is the check that the hostname you connected to (for example, deephaven.example.com) matches a name in the certificate’s SAN extension.

Typical failure symptoms include hostname-mismatch errors indicating that the FQDN the client used is not listed in the certificate's Subject Alternative Name. Trust-chain failures such as PKIX path building failed are related TLS problems, but they indicate a broader trust or certificate-chain issue rather than the hostname-verification check itself. For Deephaven-specific behavior, see SAN and hostname verification.

Signing certificate

A signing certificate is the X.509 certificate presented by a server in TLS so the client can verify the server’s identity.

In Deephaven, you must supply a signing certificate chain for the Envoy load balancer, or signing certificates for every machine in the Deephaven cluster. See Required TLS certificates.

Issuer certificate

An issuer CA certificate is a certificate that is used to create and sign other certificates.

An issuer certificate be either a root CA certificate or an intermediate CA certificate, which must be issued by another intermediate or root CA certificate.

For any TLS certificate, the Issuer field identifies the Subject field of the certificate that issued it.

For a TLS connection to be trusted, the TLS server's presented signing certificate and the TLS client's TLS truststore must form a complete TLS chain of trust. Each certificate in the signing certificate chain must be followed by the optional intermediate certificate that issued it, and the final certificate in the chain must have been issued by a certificate contained in the client's truststore certificate bundle.

Issuer certificates do not contain the Subject Alternative Name needed to perform hostname verification; only the leaf signing certificate contains the necessary extensions to verify a particular FQDN.

Self-signed certificate

A self-signed certificate is a certificate whose issuer and subject are the same, meaning it is signed by its own private key.

While self-signed root CA certificates are valid and trusted, a self-signed leaf signing certificate cannot be trusted without manually adding that certificate to operating system, browser, JVM, Python and Deephaven truststores.

TLS chain of trust

A TLS chain of trust is the sequence of certificate relationships that enables the TLS verification process to start with a presented leaf certificate and follow the issuer certificate signatures upward until it reaches a trusted root CA certificate.

Each certificate in the chain is issued by the next certificate above it, so the verifier checks the leaf certificate’s issuer certificate, then that certificate’s issuer, and so on. The chain is valid only if every certificate in the path can be linked through a matching issuer/subject relationship, and the final certificate is present in the client’s truststore certificate bundle.

Leaf certificate

A leaf certificate is the X.509 certificate used by a TLS server to sign TLS requests. It is the certificate at the end of the TLS chain of trust, not a root CA certificate or intermediate CA certificate.

In Deephaven client-facing TLS, the leaf certificate must include the correct Subject Alternative Name (SAN) entries for the hostnames clients actually use. Hostname verification is performed against those SAN entries, not the certificate common name.

Certificate Authority (CA)

A Certificate Authority (CA) issues (signs) certificates.

• A root CA is typically self-signed and is the anchor for trust.
• An intermediate CA is issued by a root CA (or another intermediate) and issues other intermediate or leaf certificates.

All TLS connections verify that the server's leaf signing certificate presented during the TLS handshake was ultimately issued by a trusted root CA certificate.

Between the client truststore and the server's signing certificate, the TLS handshake can only complete if a complete TLS chain of trust can be established.

Intermediate certificate

An intermediate certificate is a CA certificate that is issued by another CA and is used to issue leaf certificates.

Root CA certificate

A root CA certificate is a CA certificate that is trusted as an anchor.

Root CA certificates are often self-signed and are used to issue Intermediate certificates or Leaf signing certificates.

While you can include the Root CA in your Signing certificate chain, it is more common to find the root CA from the TLS client's configured truststore.

Certificate chain (leaf + intermediates)

A certificate chain is the set of certificates needed for a verifier to build a path from the presented leaf certificate back to a trusted root CA.

Operationally:

• Servers commonly present the leaf certificate plus any required intermediate certificates.
• Clients validate the chain and then decide whether the root is trusted (via a truststore).

TLS chain verification

TLS chain verification is the process of checking that:

• each certificate in the chain is signed by the next issuer certificate;
• the chain is valid for time and usage constraints; and
• the chain terminates at a root CA that the verifier trusts.

Hostname verification is a separate check which only considers whether the leaf certificate has a Subject Alternative Name matching the hostname used in the TLS connection.

Truststore

A truststore is a set of certificates (typically CA roots, sometimes intermediates) used to decide whether a presented certificate chain is trusted.

In these docs, “truststore” may refer to either a Deephaven-managed truststore generated from explicit CA material or a system-managed OS/browser/JVM/Python trust source.

Examples include:

• a Deephaven-managed truststore generated during installation or upgrade when you provide a CA bundle of trusted PEM certificates;
• an OS/browser/JVM truststore (system-managed)
  • For OS trust, Linux Operating Systems have a directory of individual certificates, and a single bundled pem certificate.
    • To find your OS truststore, OpenSSL provides a reliable symlink: ls -l /etc/ssl/cert.pem
    • Your system administrator can update Ubuntu truststores via update-ca-certificates and RedHat OSes via update-ca-trust
  • For JVM trust, your vendor may use OS trust, or a cacerts file installed with the JVM.
  • For Python trust, most libraries embed their own *.pem in their site-packages.
  • For browsers, consult your system administrator before making any changes.

External TLS trust

External TLS trust is the trust relationship between a Deephaven client and the Deephaven endpoint it reaches.

When Envoy fronts Deephaven, the client validates the Envoy endpoint certificate. When Envoy is not used, the client validates the certificate presented directly by the Deephaven service it connects to.

See TLS trust boundaries and request routing with Envoy for the current Deephaven model.

Internal TLS trust

Internal TLS trust is the trust relationship Deephaven services use when they connect to each other.

Kubernetes has long had a clear internal-versus-external TLS split. In Grizzly+, native installations formalize the same model more clearly through per-machine certificates, stricter internal trust, and optional generated internal certificates.

See TLS trust boundaries and internal TLS trust for the current Deephaven model.

External truststore

An external truststore is the truststore used for client-facing TLS trust.

In native installations that generate client truststores from explicit CA material, this is the familiar truststore-iris.p12 truststore file. In Deephaven properties, tls.truststore refers to the client or default truststore used for external trust.

See external TLS trust and optional truststores for more details.

Internal truststore

An internal truststore is the truststore Deephaven services use for internal service-to-service TLS trust.

In native installations that generate internal certificates, this truststore is separate from the external truststore and is referenced by the internal.tls.truststore property. This allows internal trust to be narrower than client-facing trust.

See internal TLS trust and optional truststores for the current Deephaven behavior.

Trust and truststores

Trust is the decision by the client/application that a certificate chain is acceptable.

A client generally trusts a server certificate if:

1. the certificate chain is valid (not expired; signatures verify; required intermediates are present),
2. the certificate is allowed for its intended usage (key usage / extended key usage),
3. hostname verification succeeds (SAN matches), and
4. the chain terminates at a root CA that the client trusts.

A truststore is the database of trusted root CA certificates (and sometimes intermediate certificates).

System CA

System CA means trust comes from the default OS/browser/JVM/Python truststores on the TLS client machine, such as the system’s installed root CA certificates.

Practically:

• TLS signing certificates are verified based on the CA certificate bundles trusted by the TLS client machine's OS/browser/JVM/Python truststore.
• If your certificates are issued by a private/corporate CA, that CA must be installed into the relevant system/JVM/browser truststores on client and/or server machines.
• Trusted certificate rotation is handled by your system administrator.

Customer CA

A Customer CA certificate bundle is used when your Deephaven installation is configured to use an explicit CA certificate bundle to act as the trust anchor for TLS connections. When a Customer CA is used, the Deephaven system administrator will supply an explicit set of root CA certificates which must be the issuer certificate that completes the TLS chain of trust for all TLS signing certificates.

• Native installations provide a customer CA via DH_CA_CERTIFICATE in cluster.cnf.
• Podman installations provide a customer CA via --ca-cert-file flag to start_command.sh.
• Kubernetes installations do not currently support explicit customer CA; a generated truststore will be used which contains the Envoy signing certificate in the deephaven-tls secret.

Practically:

• This is a common approach when certificates are issued by an internal PKI.
• The CA bundle you provide becomes the trust anchor used to generate Deephaven-managed truststores.
• When using a customer CA certificate bundle, Deephaven clients do not require updates to Operating System or JVM truststores.
• Changes to the customer CA bundle require restarting all client and server processes to load the update truststore file.

Generated CA

Generated CA means Deephaven-generated internal trust material used for internal service-to-service TLS.

Generated certificates are only used in Kubernetes installations, and native installations when Envoy is enabled and DH_GENERATE_CERTIFICATES=true.

Practically:

• Internal trust between Deephaven services is restricted to the generated trust anchor rather than the system truststore.
• External Deephaven-client-to-cluster TLS connections to Envoy or direct connections to Deephaven services never use the generated CA.
• Both the generated certificates and the internal trust material are managed by Deephaven and updated during upgrade or certificate rotation workflows.

System trust vs a custom truststore

Clients may rely on:

• System trust: OS, browser, JVM, or Python defaults.
• A custom truststore: an explicit set of trusted CA certificates.

If Deephaven is configured to use certificates issued by an internal corporate CA, that CA certificate must be added to system truststores or distributed through an explicit truststore.

• Deephaven supports one external client trust model at a time:
  • System trust, or
  • Deephaven-managed truststores generated from an explicit CA bundle provided during Deephaven installation and upgrades.
• When Deephaven-managed truststores are generated, the truststore files are replaced on disk, rather than merged with older trust material.
• The truststore passphrase file for all explicit truststore files is tls.truststore.passphrase.file=/etc/sysconfig/deephaven/trust/truststore_passphrase.

This is covered in depth in the optional truststores section.

Keystore (private keys + certificates)

A keystore is a file/container format used by some software stacks to store:

• private keys, and
• certificates (often including the chain).

It differs from a truststore, which stores certificates used for trust decisions (usually CA roots).

Certificate file formats

• PEM: base64-encoded text with -----BEGIN ...----- markers. Common for certificates and private keys.
• DER: binary encoding of the same ASN.1 structures as a PEM certificate.
• PKCS#12 (.p12/.pfx): a binary container commonly used for Java keystores/truststores.

All certificates supplied to a Deephaven installation are supplied in PEM format, and must meet Deephaven TLS certificate requirements.

To convert a DER certificate to PEM:

Deephaven installation processes convert input PEM certificates and private keys into output PKCS#12 keystore and truststore files.

All generated keystores and truststores are secured by base64 encoded passphrase files.

Private key login "keypair files"

Deephaven non-interactive private key login uses two related "public key" and "private key" files, to perform "challenge-response" authentication by signing a nonce with the private key and verifying it with the public key.

• Public key file (pub-*.base64.txt): contains only the Deephaven username and the public key. This file is safe to distribute and is used to authorize the user on the server.
• Private keypair file (priv-*.base64.txt): contains the Deephaven username, optional operateas, and both the public and private key material. This file is a credential and must be protected with restrictive filesystem permissions.

See Private key login key files (including Rotation and revocation) for file formats and operational guidance.

A Java or Python client uses a private keypair file by supplying its path via the Java system property AuthenticationClientManager.defaultPrivateKeyFile.