Internal tables overview

All Deephaven installations include several tables for the system's internal use. These internal tables are updated by various Deephaven processes, contain details about processes and workers, and are intended to aid in troubleshooting and performance monitoring. The schemas for these tables should never be changed by the customer as this will result in Deephaven failures. All tables are stored in the DbInternal namespace.

Just like other tables, authorized users can run queries against these internal tables. In a standard installation, a user can query entries related to their username, while a superuser can query all entries in a table.

In most tables, there is a ProcessInfoId column, which is a unique identifier (a 128-bit randomly generated UUID) for the process that logged the event. When joining these tables together, the ProcessInfoId serves as a useful key to identify a worker or other process.

Deephaven Core provides several internal performance tables, which Core+ workers write persistently to the database. This enables performance analysis from another worker to avoid perturbing your workload, or to analyze a terminated worker.

The following snippet shows how to retrieve the raw internal tables for the current date. Typically, you should filter by ProcessInfoId to get relevant data. See Monitor query performance and Persistent Query State Log for analysis tools.

date = today(timeZone("ET"))

pel = db.liveTable("DbInternal", "ProcessEventLog").where("Date=date")
ael = db.liveTable("DbInternal", "AuditEventLog").where("Date=date")
wdl = db.liveTable("DbInternal", "WorkspaceData").where("Date=date")

pqsl = db.liveTable("DbInternal", "PersistentQueryStateLog").where("Date=date")
pqcl = db.liveTable("DbInternal", "PersistentQueryConfigurationLogV2").where("Date=date")
rul = db.liveTable("DbInternal", "ResourceUtilization").where("Date=date")

qpl = db.liveTable("DbInternal", "QueryPerformanceLogCoreV2").where("Date=date")
qopl = db.liveTable("DbInternal", "QueryOperationPerformanceLogCoreV2").where("Date=date")
upl = db.liveTable("DbInternal", "UpdatePerformanceLogCoreV2").where("Date=date")

pil = db.liveTable("DbInternal", "ProcessInfoLogCommunity").where("Date=date")
pml = db.liveTable("DbInternal", "ProcessMetricsLogCoreV2").where("Date=date")
ssl = db.liveTable("DbInternal", "ServerStateLogCoreV2").where("Date=date")

pel = db.live_table("DbInternal", "ProcessEventLog").where("Date=today()")
ael = db.live_table("DbInternal", "AuditEventLog").where("Date=today()")
wdl = db.live_table("DbInternal", "WorkspaceData").where("Date=today()")

pqsl = db.live_table("DbInternal", "PersistentQueryStateLog").where("Date=today()")
pqcl = db.live_table("DbInternal", "PersistentQueryConfigurationLogV2").where(
    "Date=today()"
)
rul = db.live_table("DbInternal", "ResourceUtilization").where("Date=today()")

qpl = db.live_table("DbInternal", "QueryPerformanceLogCoreV2").where("Date=today()")
qopl = db.live_table("DbInternal", "QueryOperationPerformanceLogCoreV2").where(
    "Date=today()"
)
upl = db.live_table("DbInternal", "UpdatePerformanceLogCoreV2").where("Date=today()")

pil = db.live_table("DbInternal", "ProcessInfoLogCommunity").where("Date=today()")
pml = db.live_table("DbInternal", "ProcessMetricsLogCoreV2").where("Date=today()")
ssl = db.live_table("DbInternal", "ServerStateLogCoreV2").where("Date=today()")

Worker Performance Tables

Workers write four performance-related tables, which can be analyzed with Performance Overview queries. These tables have internal partition indexes to allow the Performance Overview tool to read only necessary internal partitions. These tables are based on the Core performance tables, but include a Date partitioning column and process information including the process info id and user information.

Log Index Tables

Certain high-volume internal tables have associated index tables used to optimize administrative queries commonly performed against those tables. The Data Import Server produces these Index tables as it imports data, making it possible to identify the set of internal partitions that have data for a given key. For further details please see Indexing Intraday Partitions.

Index TableBase TablePartitioning Column(s)Key Columns
ProcessEventLogIndexProcessEventLogDateProcess, ProcessInfoId, AuthenticatedUser, EffectiveUser
ProcessTelemetryIndexProcessTelemetryDateClientInstance, ClientHost, ProcessInfoId, Worker, AuthenticatedUser, EffectiveUser
QueryOperationPerformanceLogCoreV2IndexQueryOperationPerformanceLogCoreV2DateProcessInfoId, PrimaryAuthenticatedUser, PrimaryEffectiveUser, WorkerName, ServerHost
QueryPerformanceLogCoreV2IndexQueryPerformanceLogCoreV2DateProcessInfoId, PrimaryAuthenticatedUser, PrimaryEffectiveUser, WorkerName, ServerHost
ServerStateLogCoreV2IndexServerStateLogCoreDateProcessInfoId, PrimaryAuthenticatedUser, PrimaryEffectiveUser, WorkerName, ServerHost
UpdatePerformanceLogCoreV2IndexUpdatePerformanceLogCoreV2DateProcessInfoId, PrimaryAuthenticatedUser, PrimaryEffectiveUser, WorkerName, ServerHost

The following snippet shows how to retrieve the raw index data for the current date.

date = today(timeZone("ET"))
peli = db.liveTable("DbInternal", "ProcessEventLogIndex").where("Date=date")
pti = db.liveTable("DbInternal", "ProcessTelemetryIndex").where("Date=date")
qopli = db.liveTable("DbInternal", "QueryOperationPerformanceLogCoreV2Index").where("Date=date")
qpli = db.liveTable("DbInternal", "QueryPerformanceLogCoreV2Index").where("Date=date")
ssli = db.liveTable("DbInternal", "ServerStateLogCoreV2Index").where("Date=date")
upli = db.liveTable("DbInternal", "UpdatePerformanceLogCoreV2Index").where("Date=date")

peli = db.live_table("DbInternal", "ProcessEventLogIndex").where("Date=today()")
pti = db.live_table("DbInternal", "ProcessTelemetryIndex").where("Date=today()")
qopli = db.live_table("DbInternal", "QueryOperationPerformanceLogCoreV2Index").where(
    "Date=today()"
)
qpli = db.live_table("DbInternal", "QueryPerformanceLogCoreV2Index").where(
    "Date=today()"
)
ssli = db.live_table("DbInternal", "ServerStateLogCoreV2Index").where("Date=today()")
upli = db.live_table("DbInternal", "UpdatePerformanceLogCoreV2Index").where(
    "Date=today()"
)