Internal tables overview
All Deephaven installations include several tables for the system's internal use. These internal tables are updated by various Deephaven processes, contain details about processes and workers, and are intended to aid in troubleshooting and performance monitoring. The schemas for these tables should never be changed by the customer as this will result in Deephaven failures. All tables are stored in the DbInternal
namespace.
Just like other tables, authorized users can run queries against these internal tables. In a standard installation, a user can query entries related to their username, while a superuser can query all entries in a table.
In most tables, there is a ProcessInfoId
column, which is a unique identifier (a 128-bit randomly generated UUID) for the process that logged the event. When joining these tables together, the ProcessInfoId
serves as a useful key to identify a worker or other process.
Deephaven Core provides several internal performance tables, which Core+ workers write persistently to the database. This enables performance analysis from another worker to avoid perturbing your workload, or to analyze a terminated worker.
The following snippet shows how to retrieve the raw internal tables for the current date. Typically, you should filter by ProcessInfoId
to get relevant data. See Monitor query performance and Persistent Query State Log for analysis tools.
date = today(timeZone("ET"))
pel = db.liveTable("DbInternal", "ProcessEventLog").where("Date=date")
ael = db.liveTable("DbInternal", "AuditEventLog").where("Date=date")
wdl = db.liveTable("DbInternal", "WorkspaceData").where("Date=date")
pqsl = db.liveTable("DbInternal", "PersistentQueryStateLog").where("Date=date")
pqcl = db.liveTable("DbInternal", "PersistentQueryConfigurationLogV2").where("Date=date")
rul = db.liveTable("DbInternal", "ResourceUtilization").where("Date=date")
qpl = db.liveTable("DbInternal", "QueryPerformanceLogCoreV2").where("Date=date")
qopl = db.liveTable("DbInternal", "QueryOperationPerformanceLogCoreV2").where("Date=date")
upl = db.liveTable("DbInternal", "UpdatePerformanceLogCoreV2").where("Date=date")
pil = db.liveTable("DbInternal", "ProcessInfoLogCommunity").where("Date=date")
pml = db.liveTable("DbInternal", "ProcessMetricsLogCoreV2").where("Date=date")
ssl = db.liveTable("DbInternal", "ServerStateLogCoreV2").where("Date=date")
pel = db.live_table("DbInternal", "ProcessEventLog").where("Date=today()")
ael = db.live_table("DbInternal", "AuditEventLog").where("Date=today()")
wdl = db.live_table("DbInternal", "WorkspaceData").where("Date=today()")
pqsl = db.live_table("DbInternal", "PersistentQueryStateLog").where("Date=today()")
pqcl = db.live_table("DbInternal", "PersistentQueryConfigurationLogV2").where(
"Date=today()"
)
rul = db.live_table("DbInternal", "ResourceUtilization").where("Date=today()")
qpl = db.live_table("DbInternal", "QueryPerformanceLogCoreV2").where("Date=today()")
qopl = db.live_table("DbInternal", "QueryOperationPerformanceLogCoreV2").where(
"Date=today()"
)
upl = db.live_table("DbInternal", "UpdatePerformanceLogCoreV2").where("Date=today()")
pil = db.live_table("DbInternal", "ProcessInfoLogCommunity").where("Date=today()")
pml = db.live_table("DbInternal", "ProcessMetricsLogCoreV2").where("Date=today()")
ssl = db.live_table("DbInternal", "ServerStateLogCoreV2").where("Date=today()")
Worker Performance Tables
Workers write four performance-related tables, which can be analyzed with Performance Overview queries. These tables have internal partition indexes to allow the Performance Overview tool to read only necessary internal partitions. These tables are based on the Core performance tables, but include a Date
partitioning column and process information including the process info id and user information.
Log Index Tables
Certain high-volume internal tables have associated index tables used to optimize administrative queries commonly performed against those tables. The Data Import Server produces these Index tables as it imports data, making it possible to identify the set of internal partitions that have data for a given key. For further details please see Indexing Intraday Partitions.
Index Table | Base Table | Partitioning Column(s) | Key Columns |
---|---|---|---|
ProcessEventLogIndex | ProcessEventLog | Date | Process, ProcessInfoId, AuthenticatedUser, EffectiveUser |
ProcessTelemetryIndex | ProcessTelemetry | Date | ClientInstance, ClientHost, ProcessInfoId, Worker, AuthenticatedUser, EffectiveUser |
QueryOperationPerformanceLogCoreV2Index | QueryOperationPerformanceLogCoreV2 | Date | ProcessInfoId, PrimaryAuthenticatedUser, PrimaryEffectiveUser, WorkerName, ServerHost |
QueryPerformanceLogCoreV2Index | QueryPerformanceLogCoreV2 | Date | ProcessInfoId, PrimaryAuthenticatedUser, PrimaryEffectiveUser, WorkerName, ServerHost |
ServerStateLogCoreV2Index | ServerStateLogCore | Date | ProcessInfoId, PrimaryAuthenticatedUser, PrimaryEffectiveUser, WorkerName, ServerHost |
UpdatePerformanceLogCoreV2Index | UpdatePerformanceLogCoreV2 | Date | ProcessInfoId, PrimaryAuthenticatedUser, PrimaryEffectiveUser, WorkerName, ServerHost |
The following snippet shows how to retrieve the raw index data for the current date.
date = today(timeZone("ET"))
peli = db.liveTable("DbInternal", "ProcessEventLogIndex").where("Date=date")
pti = db.liveTable("DbInternal", "ProcessTelemetryIndex").where("Date=date")
qopli = db.liveTable("DbInternal", "QueryOperationPerformanceLogCoreV2Index").where("Date=date")
qpli = db.liveTable("DbInternal", "QueryPerformanceLogCoreV2Index").where("Date=date")
ssli = db.liveTable("DbInternal", "ServerStateLogCoreV2Index").where("Date=date")
upli = db.liveTable("DbInternal", "UpdatePerformanceLogCoreV2Index").where("Date=date")
peli = db.live_table("DbInternal", "ProcessEventLogIndex").where("Date=today()")
pti = db.live_table("DbInternal", "ProcessTelemetryIndex").where("Date=today()")
qopli = db.live_table("DbInternal", "QueryOperationPerformanceLogCoreV2Index").where(
"Date=today()"
)
qpli = db.live_table("DbInternal", "QueryPerformanceLogCoreV2Index").where(
"Date=today()"
)
ssli = db.live_table("DbInternal", "ServerStateLogCoreV2Index").where("Date=today()")
upli = db.live_table("DbInternal", "UpdatePerformanceLogCoreV2Index").where(
"Date=today()"
)
Related documentation
- Core performance tables
- Indexing Intraday Partitions
- Persistent Query Controller configuration
- Performance Tables
- Monitor query performance
AuditEventLog
PersistentQueryConfigurationLog
PersistentQueryStateLog
ProcessEventLogIndex
ProcessEventLog
ProcessInfo
ProcessMetrics
QueryOperationPerformanceLogIndex
QueryOperationPerformanceLog
QueryPerformanceLog
QueryUserAssignmentLog
ResourceUtilization
ServerStateLogIndex
ServerStateLog
UpdatePerformanceLogIndex
UpdatePerformanceLog
WorkspaceDataSnapshot
WorkspaceData