How to configure authenticated Table Data Protocol (TDP)

Deephaven's Table Data Protocol (TDP) is a specialized communication protocol designed for the efficient exchange of columnar data between Deephaven services or between a client and a Deephaven server. When data is accessed remotely via the TDP, it's crucial to ensure that only authorized clients can access it. This guide outlines how to configure authentication for the TDP in the TDCP, where access control is enforced at the table level.

Prerequisites

Before configuring authenticated Table Data Protocol (TDP) access, ensure you have the following:

Administrative permissions to modify Deephaven configuration files.
The ability to view server logs for these components for troubleshooting purposes.

Data Import Servers (DIS)

The Data Import Server (DIS) is a Deephaven component that ingests data from various external sources and makes it available as tables in the Deephaven environment, via the Table Data Protocol. When configured for authenticated TDP, it enforces authentication and authorization checks before allowing data access.

To enable authenticated TDP for a DIS, you must set the following Java system properties when launching the DIS process. The DIS requires membership in a set of allowed groups, as defined by property DataImportServer.allowedGroups, with default value of dis-tdp-readers. This property value is interpreted as a comma-separated list of ACL groups that will be granted full access to all tables. The special value * indicates unrestricted data access.

By default, the user tdcp is a member of this group. This grants the TDCP full access to data, which will apply more fine-grained controls per-user.

Table Data Cache Proxy (TDCP)

The Table Data Cache Proxy (TDCP) often acts as an intermediary for TDP connections, providing caching and connection management. Authenticated TDP access through the TDCP involves the following:

Authenticated Connections: All client connections to the TDCP must be authenticated.
Initial Table Access Check: When a client requests data, the TDCP consults the underlying row-level Access Control Lists (ACLs) of the target table.
- If the ACLs indicate that the authenticated user has permission to access rows in the table, the TDCP grants table-level access for the TDP connection.
- If the user has no permissions to rows in the table, the TDCP rejects the request, effectively denying access to the entire table for that user over TDP.
Data Transfer and Fine-Grained Enforcement:
- If table-level access is granted by the TDCP, it permits the transfer of block data.
- The TDCP itself does not perform fine-grained row-level filtering. The Deephaven worker (or another downstream proxy component) is responsible for enforcing the specific row-level ACLs, ensuring the client only receives the data from rows they are explicitly permitted to see.

In essence, the TDCP performs a preliminary, coarse-grained check (can the user see anything in this table?), while the worker handles the detailed, row-by-row permission enforcement.

Troubleshooting

Authentication attempts are logged to the text logs for the DIS and TDCP, and to the AuditEventLog table. When a user cannot access a table, you might see one of the following error messages.

AuditEventLog

The AuditEventLog table provides a record of security-related events, including permission checks. When a TDP authentication or authorization issue occurs, you might find entries similar to this:

Date	Timestamp	ClientHost	ClientPort	ClientUuid	ServerHost	ServerPort	Process	ProcessInfoId	AuthenticatedUser	EffectiveUser	ProcessUser	Namespace	Table	Id	Event	Details
2023-06-19	2023-06-19T15:28:59.099000000 NY				your-own-namespace-here		TableDataCacheProxy	e68040b3-4e16-4e56-84cb-afa47111ff0f	iris	any-username	irisadmin	DbInternal	AuditEventLog		TablePermissionCheck	Permission for table DbInternal.AuditEventLog denied for user {iris operating as any-username}

Explanation of the log entry:

Process: TableDataCacheProxy indicates the TDCP was involved.
AuthenticatedUser / EffectiveUser: Shows the user identities involved (iris and any-username).
Event: TablePermissionCheck is the operation that was performed.
Details: "Permission for table DbInternal.AuditEventLog denied..." clearly states that the EffectiveUser lacks the necessary permissions for the specified table.
Action: Review the ACLs for the DbInternal.AuditEventLog table and grant the EffectiveUser (or a group they belong to) the required permissions if access is intended.

If the query worker (not just the TDCP or DIS) denies access due to row-level ACLs during a TDP operation, you might see exceptions in the worker logs containing messages like: caused by: com.illumon.iris.db.exceptions.TableAccessException: User {iris operating as carlos} may not access: DbInternal.AuditEventLog

Legacy Worker ProcessEventLog

A Legacy Deephaven worker's ProcessEventLog (typically found in its log files) will show exceptions when a script attempts to access a table via TDP and is denied. The error often wraps a TableAccessException:

2023-06-19 15:25:09.367 ERROR Groovy command, "ael=db.i("DbInternal", "AuditEventLog")" failed:

com.illumon.iris.db.exceptions.ScriptEvaluationException: Error encountered at line 1: ael=db.i("DbInternal", "AuditEventLog")
        at com.illumon.iris.db.util.IrisDbGroovySession.maybeRewriteStackTrace(IrisDbGroovySession.java:255)
        at com.illumon.iris.db.util.IrisDbGroovySession.wrapAndRewriteStackTrace(IrisDbGroovySession.java:236)
        at com.illumon.iris.db.util.IrisDbGroovySession.evaluate(IrisDbGroovySession.java:228)
        at com.illumon.iris.console.events.RemoteScriptCommandQuery.execute(RemoteScriptCommandQuery.java:89)
        at com.illumon.iris.console.events.RemoteScriptCommandQuery.execute(RemoteScriptCommandQuery.java:24)
        at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$QueryAction.lambda$execute$0(RemoteQueryProcessor.java:1922)
        at com.illumon.util.locks.FunctionalLock.computeLockedInterruptibly(FunctionalLock.java:97)
        at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$QueryAction.execute(RemoteQueryProcessor.java:1922)
        at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$ClientConnectionHandler.runSyncQueryAndSendResult(RemoteQueryProcessor.java:1683)
        at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$ClientConnectionHandler.handleCommandST(RemoteQueryProcessor.java:1577)
        at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$ClientConnectionHandler$HandleCommandRunnable.run(RemoteQueryProcessor.java:1172)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.lang.Thread.run(Thread.java:833)
caused by:
com.illumon.iris.db.exceptions.TableAccessException: User {iris operating as carlos} may not access: DbInternal.AuditEventLog
        at com.illumon.iris.db.v2.permissions.AclHelper.applyRowAcls(AclHelper.java:78)
        at com.illumon.iris.db.v2.permissions.AclHelper.applyUserPermissions(AclHelper.java:66)
        at com.illumon.iris.db.tables.databases.OnDiskDatabase.lambda$getIntradayTableV2$4(OnDiskDatabase.java:362)
        at com.illumon.iris.db.tables.utils.QueryPerformanceRecorder.withNugget(QueryPerformanceRecorder.java:437)
        at com.illumon.iris.db.tables.databases.OnDiskDatabase.getIntradayTableV2(OnDiskDatabase.java:354)
        at com.illumon.iris.db.tables.databases.Database.getIntradayTableV2(Database.java:460)
        at com.illumon.iris.db.tables.databases.Database.getIntradayTable(Database.java:434)
        at com.illumon.iris.db.tables.databases.Database.i(Database.java:447)
        at com.illumon.iris.db.tables.databases.Database$i.call(null:-1)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:148)
        at com.illumon.iris.db.util.dynamic.illumon_gce_vpn_9z9t_c_illumon_eng_170715_internalworker_41_2.run(illumon_gce_vpn_9z9t_c_illumon_eng_170715_internalworker_41_2.groovy:45)
        at groovy.lang.GroovyShell.evaluate(GroovyShell.java:427)
        at groovy.lang.GroovyShell.evaluate(GroovyShell.java:461)
        at groovy.lang.GroovyShell.evaluate(GroovyShell.java:436)
        at com.illumon.iris.db.util.IrisDbGroovySession.evaluateCommand(IrisDbGroovySession.java:195)
        at com.illumon.iris.db.util.IrisDbGroovySession.lambda$evaluate$0(IrisDbGroovySession.java:224)
        at com.illumon.util.locks.FunctionalLock.doLockedInterruptibly(FunctionalLock.java:45)
        at com.illumon.iris.db.util.IrisDbGroovySession.evaluate(IrisDbGroovySession.java:224)
        at com.illumon.iris.console.events.RemoteScriptCommandQuery.execute(RemoteScriptCommandQuery.java:89)
        at com.illumon.iris.console.events.RemoteScriptCommandQuery.execute(RemoteScriptCommandQuery.java:24)
        at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$QueryAction.lambda$execute$0(RemoteQueryProcessor.java:1922)
        at com.illumon.util.locks.FunctionalLock.computeLockedInterruptibly(FunctionalLock.java:97)
        at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$QueryAction.execute(RemoteQueryProcessor.java:1922)
        at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$ClientConnectionHandler.runSyncQueryAndSendResult(RemoteQueryProcessor.java:1683)
        at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$ClientConnectionHandler.handleCommandST(RemoteQueryProcessor.java:1577)
        at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$ClientConnectionHandler$HandleCommandRunnable.run(RemoteQueryProcessor.java:1172)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.lang.Thread.run(Thread.java:833)

Explanation of the log entry:

The top-level error shows a Groovy command (ael=db.i("DbInternal", "AuditEventLog")) failed.
The caused by: com.illumon.iris.db.exceptions.TableAccessException is the key. It indicates that the user carlos (effective user, authenticated as iris) was denied access to the DbInternal.AuditEventLog table.
This is similar to the direct TableAccessException seen previously, but here it's surfaced through a script execution failure in a legacy worker.
Action: Check the ACLs on the DbInternal.AuditEventLog table for the user carlos.

If the ACL check in the worker is bypassed, by using a different client or access method, or in an attempt to bypass ACL restrictions, the TDCP log in /var/log/deephaven/tdcp/TableDataCacheProxy.<timestamp> will have messages of this form: Error while responding to request [requestMessageType=TABLE_LOCATIONS_REQUEST, requestKey=0]: com.illumon.iris.db.exceptions.TableAccessException: User {iris operating as carlos} may not access: DbInternal.AuditEventLog

The full log will look something like this:

[2023-06-19T15:28:55.662450-0400] - INFO - TablePermissionCheck: Permission for table DbInternal.AuditEventLog denied for user {iris operating as carlos} (no filters defined)
[2023-06-19T15:28:55.666240-0400] - WARN - TableDataServiceExporter-IntradayProxy: TableDataProtocol.RemoteClient[job:1716875137/TableDataServiceExporter/127.0.0.1:22016->127.0.0.1:38886]: Error while responding to request [requestMessageType=TABLE_LOCATIONS_REQUEST, requestKey=0]: com.illumon.iris.db.exceptions.TableAccessException: User {iris operating as carlos} may not access: DbInternal.AuditEventLog
        at com.illumon.iris.db.v2.locations.server.TableDataServiceExporter$ClientHandler.respondToTableLocationsRequest(TableDataServiceExporter.java:579)
        at com.illumon.iris.db.v2.locations.server.TableDataServiceExporter$ClientHandler.handleTableLocationsRequest(TableDataServiceExporter.java:319)
        at com.illumon.iris.db.v2.locations.protocol.TableDataProtocol$RemoteClient.dispatch(TableDataProtocol.java:664)
        at com.fishlib.net.impl.nio.AbstractReadaheadRemotePeer.handleIncoming(AbstractReadaheadRemotePeer.java:160)
        at com.fishlib.io.sched.IOJobImpl.notifyIncoming(IOJobImpl.java:1675)
        at com.fishlib.io.sched.IOJobImpl.fillAndHandle(IOJobImpl.java:539)
        at com.fishlib.io.sched.IOJobImpl.invoke(IOJobImpl.java:401)
        at com.fishlib.io.sched.YASchedulerImpl.dispatch(YASchedulerImpl.java:703)
        at com.fishlib.io.sched.YASchedulerImpl.work(YASchedulerImpl.java:764)
        at com.fishlib.net.impl.nio.FastNIODriver.run(FastNIODriver.java:190)
        at java.lang.Thread.run(Thread.java:833)

If a user attempts to read data directly from a DIS, the worker will have a message similar to caused by: com.illumon.iris.db.v2.locations.TableDataException: RemoteTableDataService-Remote_db_dis: Problem maintaining subscription for RemoteTableLocationProvider[DbInternal/AuditEventLog/I]: Subscription request rejected: TableDataProtocolDriverImpl-Remote_db_dis: Received rejection for request (requestKey=0): User {iris} may not access: DbInternal.AuditEventLog.

The full message in the worker log will resemble:

2023-06-19 15:10:27.709 WARN Error for client <job:440829237/RemoteQueryDispatcher_worker_195_Server/10.128.0.202:32194->10.128.1.24:64006/CommandConnection>: Error while executing query: java.lang.RuntimeException: com.illumon.iris.db.v2.locations.TableDataException: Processed pending exception: java.lang.RuntimeException: com.illumon.iris.db.v2.locations.TableDataException: Processed pending exception
        at com.illumon.iris.db.tables.remote.TableApplyQuery.execute(TableApplyQuery.java:38)
        at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$QueryAction.lambda$execute$0(RemoteQueryProcessor.java:1922)
        at com.illumon.util.locks.FunctionalLock.computeLockedInterruptibly(FunctionalLock.java:97)
        at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$QueryAction.execute(RemoteQueryProcessor.java:1922)
        at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$ClientConnectionHandler.runSyncQueryAndSendResult(RemoteQueryProcessor.java:1683)
        at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$ClientConnectionHandler.handleCommandST(RemoteQueryProcessor.java:1577)
        at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$ClientConnectionHandler$HandleCommandRunnable.run(RemoteQueryProcessor.java:1172)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.lang.Thread.run(Thread.java:833)
caused by:
com.illumon.iris.db.v2.locations.TableDataException: Processed pending exception
        at com.illumon.iris.db.v2.locations.TableLocationSubscriptionBuffer.processPending(TableLocationSubscriptionBuffer.java:62)
        at com.illumon.iris.db.v2.SourceTable.lambda$initializeAvailableLocations$0(SourceTable.java:151)
        at com.illumon.iris.db.tables.utils.QueryPerformanceRecorder.withNugget(QueryPerformanceRecorder.java:418)
        at com.illumon.iris.db.v2.SourceTable.initializeAvailableLocations(SourceTable.java:148)
        at com.illumon.iris.db.v2.PartitionAwareSourceTable.selectDistinct(PartitionAwareSourceTable.java:285)
        at com.illumon.iris.db.tables.Table.selectDistinct(Table.java:511)
        at com.illumon.iris.console.events.GetDisplayAndTopChoicesQuery.call(GetDisplayAndTopChoicesQuery.java:36)
        at com.illumon.iris.console.events.GetDisplayAndTopChoicesQuery.call(GetDisplayAndTopChoicesQuery.java:22)
        at com.illumon.iris.db.tables.Table.apply(Table.java:2639)
        at com.illumon.iris.db.tables.remote.TableApplyQuery.execute(TableApplyQuery.java:35)
        at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$QueryAction.lambda$execute$0(RemoteQueryProcessor.java:1922)
        at com.illumon.util.locks.FunctionalLock.computeLockedInterruptibly(FunctionalLock.java:97)
        at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$QueryAction.execute(RemoteQueryProcessor.java:1922)
        at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$ClientConnectionHandler.runSyncQueryAndSendResult(RemoteQueryProcessor.java:1683)
        at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$ClientConnectionHandler.handleCommandST(RemoteQueryProcessor.java:1577)
        at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$ClientConnectionHandler$HandleCommandRunnable.run(RemoteQueryProcessor.java:1172)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.lang.Thread.run(Thread.java:833)
caused by:
com.illumon.iris.db.v2.locations.TableDataException: RemoteTableDataService-Remote_db_dis: Problem maintaining subscription for RemoteTableLocationProvider[DbInternal/AuditEventLog/I]: Subscription request rejected: TableDataProtocolDriverImpl-Remote_db_dis: Received rejection for request (requestKey=0): User {iris} may not access: DbInternal.AuditEventLog
        at com.illumon.iris.db.v2.locations.remote.RemoteTableDataService$TableLocationProviderImpl.onRejection(RemoteTableDataService.java:193)
        at com.illumon.iris.db.v2.locations.protocol.TableDataRequestImpl.reportRejection(TableDataRequestImpl.java:515)
        at com.illumon.iris.db.v2.locations.protocol.TableDataRequestImpl.deliverResultStatus(TableDataRequestImpl.java:492)
        at com.illumon.iris.db.v2.locations.protocol.TableDataRequestImpl.rejected(TableDataRequestImpl.java:511)
        at com.illumon.iris.db.v2.locations.protocol.TableDataRequestImpl.handleRequestRejection(TableDataRequestImpl.java:585)
        at com.illumon.iris.db.v2.locations.protocol.TableDataProtocolDriverImpl$ResponseHandler.handleRequestRejection(TableDataProtocolDriverImpl.java:164)
        at com.illumon.iris.db.v2.locations.protocol.TableDataProtocol$RemoteServer.dispatch(TableDataProtocol.java:408)
        at com.fishlib.net.impl.nio.AbstractReadaheadRemotePeer.handleIncoming(AbstractReadaheadRemotePeer.java:160)
        at com.fishlib.io.sched.IOJobImpl.notifyIncoming(IOJobImpl.java:1675)
        at com.fishlib.io.sched.IOJobImpl.fillAndHandle(IOJobImpl.java:539)
        at com.fishlib.io.sched.IOJobImpl.invoke(IOJobImpl.java:387)
        at com.fishlib.io.sched.YASchedulerImpl.dispatch(YASchedulerImpl.java:703)
        at com.fishlib.io.sched.YASchedulerImpl.work(YASchedulerImpl.java:789)
        at com.fishlib.net.impl.nio.FastNIODriver.run(FastNIODriver.java:190)
        at java.lang.Thread.run(Thread.java:833)

DIS Log

If the Data Import Server (DIS) is the component denying access, its log file (e.g., /var/log/deephaven/dis/DataImportServer.log.<timestamp>) will contain more specific information.

[2023-06-19T15:10:27.699129-0400] - INFO - TablePermissionCheck: Permission for table DbInternal.AuditEventLog denied for user {iris}
[2023-06-19T15:10:27.702400-0400] - WARN - TableDataServiceExporter-DataImportServer-db_dis: TableDataProtocol.RemoteClient[job:782108321/TableDataServiceExporter/10.128.0.202:22015->10.128.0.202:56020]: Error while responding to request [requestMessageType=TABLE_LOCATIONS_REQUEST, requestKey=0]: com.illumon.iris.db.exceptions.TableAccessException: User {iris} may not access: DbInternal.AuditEventLog
        at com.illumon.iris.db.v2.locations.server.TableDataServiceExporter$ClientHandler.respondToTableLocationsRequest(TableDataServiceExporter.java:579)
        at com.illumon.iris.db.v2.locations.server.TableDataServiceExporter$ClientHandler.handleTableLocationsRequest(TableDataServiceExporter.java:319)
        at com.illumon.iris.db.v2.locations.protocol.TableDataProtocol$RemoteClient.dispatch(TableDataProtocol.java:664)
        at com.fishlib.net.impl.nio.AbstractReadaheadRemotePeer.handleIncoming(AbstractReadaheadRemotePeer.java:160)
        at com.fishlib.io.sched.IOJobImpl.notifyIncoming(IOJobImpl.java:1675)
        at com.fishlib.io.sched.IOJobImpl.fillAndHandle(IOJobImpl.java:539)
        at com.fishlib.io.sched.IOJobImpl.invoke(IOJobImpl.java:401)
        at com.fishlib.io.sched.YASchedulerImpl.dispatch(YASchedulerImpl.java:703)
        at com.fishlib.io.sched.YASchedulerImpl.work(YASchedulerImpl.java:789)
        at com.fishlib.net.impl.nio.FastNIODriver.run(FastNIODriver.java:190)
        at java.lang.Thread.run(Thread.java:833)

Explanation of the DIS log entries:

First line: INFO - TablePermissionCheck: Permission for table DbInternal.AuditEventLog denied for user {iris}. This is a clear statement from the DIS's permission checker that user iris was denied access to DbInternal.AuditEventLog.
Second entry (WARN): This shows the TableDataServiceExporter within the DIS logging a TableAccessException for the same reason when trying to respond to a TABLE_LOCATIONS_REQUEST.
- The message User {iris} may not access: DbInternal.AuditEventLog is repeated.
Action: These logs confirm the denial is at the DIS level. Check the DataImportServer.allowedGroups property: Is the user iris (or a group they belong to) part of the allowed groups for general DIS access?