How to configure authenticated Table Data Protocol (TDP)
Remote table data providers require authentication from clients. Enforcement is at the table level.
Data Import Servers (DIS)
The DIS requires membership in a set of allowed groups, as defined by property DataImportServer.allowedGroups, with default value of dis-tdp-readers. This property value is interpreted as a comma separated list of ACL groups that will be granted full access to all tables. The special value * indicates unrestricted data access.
By default, the user tdcp is a member of this group. This grants the TDCP full access to data, which will apply more fine-grained controls per-user.
Table Data Cache Proxy (TDCP)
Connections to the TDCP must be authenticated. The TDCP uses the row-level ACLS to determine if a connected client may access a table. If an authenticated user has access to any rows of the table, then the TDCP permits the user to read block data. If an authenticated user does not have access to any rows of the table, then the TDCP rejects block data requests for the table. This access control is at the table level, row level enforcement is performed by the deephaven worker or other proxy.
Troubleshooting
Authentication attempts are logged to the text logs for the DIS and TDCP, and to the AuditEventLog table. When a user cannot access a table, you might see one of the following error messages.
AuditEventLog
The AuditEventLog may have messages in this format:
| Date | Timestamp | ClientHost | ClientPort | ClientUuid | ServerHost | ServerPort | Process | ProcessInfoId | AuthenticatedUser | EffectiveUser | ProcessUser | Namespace | Table | Id | Event | Details |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2023-06-19 | 2023-06-19T15:28:59.099000000 NY | your-own-namespace-here | TableDataCacheProxy | e68040b3-4e16-4e56-84cb-afa47111ff0f | iris | any-username | irisadmin | DbInternal | AuditEventLog | TablePermissionCheck | Permission for table DbInternal.AuditEventLog denied for user {iris operating as any-username} |
When the query worker denies access because of row ACLS, you may see messages containing caused by: com.illumon.iris.db.exceptions.TableAccessException: User {iris operating as carlos} may not access: DbInternal.AuditEventLog.
A Legacy worker's ProcessEventLog will have exceptions of this form:
2023-06-19 15:25:09.367 ERROR Groovy command, "ael=db.i("DbInternal", "AuditEventLog")" failed:
com.illumon.iris.db.exceptions.ScriptEvaluationException: Error encountered at line 1: ael=db.i("DbInternal", "AuditEventLog")
at com.illumon.iris.db.util.IrisDbGroovySession.maybeRewriteStackTrace(IrisDbGroovySession.java:255)
at com.illumon.iris.db.util.IrisDbGroovySession.wrapAndRewriteStackTrace(IrisDbGroovySession.java:236)
at com.illumon.iris.db.util.IrisDbGroovySession.evaluate(IrisDbGroovySession.java:228)
at com.illumon.iris.console.events.RemoteScriptCommandQuery.execute(RemoteScriptCommandQuery.java:89)
at com.illumon.iris.console.events.RemoteScriptCommandQuery.execute(RemoteScriptCommandQuery.java:24)
at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$QueryAction.lambda$execute$0(RemoteQueryProcessor.java:1922)
at com.illumon.util.locks.FunctionalLock.computeLockedInterruptibly(FunctionalLock.java:97)
at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$QueryAction.execute(RemoteQueryProcessor.java:1922)
at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$ClientConnectionHandler.runSyncQueryAndSendResult(RemoteQueryProcessor.java:1683)
at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$ClientConnectionHandler.handleCommandST(RemoteQueryProcessor.java:1577)
at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$ClientConnectionHandler$HandleCommandRunnable.run(RemoteQueryProcessor.java:1172)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.lang.Thread.run(Thread.java:833)
caused by:
com.illumon.iris.db.exceptions.TableAccessException: User {iris operating as carlos} may not access: DbInternal.AuditEventLog
at com.illumon.iris.db.v2.permissions.AclHelper.applyRowAcls(AclHelper.java:78)
at com.illumon.iris.db.v2.permissions.AclHelper.applyUserPermissions(AclHelper.java:66)
at com.illumon.iris.db.tables.databases.OnDiskDatabase.lambda$getIntradayTableV2$4(OnDiskDatabase.java:362)
at com.illumon.iris.db.tables.utils.QueryPerformanceRecorder.withNugget(QueryPerformanceRecorder.java:437)
at com.illumon.iris.db.tables.databases.OnDiskDatabase.getIntradayTableV2(OnDiskDatabase.java:354)
at com.illumon.iris.db.tables.databases.Database.getIntradayTableV2(Database.java:460)
at com.illumon.iris.db.tables.databases.Database.getIntradayTable(Database.java:434)
at com.illumon.iris.db.tables.databases.Database.i(Database.java:447)
at com.illumon.iris.db.tables.databases.Database$i.call(null:-1)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:148)
at com.illumon.iris.db.util.dynamic.illumon_gce_vpn_9z9t_c_illumon_eng_170715_internalworker_41_2.run(illumon_gce_vpn_9z9t_c_illumon_eng_170715_internalworker_41_2.groovy:45)
at groovy.lang.GroovyShell.evaluate(GroovyShell.java:427)
at groovy.lang.GroovyShell.evaluate(GroovyShell.java:461)
at groovy.lang.GroovyShell.evaluate(GroovyShell.java:436)
at com.illumon.iris.db.util.IrisDbGroovySession.evaluateCommand(IrisDbGroovySession.java:195)
at com.illumon.iris.db.util.IrisDbGroovySession.lambda$evaluate$0(IrisDbGroovySession.java:224)
at com.illumon.util.locks.FunctionalLock.doLockedInterruptibly(FunctionalLock.java:45)
at com.illumon.iris.db.util.IrisDbGroovySession.evaluate(IrisDbGroovySession.java:224)
at com.illumon.iris.console.events.RemoteScriptCommandQuery.execute(RemoteScriptCommandQuery.java:89)
at com.illumon.iris.console.events.RemoteScriptCommandQuery.execute(RemoteScriptCommandQuery.java:24)
at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$QueryAction.lambda$execute$0(RemoteQueryProcessor.java:1922)
at com.illumon.util.locks.FunctionalLock.computeLockedInterruptibly(FunctionalLock.java:97)
at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$QueryAction.execute(RemoteQueryProcessor.java:1922)
at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$ClientConnectionHandler.runSyncQueryAndSendResult(RemoteQueryProcessor.java:1683)
at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$ClientConnectionHandler.handleCommandST(RemoteQueryProcessor.java:1577)
at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$ClientConnectionHandler$HandleCommandRunnable.run(RemoteQueryProcessor.java:1172)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.lang.Thread.run(Thread.java:833)
If the ACL check in the worker is bypassed, by using a different client or access method, or in an attempt to bypass ACL restrictions, the TDCP log in /var/log/deephaven/tdcp/TableDataCacheProxy.<timestamp> will have messages of this form: Error while responding to request [requestMessageType=TABLE_LOCATIONS_REQUEST, requestKey=0]: com.illumon.iris.db.exceptions.TableAccessException: User {iris operating as carlos} may not access: DbInternal.AuditEventLog
The full log will look something like this:
[2023-06-19T15:28:55.662450-0400] - INFO - TablePermissionCheck: Permission for table DbInternal.AuditEventLog denied for user {iris operating as carlos} (no filters defined)
[2023-06-19T15:28:55.666240-0400] - WARN - TableDataServiceExporter-IntradayProxy: TableDataProtocol.RemoteClient[job:1716875137/TableDataServiceExporter/127.0.0.1:22016->127.0.0.1:38886]: Error while responding to request [requestMessageType=TABLE_LOCATIONS_REQUEST, requestKey=0]: com.illumon.iris.db.exceptions.TableAccessException: User {iris operating as carlos} may not access: DbInternal.AuditEventLog
at com.illumon.iris.db.v2.locations.server.TableDataServiceExporter$ClientHandler.respondToTableLocationsRequest(TableDataServiceExporter.java:579)
at com.illumon.iris.db.v2.locations.server.TableDataServiceExporter$ClientHandler.handleTableLocationsRequest(TableDataServiceExporter.java:319)
at com.illumon.iris.db.v2.locations.protocol.TableDataProtocol$RemoteClient.dispatch(TableDataProtocol.java:664)
at com.fishlib.net.impl.nio.AbstractReadaheadRemotePeer.handleIncoming(AbstractReadaheadRemotePeer.java:160)
at com.fishlib.io.sched.IOJobImpl.notifyIncoming(IOJobImpl.java:1675)
at com.fishlib.io.sched.IOJobImpl.fillAndHandle(IOJobImpl.java:539)
at com.fishlib.io.sched.IOJobImpl.invoke(IOJobImpl.java:401)
at com.fishlib.io.sched.YASchedulerImpl.dispatch(YASchedulerImpl.java:703)
at com.fishlib.io.sched.YASchedulerImpl.work(YASchedulerImpl.java:764)
at com.fishlib.net.impl.nio.FastNIODriver.run(FastNIODriver.java:190)
at java.lang.Thread.run(Thread.java:833)
If a user attempts to read data directly from a DIS, the worker will have a message similar to caused by: com.illumon.iris.db.v2.locations.TableDataException: RemoteTableDataService-Remote_db_dis: Problem maintaining subscription for RemoteTableLocationProvider[DbInternal/AuditEventLog/I]: Subscription request rejected: TableDataProtocolDriverImpl-Remote_db_dis: Received rejection for request (requestKey=0): User {iris} may not access: DbInternal.AuditEventLog.
The full message in the worker log will resemble:
2023-06-19 15:10:27.709 WARN Error for client <job:440829237/RemoteQueryDispatcher_worker_195_Server/10.128.0.202:32194->10.128.1.24:64006/CommandConnection>: Error while executing query: java.lang.RuntimeException: com.illumon.iris.db.v2.locations.TableDataException: Processed pending exception: java.lang.RuntimeException: com.illumon.iris.db.v2.locations.TableDataException: Processed pending exception
at com.illumon.iris.db.tables.remote.TableApplyQuery.execute(TableApplyQuery.java:38)
at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$QueryAction.lambda$execute$0(RemoteQueryProcessor.java:1922)
at com.illumon.util.locks.FunctionalLock.computeLockedInterruptibly(FunctionalLock.java:97)
at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$QueryAction.execute(RemoteQueryProcessor.java:1922)
at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$ClientConnectionHandler.runSyncQueryAndSendResult(RemoteQueryProcessor.java:1683)
at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$ClientConnectionHandler.handleCommandST(RemoteQueryProcessor.java:1577)
at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$ClientConnectionHandler$HandleCommandRunnable.run(RemoteQueryProcessor.java:1172)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.lang.Thread.run(Thread.java:833)
caused by:
com.illumon.iris.db.v2.locations.TableDataException: Processed pending exception
at com.illumon.iris.db.v2.locations.TableLocationSubscriptionBuffer.processPending(TableLocationSubscriptionBuffer.java:62)
at com.illumon.iris.db.v2.SourceTable.lambda$initializeAvailableLocations$0(SourceTable.java:151)
at com.illumon.iris.db.tables.utils.QueryPerformanceRecorder.withNugget(QueryPerformanceRecorder.java:418)
at com.illumon.iris.db.v2.SourceTable.initializeAvailableLocations(SourceTable.java:148)
at com.illumon.iris.db.v2.PartitionAwareSourceTable.selectDistinct(PartitionAwareSourceTable.java:285)
at com.illumon.iris.db.tables.Table.selectDistinct(Table.java:511)
at com.illumon.iris.console.events.GetDisplayAndTopChoicesQuery.call(GetDisplayAndTopChoicesQuery.java:36)
at com.illumon.iris.console.events.GetDisplayAndTopChoicesQuery.call(GetDisplayAndTopChoicesQuery.java:22)
at com.illumon.iris.db.tables.Table.apply(Table.java:2639)
at com.illumon.iris.db.tables.remote.TableApplyQuery.execute(TableApplyQuery.java:35)
at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$QueryAction.lambda$execute$0(RemoteQueryProcessor.java:1922)
at com.illumon.util.locks.FunctionalLock.computeLockedInterruptibly(FunctionalLock.java:97)
at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$QueryAction.execute(RemoteQueryProcessor.java:1922)
at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$ClientConnectionHandler.runSyncQueryAndSendResult(RemoteQueryProcessor.java:1683)
at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$ClientConnectionHandler.handleCommandST(RemoteQueryProcessor.java:1577)
at com.illumon.iris.db.tables.remotequery.RemoteQueryProcessor$ClientConnectionHandler$HandleCommandRunnable.run(RemoteQueryProcessor.java:1172)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.lang.Thread.run(Thread.java:833)
caused by:
com.illumon.iris.db.v2.locations.TableDataException: RemoteTableDataService-Remote_db_dis: Problem maintaining subscription for RemoteTableLocationProvider[DbInternal/AuditEventLog/I]: Subscription request rejected: TableDataProtocolDriverImpl-Remote_db_dis: Received rejection for request (requestKey=0): User {iris} may not access: DbInternal.AuditEventLog
at com.illumon.iris.db.v2.locations.remote.RemoteTableDataService$TableLocationProviderImpl.onRejection(RemoteTableDataService.java:193)
at com.illumon.iris.db.v2.locations.protocol.TableDataRequestImpl.reportRejection(TableDataRequestImpl.java:515)
at com.illumon.iris.db.v2.locations.protocol.TableDataRequestImpl.deliverResultStatus(TableDataRequestImpl.java:492)
at com.illumon.iris.db.v2.locations.protocol.TableDataRequestImpl.rejected(TableDataRequestImpl.java:511)
at com.illumon.iris.db.v2.locations.protocol.TableDataRequestImpl.handleRequestRejection(TableDataRequestImpl.java:585)
at com.illumon.iris.db.v2.locations.protocol.TableDataProtocolDriverImpl$ResponseHandler.handleRequestRejection(TableDataProtocolDriverImpl.java:164)
at com.illumon.iris.db.v2.locations.protocol.TableDataProtocol$RemoteServer.dispatch(TableDataProtocol.java:408)
at com.fishlib.net.impl.nio.AbstractReadaheadRemotePeer.handleIncoming(AbstractReadaheadRemotePeer.java:160)
at com.fishlib.io.sched.IOJobImpl.notifyIncoming(IOJobImpl.java:1675)
at com.fishlib.io.sched.IOJobImpl.fillAndHandle(IOJobImpl.java:539)
at com.fishlib.io.sched.IOJobImpl.invoke(IOJobImpl.java:387)
at com.fishlib.io.sched.YASchedulerImpl.dispatch(YASchedulerImpl.java:703)
at com.fishlib.io.sched.YASchedulerImpl.work(YASchedulerImpl.java:789)
at com.fishlib.net.impl.nio.FastNIODriver.run(FastNIODriver.java:190)
at java.lang.Thread.run(Thread.java:833)
In this case, the DIS log in /var/log/deephaven/dis/DataImportServer.log.<timestamp> will have a detailed message like this:
[2023-06-19T15:10:27.699129-0400] - INFO - TablePermissionCheck: Permission for table DbInternal.AuditEventLog denied for user {iris}
[2023-06-19T15:10:27.702400-0400] - WARN - TableDataServiceExporter-DataImportServer-db_dis: TableDataProtocol.RemoteClient[job:782108321/TableDataServiceExporter/10.128.0.202:22015->10.128.0.202:56020]: Error while responding to request [requestMessageType=TABLE_LOCATIONS_REQUEST, requestKey=0]: com.illumon.iris.db.exceptions.TableAccessException: User {iris} may not access: DbInternal.AuditEventLog
at com.illumon.iris.db.v2.locations.server.TableDataServiceExporter$ClientHandler.respondToTableLocationsRequest(TableDataServiceExporter.java:579)
at com.illumon.iris.db.v2.locations.server.TableDataServiceExporter$ClientHandler.handleTableLocationsRequest(TableDataServiceExporter.java:319)
at com.illumon.iris.db.v2.locations.protocol.TableDataProtocol$RemoteClient.dispatch(TableDataProtocol.java:664)
at com.fishlib.net.impl.nio.AbstractReadaheadRemotePeer.handleIncoming(AbstractReadaheadRemotePeer.java:160)
at com.fishlib.io.sched.IOJobImpl.notifyIncoming(IOJobImpl.java:1675)
at com.fishlib.io.sched.IOJobImpl.fillAndHandle(IOJobImpl.java:539)
at com.fishlib.io.sched.IOJobImpl.invoke(IOJobImpl.java:401)
at com.fishlib.io.sched.YASchedulerImpl.dispatch(YASchedulerImpl.java:703)
at com.fishlib.io.sched.YASchedulerImpl.work(YASchedulerImpl.java:789)
at com.fishlib.net.impl.nio.FastNIODriver.run(FastNIODriver.java:190)
at java.lang.Thread.run(Thread.java:833)