Access control lists

Deephaven enables fine-grained access control to data, queries, and reports. This is accomplished through the individual authorization of users and groups, access control lists (ACLs), and account information.

There are two aspects to access control in Deephaven:

• Authentication is the verification of a user's identity.
• Authorization is verification of what permissions the user has within the system.

Authentication can be handled internally, by storing a password validation hash in the ACL store, or externally, by configuring Deephaven integration with an external identity provider. Commmon external authentication integrations are LDAP (for Active Directory or OpenLDAP), SAML (for OAUTH2 systems such as Okta), and key-based authentication using key pairs. If external authentication is in use, a new user with access to Deephaven will need to be configured both in the external system and in Deephaven before they will be able to log in and access Deephaven features.

Authorization for Deephaven is handled entirely internally. Deephaven data access rights and system privileges can be granted to individual users or groups. Groups in Deephaven are internal to the Deephaven product. Groups in external authentication systems are not currently used by Deephaven, therefore, when group-based permissions are used in Deephaven, it will be necessary to add new users to appropriate Deephaven groups regardless of what group memberships they may have in systems such as Active Directory.

ACL Storage

Deephaven currently has two options for managing the storage of ACL data:

• SQL (default). Either MariaDB or MySQL.
• etcd, using the same etcd instance that is used to store other system configuration data.

See the Customize the installation topic for details on reconfiguring a system between the two options.

Editing ACLs

Authorized users can edit the access settings and permissions by using the DB ACL Editor tool in the console, or through the command line tool, IrisDbUserMod. The following command will run the tool:

sudo -u [admin account (default is irisadmin)] /usr/illumon/latest/bin/iris iris_db_user_mod

Running the tool with no arguments will show usage information.

The main arguments are:

• --interactive - launches the tool in interactive mode. Use exit to end the session.
• --file <file name> - reads instructions from a file.
• --direct <file name> - connects directly to the back-end ACL database and reads properties directly from etcd while applying changes from a file. This is normally used when initializing basic entries for a new installation.

Most commands, e.g. -create_user, can also be run as single-line executions, non-interactively. This example will create a user named test5, assuming that user doesn't already exist, and that the system is using the default irisadmin account:

sudo -u irisadmin /usr/illumon/latest/bin/iris iris_db_user_mod -create_user -user test5

The IrisDbUserMod tool can also be used to export the full ACL DB to a file, for migration or backup purposes, or import the full set from a file. The corresponding actions are -export_all_acls and -import_all_acls. These actions require a file name to be provided with the -acl_file argument.

When importing data which may already exist, the -overwrite_existing argument should be added to each line that should succeed regardless of whether there might already be a matching entry in the ACL DB. This can also be used with the -import_all_acls action. Alternatively, the -import_all_acls action can be called with the -replace_existing option.

Only one of -replace_existing or -overwrite_existing can be specified in conjunction with -import_all_acls; if both are provided the command will be rejected.

ACL Editor

The ACL Editor can be accessed by clicking the Advanced button in the Deephaven Console.

When selected, a dialog window will provide an option to rename the panel:

You may rename the panel title or accept the defaults. Click OK to open the DB ACL Editor window:

User configuration

The User Administration section in the User / Groups tab includes options for adding and deleting users, or changing an individual user's password.

Adding users

To add a new user, type the new user's username in the field directly under "User Administration". If Managed User Authentication is enabled, there will also be a field for the user's initial password. Click Add User to create the new user.

Deleting users

To delete a user, select the user's name in the drop-down menu to the left of the Delete User button, then click Delete User. This will delete the user, the user's eponymous group, and all Table ACL entries associated with that user.

Password requirements

The Change Password field indicates if the selected user has internal/Deephaven authentication enabled, or if in external (LDAP/SAML) method is required.

The ACL Editor will permit user-passwords to be set to (blank), which will disable password authentication for the given user. An existing user may also have their password "removed" in the Change Password section by leaving the password blank. This can by disabled by the boolean property DbAclEditor.permitBlankPassword, which defaults to true. When set to false, a user cannot be created without a Deephaven password, and a user may not have their password removed.

If the DbAclEditor.permitBlankPassword property is enabled, then each external-only user will be identified in the InternalPasswordDisabled column at the right of the ACL Editor table.

Changing passwords

If Managed User Authentication is enabled, it is possible to change a user's password in the ACL editor. To change an existing user's password, select the user's name in the drop-down menu to the left of the Change Password button, enter the new password in the next field, and then click Change Password.

Note: Change password operations are only available when iris.enableManagedUserAuthentication is set to true.

Additionally, the authentication server must have MySQL authentication enabled via:

• authentication.server.customauth.class=com.illumon.iris.db.v2.permissions.MysqlDbAclProvider

Or etcd authentication enabled with:

• authentication.server.customauth.class=com.illumon.iris.db.v2.permissions.EtcdDbAclProvider

Group adminstration

The Group Administration section in the User/Groups tab includes options for adding users to groups, and deleting groups.

Adding groups

To add a user to a specific group, select the user's name in the drop-down menu directly below Group Administration. Then select the Group Name in the next drop-down field and click Add User to Group.

Deleting groups

To delete a group, select the group name from the drop-down list to the left of the Delete Group button, then click Delete Group.

The configuration property DbAclEditor.SystemGroups defines groups that cannot be deleted from the ACL Editor. The default includes the groups:

• superusers
• acleditors
• schemamanagers
• querymanagers
• allusers

If the system administrator wishes to add other groups to this list, they must ensure that the existing five groups are included in the list.

Creating a new group

Each group must have at least one member. Therefore, to create a new group, you must first select a user's name in the drop-down menu directly below the Group Administration title. Then, highlight all the text in the Group field to the right, and type in the name of the new group. You will be asked to confirm the name of the new group. Then, click Add User To Group.

The image below demonstrates a user named Fred being added to a new group called NewCoolGroup:

A list showing User names and Group names is presented in the lower portion of the panel.

This list can be filtered by entering an applicable term in the Filter field above the list. Regular expressions can be used in the filter after selecting the Regex check box. Default group mappings can be shown by selecting the Show Default Group Mappings check box.

Deleting a user from a group

To delete a user from a specific group, find the row in the table that shows the appropriate user's name and the group name for which he/she should be removed. Right-click anywhere in the row and select Delete User To Group Mappings from the context menu.

After Delete User to Group Mappings is selected, a dialog window will prompt you to confirm the deletion.

Special groups

The following groups are used to provide specific access in Deephaven:

• allusers - All Deephaven users are assigned to this group.
  • By default, members of this group may open interactive consoles and create persistent queries; this is considered a "full-access" group.
  • The name of the group allowed to perform these operations can be changed by updating the following property: deephaven.access.fullaccess.group=allusers
  • For example, to create a deephaven-fullaccess group for full access, the following property could be added: deephaven.access.fullaccess.group=deephaven-fullaccess A user not assigned to the deephaven-fullaccess group would not be allowed to create persistent queries or start an interactive console.
• deephaven-noninteractive - Members of this group are not able to open a console, but will be allowed access through the controller if they have the required privileges (i.e., they can start and stop workers, create queries, and so on). They will also be allowed to connect to a query if they have the required privileges for that query.
  • The name of the group assigned to this privilege level can be changed by editing the following property: deephaven.access.noninteractive.group=deephaven-noninteractive
  • This group does not exist when the system is first installed. See: Creating a new group above.
• deephaven-queryviewonly - Members of this group can view queries, but are not allowed to create or edit queries.
  • If a user is not assigned to a different privileged group, they are presumed to be in this group (in a default installation, all users are able to create and edit queries because of the allusers group).
  • The name of the group assigned to this privilege level can be changed by editing the following property: deephaven.access.queryviewonly.group=deephaven-queryviewonly
  • This group does not exist when the system is first installed. See: Creating a new group above.
• iris-acleditors - Members of this group can use the ACL Editor panel.
• iris-datamanagers - Members of this group are privileged users who can:
  • create import, merge, and data validation queries, and in-worker services.
  • create live queries and batch queries to run on merge servers.
  • Note that a data validation query can delete intraday data, so a member of this group can also do this.
• iris-dataimporters - Members of this group can:
  • create import queries and in-worker services.
  • create live queries and batch queries on merge servers.
  • If they are not a member of a group with those privileges, they cannot create merge or data validation queries.
• iris-datamergers - Members of this group can create merge queries.
• iris-querymanagers - Members of this group can stop and start all of the queries, but are not allowed to edit them or see the tables (unless otherwise permissioned).
• iris-schemamanagers - Members of this group can use the Schema Editor panel.
• iris-superusers - Members of this group can view, edit, start, stop, and delete any persistent queries and all tables within a persistent query.

Group supervisors

Another level of authorization is available for group supervisors. Group supervisors automatically have owner/admin rights for persistent queries owned by anyone in the group. This is accomplished by adding the supervisor to a new group named with the following syntax: <group name>-supervisors.

For example, for a group named quantpros, a supervisor of that group can be granted owner/admin authorization for that group by adding him/her to a new group quantpros-supervisors. Multiple individuals can have the supervisor authorization for a group.

Accounts

The Accounts tab is only visible if configured by the Deephaven system administrator using the DbAclEditor.showAccountTab property.

The Accounts tab only applies to tables filtered with AccountFilterGenerator() and StrategyFilterGenerator(), and provides two levels of mappings from groups to accounts.

Groups are associated with a "Strategy" (e.g., "TradeGroup1", "TradeGroup2" or "TradeGroup3"). "Strategies" are associated with Accounts (e.g., "TradeGroup1" is associated with "ABC_ACCOUNT", "XYZ_ACCOUNT", and "MNO_ACCOUNT").

This extra level of indirection makes it simpler to add a user to a set of accounts.

Table ACLs

The Table ACLs panel enables the authorization of group access to namespaces and tables. Before returning a table to a user, Deephaven applies all of the ACLs for that user.

By default, regular users (i.e., not part of the iris-superusers group) have very limited table access. They can access some data in DbInternal tables, such as log entries for workers that they started and for persistent queries that they own, and (if it is installed), they can view data from the LearnDeephaven sample namespace. All other table permissions must be explicitly granted to regular users.

To configure the system so that users are automatically granted read access to user tables created in a namespace that matches their user name, add a table ACL that uses the OwnNamespaceFilterGenerator. This filter generator allows all users to read from any tables in a namespace that matches their user name. Typically this ACL is added for the allusers group, with * for namespace and * for table, but more restrictive ACLs can be used. One such possibility would be setting an OwnNamespaceFilterGenerator ACL for a UserTablesUsers group so that only members of that group automatically have access to user tables in a namespace that matches their user name.

Row ACLs

Row ACLs determine the rows that a user can see in the table; they cannot change the columns that a user can see in a table. Row ACLs are implemented with where operations that are automatically applied to the table.

Each Row ACL consists of four components:

• Group: The name of the group to which this ACL is associated.
• Namespace: The namespace associated with this ACL, to apply to all tables. This may be the wild card *, which means all namespaces are included. Note: If the namespace is *, the table must also be *.
• Table: The table with which this ACL is associated. The wild card, *, applies this entry to all tables within the namespace.
• Table Filter: A string that represents Java code to create a filter for this table.

Table filters

Using Java code to generate the filters provides additional flexibility. For example, a filter generator could be new AccountFilterGenerator(), which would filter the Account column based on the accounts for which the current user is authorized.

However, writing Java code to generate the filters can be a bit tedious, so there are two shorthand notations available:

• * means that all rows in the table should be presented to the user, and
• if a filter generator is of the form whereClause(A, B), whereClause(C, D), then the rows that match (A && B) || (C && D) are available to the user.

To compute the ACLs that should be used for a given user, the following steps are taken:

1. All of the user's groups are collected into a list.
2. For each group, get the ACL generator for the table, namespace, or default (in that order).
3. If no ACL generator was found, then deny permission.
4. Run all of the ACL generators (unless one returns the empty set of filters, which means permit everything).
5. If all generators returned null, deny permission (Note: If a generator returns null, that means deny; if it returns an empty list, that means permit everything).
6. Apply a .whereOneOf() operation to the table with all the returned filters.

When computing the list of ACLs for a table, the most specific ACL for a particular group is used. For example, if "UserXYZ" belongs to the groups "groupABC", "groupXYZ", and "allusers", and if the following ACLs exist:

Then, the ACL generators that would be used for the SystemEQ.PositionCache table are:

• new OwnNamespaceFilterGenerator() for allusers, because no more specific ACL exists for that group.
• new AccountFilterGenerator() for groupABC, because SystemEQ.PositionCache is more specific than SystemEQ.* (the * filter would be unused in this case).
• whereClause(`false`) for groupXYZ

The OwnNamespaceFilterGenerator would return null because the SystemEQ namespace is not equal to the username UserXYZ, thus contributing no filters. The AccountFilterGenerator would return a match filter for the account column. Finally, the whereClause(`false`) produces a match filter with no results. This has the effect of allowing the user to see an empty table, but without denying permission. When taken together, groupABC could see rows in SystemEQ.PositionCache that have the appropriate accounts configured.

If any other table from the SystemEQ namespace were requested, then the SystemEQ.* ACL would be the most specific ACL for group ABC, resulting in an ACL of *, which would short-circuit further evaluation; and all rows would be visible.

Column ACLs

Column ACLs provide administrators a way to restrict access to data on a columnar level. This stands in contrast to Table ACLs which restrict access by explicitly filtering rows out of the table. This can be useful when all users must have access to some subset of the data in all rows of a particular table. For example, a manager may need to be able to monitor the current execution of all orders within a system, but may not be allowed access to pricing data for some, or all of the symbols.

The rules for Column ACL application follow the same rules as for Table (Row) ACLs. The most specific ACL for each group is the one applied. Additionally, Column ACLs are defined with a comma-separated list of column names to which the ACL applies.

Each Column ACL consists of five components:

• Group: The name of the group to which this ACL is associated.
• Namespace: The namespace associated with this ACL, or * for all namespaces.
• Table: The table with which this ACL is associated, or * for all tables.
• Applicable Columns: The set of columns in the table for which this ACL applies, or * for all columns. Note that the columns must exist for the table or tables denoted by the above parameters.
• Column Filter: A string that represents the filter for the specified columns.

Filter generators

The filter generator string in the ACL should create a FilterGenerator object. Most of these strings will use one of the following built-in filter generators:

• EmptyFilterGenerator() - Provides full access to a table; it can be abbreviated as *.
• SimpleFilterGenerator() - Passes through a simple where clause; it can be abbreviated by using a filter generator string beginning with whereClause.
• AccountFilterGenerator() - Creates a match filter for accounts for which this user is authorized; suitable for restricting position logs, trade logs, etc. to just the accounts a user should see. By default, this filters the Account column, but an optional column name can be passed into the generator.
• StrategyFilterGenerator() - Creates a match filter for strategies for which this user is authorized. By default, this filters the Strategy column, but an optional column name can be passed in to the generator.
• OwnNamespaceFilterGenerator() - Produces an empty filter (i.e., allow everything) when the namespace matches the effective user name; otherwise produces null (i.e., no filter). This allows users to have full access to their own user tables.
• UsernameFilterGenerator() - Creates a match filter for this user's username. By default, this filters the Username column, but an optional column name can be passed in to the generator.
• UserCollectionFilterGenerator() - Creates a filter for this user's username within a collection. By default, this filters the Username column, but an optional column name can be passed in to the generator. This is useful, for example if a table column contains a list of authorized users for each row.
• GroupFilterGenerator() - Creates a RefreshingGroupFilter for a user. The RefreshinGroupFilter matches String columns containing a group that the user is a member of. By default, this filters the Group column, but an optional column name can be passed in to the generator. If the user’s group membership changes, then the filter will update on a default period of 60 seconds for live tables. For static tables, the groups will not refresh.
• StaticGroupFilterGenerator() Creates a match filter for groups to which this user belongs. By default, this filters the Group column, but an optional column name can be passed in to the generator. The user’s group list is determined at table access time and is not updated. The user’s group that matches their username is excluded from the filter, which allows the filter to be memoized for two users that belong to the same groups.
• WorkerNameFilterGenerator() - Produces a filter that matches only this worker. By default, this filters the WorkerName column, but an optional column name can be passed in to the generator. This is useful for filtering query logs.
  • Note: the QueryPerformanceLog, UpdatePerformanceLog, and QueryOperationPerformanceLog contain several user name columns which may be used for filtering in the latest version of Deephaven, so this FilterGenerator is no longer commonly used.
• RequestIdFromWorkerNameFilterGenerator() - Produces a dynamic where filter based on the DbInternal.QueryPerformanceLog table (we know the worker of a database, and this log lets us correlate a worker with a request ID). By default, this filters the RequestId column, but an optional column name can be passed in to the generator.
  • Note: the QueryPerformanceLog, UpdatePerformanceLog, and QueryOperationPerformanceLog contain several user name columns which may be used for filtering in the latest version of Deephaven, so this FilterGenerator is no longer commonly used.
• NullFilterGenerator() - Provides no access to a table; only useful to override a less-specific ACL for a group.
• ConjunctiveFilterGenerator(fg1, fg2, ...) - Runs filter generators fg1, fg2, etc., and returns a filter that is the conjunction of the filters returned by the passed-in filter generators. You can use this to further restrict the output of another filter generator.
• CopyFilterGenerator(Namespace, Table) - Copies the ACLs from another table to this table.
• CombiningFilterGenerator(Type) - ACLs are normally applied disjunctively. In some cases, users may want to override this behavior and combine some group filters conjunctively instead. This generator is initialized with the operation type desired (Type.Conjunctive or Type.Disjunctive) and as a single method addFilter(group, generator) to which specific filter generators can be added. Note that this generator is intended for programmatic ACLs.

Permission Analyzer

The Permission Analyzer can be used to review the table permissions associated with any particular user. To access this tool, click the Permission Analyzer button on the Table ACLs panel. Then select the appropriate User, Namespace and TableName in the drop-down menus to review the permissions granted. The example below shows the permissions for user123 in regard to the DXFeed namespace and the TradeStock table. The bottom part of the panel shows that full read permission is allowed for that user (as noted by the *).

Selecting the Show groups without ACLs check box adds the names of the groups associated with that user and each group's respective permission levels.

System ACLs

The System ACLs panel permits users in the acleditors group to add, remove, and edit system ACLs, similar to how they can add, remove, and edit table ACLs.

Each System ACL consists of three components:

• Group: The name of the group to which this ACL is associated.
• Key: System ACLs have a single key and are are not limited in scope to a single namespace or table. This key may be entered directed in the Key field and does not need to be predefined.
• Filter: A string that represents Java code to create a filter. This works the same way as Table filters.

Input Tables

Setting the authorization for individual/group access for reading data in tables is accomplished through the Table ACLs tab. However, setting the authorization of individuals or groups to edit tables is accomplished in the Input Tables tab.

Persistent Query ACLs

Just as Access Control Lists can be applied to tables, they can also be applied to persistent queries. The owner of a query, as well as designated administrator groups, have full access to the query. They can see every table and are able to edit the query, restart it, stop it, etc. The owner and administrator can also select groups that can view the query. Viewer groups are not able to view the query source code, stop, start, or edit the query.

However, permissions do not carryover to the plots generated by those same persistent queries. To set access control for viewing these plots, you must use the setValidGroups method in the persistent query that generates the plot. See setting permissions for viewing plots.

Tables

As part of the Groovy code, filters can be added to tables. These filters do not take effect for the owner, admin, or members of the iris-superusers group; only for viewers. If no filters are defined on any table, then viewers can see all of the rows in all of the tables produced by the query. If there is a filter defined for at least one table, then viewers cannot see any tables without filters. This has the effect of making it convenient to let people see the full results of a query. Assuming care is taken to restrict part of the query, the parts that have not been addressed from a security perspective are hidden.

Row and Column filters are applied to tables using a TableFilterProvider instance, created as follows:

TableFilterProvider.FACTORY.create(db, table)

Operations to add Row or Column ACLs can then be chained to apply the desired permissions to the table.

Row ACLs are applied using the following method:

.addFilter("group", "filter1", "filter2", ...)

• db - is the database object
• group is the group to which access is provided,
• filter1, filter2, etc. are filter generator expressions to add (disjunctively). The format of the filter expressions is the same as for Table ACLs defined in the DB ACL Editor.

Column ACLs are applied using the following method:

.addColumnACL(“group”,”Column1,Column2,...”, “filter1”, “filter2”, ...)

Filters can be chained, for example:

TableFilterProvider.FACTORY.create(db, table)
    .addFilter("group1", "*")
    .addFilter("group2", "whereClause(\"A=`B`\")")
    .addFilter("group2", "whereClause(\"A=`C`\", \"D=`E`\")")
    .addColumnACL(“group3”, “Bid,Ask”, “new GroupFilterGenerator(\”MarketDataGrp\”)”)

In this example, group1 may access the entire table, but group2 can only view rows where A is equal to B, or rows where A is equal to C, and D is equal to E. Additionally, group3 can only view the specified rows.

Plots

The ability to see and edit persistent queries and the tables they generate is determined by the settings used by the query author in the Access Control tab of the Persistent Query Configuration panel. However, those permissions do not carryover to the plots generated by those same persistent queries.

To set access control for viewing these plots, you must use the setValidGroups method in the persistent query that generates the plot.

.setValidGroups(String... validGroups)

.setValidGroups(Collection<String> validGroups)

To limit viewing access to a particular plot, first write the query to create and show the plot. Then, assign the setValidGroups method to that plot.

For example, the following query creates an XY Series chart to show the price of a single security (PFE) over the course of one day. No viewing restrictions are currently in place.

t1 = db.t("LearnDeephaven","StockTrades").where("Date=`2017-08-24`","USym=`PFE`")
PlotPFE = plot("PFE", t1, "Timestamp", "Last")
    .xBusinessTime()
    .show()

To set the viewing permissions for this plot, add the following line to the query.

PlotPFE.setValidGroups("User1", "User2", "Group1")

When this query is executed in Deephaven, only User1, User2, and individuals included in Group1 would be allowed to open and view the plot using the Show Widget button in the Deephaven console.

For example, if User3 was not included in Group1, User3 would not be able to view PlotPFE even if User3 was authorized to view the persistent query and/or the tables generated by the persistent query.

Keep in mind that the setValidGroups method does not differentiate the data shown to each user. The widget can make use of the IrisWidgetSupport getIrisUserContext method to restrict information based on the user.

The LiveWidgetVisibilityProvider interface can be implemented by a widget to restrict the set of users that can load the widget. However, it does not differentiate the data shown to each user. The widget can make use of the IrisWidgetSupport getIrisUserContext method to restrict information based on the user.