Package io.deephaven.engine.table.impl

Class QueryTable

java.lang.Object
io.deephaven.util.referencecounting.ReferenceCounted
io.deephaven.engine.liveness.ReferenceCountedLivenessReferent
io.deephaven.engine.liveness.ReferenceCountedLivenessNode
io.deephaven.engine.liveness.LivenessArtifact
io.deephaven.engine.table.impl.LiveAttributeMap<IFACE_TYPE,IMPL_TYPE>
io.deephaven.engine.table.impl.BaseGridAttributes<Table,IMPL_TYPE>
io.deephaven.engine.table.impl.BaseTable<QueryTable>
io.deephaven.engine.table.impl.QueryTable
All Implemented Interfaces:
TableOperations<Table,Table>, TableOperationsDefaults<Table,Table>, LogOutputAppendable, LivenessManager, LivenessNode, LivenessReferent, AttributeMap<Table>, GridAttributes<Table>, NotificationStepReceiver, NotificationStepSource, TableDefaults, Table, DynamicNode, NotificationQueue.Dependency, SystemicObject<Table>, LongSizedDataStructure, Serializable
Direct Known Subclasses:
BarrageTable, InitialSnapshotTable, InMemoryTable, QueryTable.FilteredTable, ReplayTableBase, TimeTable, UpdatableTable, UpdateSourceQueryTable
public class QueryTable extends BaseTable<QueryTable>
Primary coalesced table implementation.
See Also:

  • Field Details

    • MINIMUM_PARALLEL_SELECT_ROWS

      public static long MINIMUM_PARALLEL_SELECT_ROWS
      Minimum select "chunk" size, defaults to 4 million.

    • ENABLE_PARALLEL_SNAPSHOT

      public static boolean ENABLE_PARALLEL_SNAPSHOT
      You can choose to enable or disable the parallel snapshot.

    • MINIMUM_PARALLEL_SNAPSHOT_ROWS

      public static long MINIMUM_PARALLEL_SNAPSHOT_ROWS
      Minimum snapshot "chunk" size for parallel reading of columns, defaults to 1 million.

    • TRACKED_LAST_BY

      @VisibleForTesting public static boolean TRACKED_LAST_BY

    • TRACKED_FIRST_BY

      @VisibleForTesting public static boolean TRACKED_FIRST_BY

    • USE_OLDER_CHUNKED_BY

      @VisibleForTesting public static boolean USE_OLDER_CHUNKED_BY

    • USE_CHUNKED_CROSS_JOIN

      @VisibleForTesting public static boolean USE_CHUNKED_CROSS_JOIN

  • Constructor Details

    • QueryTable

      public QueryTable(@NotNull @NotNull TrackingRowSet rowSet, @NotNull @NotNull Map<String,? extends ColumnSource<?>> columns)
      Creates a new table, inferring a definition but creating a new column source map.
      Parameters:
      rowSet - The RowSet of the new table. Callers may need to convert.
      columns - The column source map for the table, which will be copied into a new column source map

    • QueryTable

      public QueryTable(@NotNull @NotNull TableDefinition definition, @NotNull @NotNull TrackingRowSet rowSet, @NotNull @NotNull Map<String,? extends ColumnSource<?>> columns)
      Creates a new table, reusing a definition but creating a new column source map.
      Parameters:
      definition - The definition to use for this table, which will be re-ordered to match the same order as columns if it does not match
      rowSet - The RowSet of the new table. Callers may need to convert.
      columns - The column source map for the table, which will be copied into a new column source map

    • QueryTable

      public QueryTable(@NotNull @NotNull TableDefinition definition, @NotNull @NotNull TrackingRowSet rowSet, @NotNull @NotNull LinkedHashMap<String,ColumnSource<?>> columns, @Nullable @Nullable ModifiedColumnSet modifiedColumnSet, @Nullable @Nullable Map<String,Object> attributes)
      Creates a new table, reusing a definition and column source map.
      Parameters:
      definition - The definition to use for this table, which will not be validated or re-ordered.
      rowSet - The RowSet of the new table. Callers may need to convert.
      columns - The column source map for the table, which is not copied.
      modifiedColumnSet - Optional ModifiedColumnSet that should be re-used if supplied
      attributes - Optional value to use for initial attributes

  • Method Details

    • withDefinitionUnsafe

      @Deprecated public QueryTable withDefinitionUnsafe(TableDefinition template)
      Deprecated.
      this is being used a workaround for testing purposes where previously mutations were being used at the ColumnDefinition level. Do not use this method without good reason.
      Create a new query table with the ColumnDefinitions of template, but in the order of this. The tables must be mutually compatible, as defined via TableDefinition.checkMutualCompatibility(TableDefinition).
      Parameters:
      template - the new definition template to use
      Returns:
      the new query table

    • getRowSet

      public TrackingRowSet getRowSet()
      Returns:
      The TrackingRowSet that exposes the row keys present in this Table

    • size

      public long size()
      Description copied from interface: LongSizedDataStructure
      The size of this data structure.
      Returns:
      The size

    • getColumnSource

      public <T> ColumnSource<T> getColumnSource(String sourceName)
      Description copied from interface: Table
      Retrieves a ColumnSource. It is conveniently cast to ColumnSource<Object> using the type that caller expects. This differs from Table.getColumnSource(String, Class) which uses the provided Class object to verify that the data type is a subclass of the expected class.

      The success of this call is equivalent to getDefinition().checkColumn(sourceName), which is the preferred way to check for compatibility in scenarios where the caller does not want the implementation to potentially invoke Table.coalesce().

      Type Parameters:
      T - The target type, as a type parameter. Inferred from context.
      Parameters:
      sourceName - The name of the column
      Returns:
      The column source for sourceName, parameterized by T
      See Also:

    • getColumnSourceMap

      public Map<String,ColumnSource<?>> getColumnSourceMap()

    • getColumnSources

      public Collection<? extends ColumnSource<?>> getColumnSources()

    • columnIterator

      public <TYPE> CloseableIterator<TYPE> columnIterator(@NotNull @NotNull String columnName)

    • characterColumnIterator

      public CloseablePrimitiveIteratorOfChar characterColumnIterator(@NotNull @NotNull String columnName)

    • byteColumnIterator

      public CloseablePrimitiveIteratorOfByte byteColumnIterator(@NotNull @NotNull String columnName)

    • shortColumnIterator

      public CloseablePrimitiveIteratorOfShort shortColumnIterator(@NotNull @NotNull String columnName)

    • integerColumnIterator

      public CloseablePrimitiveIteratorOfInt integerColumnIterator(@NotNull @NotNull String columnName)

    • longColumnIterator

      public CloseablePrimitiveIteratorOfLong longColumnIterator(@NotNull @NotNull String columnName)

    • floatColumnIterator

      public CloseablePrimitiveIteratorOfFloat floatColumnIterator(@NotNull @NotNull String columnName)

    • doubleColumnIterator

      public CloseablePrimitiveIteratorOfDouble doubleColumnIterator(@NotNull @NotNull String columnName)

    • objectColumnIterator

      public <DATA_TYPE> CloseableIterator<DATA_TYPE> objectColumnIterator(@NotNull @NotNull String columnName)

    • getModifiedColumnSetForUpdates

      public ModifiedColumnSet getModifiedColumnSetForUpdates()
      Producers of tables should use the modified column set embedded within the table for their result.

      You must not mutate the result of this method if you are not generating the updates for this table. Callers should not rely on the dirty state of this modified column set.

      Returns:
      the modified column set for this table

    • newModifiedColumnSet

      public ModifiedColumnSet newModifiedColumnSet(String... columnNames)
      Create a ModifiedColumnSet to use when propagating updates from this table.
      Parameters:
      columnNames - The columns that should belong to the resulting set
      Returns:
      The resulting ModifiedColumnSet for the given columnNames

    • newModifiedColumnSetTransformer

      public ModifiedColumnSet.Transformer newModifiedColumnSetTransformer(QueryTable resultTable, String... columnNames)
      Create a ModifiedColumnSet.Transformer that can be used to propagate dirty columns from this table to listeners of the provided resultTable.
      Parameters:
      resultTable - the destination table
      columnNames - the columns that map one-to-one with the result table
      Returns:
      a transformer that passes dirty details via an identity mapping

    • newModifiedColumnSetTransformer

      public ModifiedColumnSet.Transformer newModifiedColumnSetTransformer(@NotNull @NotNull QueryTable resultTable, @NotNull @NotNull MatchPair... matchPairs)
      Create a ModifiedColumnSet.Transformer that can be used to propagate dirty columns from this table to listeners of the provided resultTable.
      Parameters:
      resultTable - the destination table
      matchPairs - the columns that map one-to-one with the result table
      Returns:
      a transformer that passes dirty details via an identity mapping

    • newModifiedColumnSetTransformer

      public ModifiedColumnSet.Transformer newModifiedColumnSetTransformer(@NotNull @NotNull QueryTable resultTable, @NotNull @NotNull Pair... pairs)
      Create a ModifiedColumnSet.Transformer that can be used to propagate dirty columns from this table to listeners of the provided resultTable.
      Parameters:
      resultTable - the destination table
      pairs - the columns that map one-to-one with the result table
      Returns:
      a transformer that passes dirty details via an identity mapping

    • newModifiedColumnSetTransformer

      public ModifiedColumnSet.Transformer newModifiedColumnSetTransformer(String[] columnNames, ModifiedColumnSet[] columnSets)
      Create a ModifiedColumnSet.Transformer that can be used to propagate dirty columns from this table to listeners of the table used to construct columnSets. It is an error if columnNames and columnSets are not the same length. The transformer will mark columnSets[i] as dirty if the column represented by columnNames[i] is dirty.
      Parameters:
      columnNames - the source columns
      columnSets - the destination columns in the convenient ModifiedColumnSet form
      Returns:
      a transformer that knows the dirty details

    • newModifiedColumnSetIdentityTransformer

      public ModifiedColumnSet.Transformer newModifiedColumnSetIdentityTransformer(Map<String,ColumnSource<?>> newColumns)
      Create a transformer that uses an identity mapping from one ColumnSourceMap to another. The two CSMs must have equivalent column names and column ordering.
      Parameters:
      newColumns - the column source map for result table
      Returns:
      a simple Transformer that makes a cheap, but CSM compatible copy

    • newModifiedColumnSetIdentityTransformer

      public ModifiedColumnSet.Transformer newModifiedColumnSetIdentityTransformer(Table other)
      Create a transformer that uses an identity mapping from one Table another. The two tables must have equivalent column names and column ordering.
      Parameters:
      other - the result table
      Returns:
      a simple Transformer that makes a cheap, but CSM compatible copy

    • partitionBy

      public PartitionedTable partitionBy(boolean dropKeys, String... keyColumnNames)
      Description copied from interface: Table
      Create a PartitionedTable from this table, partitioned according to the specified key columns.

      The underlying partitioned table backing the result contains each row in this table in exactly one of the result's constituent tables.

      Parameters:
      dropKeys - Whether to drop key columns in the output constituent tables
      keyColumnNames - The names of the key columns to partition by
      Returns:
      A PartitionedTable keyed by keyColumnNames

    • partitionedAggBy

      public PartitionedTable partitionedAggBy(Collection<? extends Aggregation> aggregations, boolean preserveEmpty, @Nullable @Nullable Table initialGroups, String... keyColumnNames)
      Description copied from interface: Table
      Convenience method that performs an TableOperations.aggBy(io.deephaven.api.agg.Aggregation) and wraps the result in a PartitionedTable. If aggregations does not include a partition, one will be added automatically with the default constituent column name and behavior used in Table.partitionBy(String...).
      Parameters:
      aggregations - The aggregations to apply
      preserveEmpty - Whether to keep result rows for groups that are initially empty or become empty as a result of updates. Each aggregation operator defines its own value for empty groups.
      initialGroups - A table whose distinct combinations of values for the groupByColumns should be used to create an initial set of aggregation groups. All other columns are ignored. This is useful in combination with preserveEmpty == true to ensure that particular groups appear in the result table, or with preserveEmpty == false to control the encounter order for a collection of groups and thus their relative order in the result. Changes to initialGroups are not expected or handled; if initialGroups is a refreshing table, only its contents at instantiation time will be used. If initialGroups == null, the result will be the same as if a table with no rows was supplied.
      keyColumnNames - The names of the key columns to aggregate by
      Returns:
      A PartitionedTable keyed by keyColumnNames

    • rollup

      public RollupTable rollup(Collection<? extends Aggregation> aggregations, boolean includeConstituents, Collection<? extends ColumnName> groupByColumns)
      Description copied from interface: Table
      Create a rollup table.

      A rollup table aggregates by the specified columns, and then creates a hierarchical table which re-aggregates using one less aggregation column on each level. The column that is no longer part of the aggregation key is replaced with null on each level.

      Parameters:
      aggregations - The aggregations to perform
      includeConstituents - set to true to include the constituent rows at the leaf level
      groupByColumns - the columns to group by
      Returns:
      a hierarchical table with the rollup applied

    • tree

      public TreeTable tree(String idColumn, String parentColumn)
      Description copied from interface: Table
      Create a hierarchical tree table.

      The structure of the table is encoded by an "id" and a "parent" column. The id column should represent a unique identifier for a given row, and the parent column indicates which row is the parent for a given row. Rows that have a null parent are part of the "root" table.

      It is possible for rows to be "orphaned" if their parent is non-null and does not exist in the table. See TreeTable.promoteOrphans(Table, String, String).

      Parameters:
      idColumn - The name of a column containing a unique identifier for a particular row in the table
      parentColumn - The name of a column containing the parent's identifier, null for rows that are part of the root table
      Returns:
      A TreeTable organized according to the parent-child relationships expressed by idColumn and parentColumn

    • slice

      public Table slice(long firstPositionInclusive, long lastPositionExclusive)
      Description copied from interface: Table
      Extracts a subset of a table by row position.

      If both firstPosition and lastPosition are positive, then the rows are counted from the beginning of the table. The firstPosition is inclusive, and the lastPosition is exclusive. The TableOperations.head(long)(N) call is equivalent to slice(0, N). The firstPosition must be less than or equal to the lastPosition.

      If firstPosition is positive and lastPosition is negative, then the firstRow is counted from the beginning of the table, inclusively. The lastPosition is counted from the end of the table. For example, slice(1, -1) includes all rows but the first and last. If the lastPosition would be before the firstRow, the result is an emptyTable.

      If firstPosition is negative, and lastPosition is zero, then the firstRow is counted from the end of the table, and the end of the slice is the size of the table. slice(-N, 0) is equivalent to TableOperations.tail(long)(N).

      If the firstPosition is negative and the lastPosition is negative, they are both counted from the end of the table. For example, slice(-2, -1) returns the second to last row of the table.

      If firstPosition is negative and lastPosition is positive, then firstPosition is counted from the end of the table, inclusively. The lastPosition is counted from the beginning of the table, exclusively. For example, slice(-3, 5) returns all rows starting from the third-last row to the fifth row of the table. If there are no rows between these positions, the function will return an empty table.

      Parameters:
      firstPositionInclusive - the first position to include in the result
      lastPositionExclusive - the last position to include in the result
      Returns:
      a new Table, which is the request subset of rows from the original table

    • slicePct

      public Table slicePct(double startPercentInclusive, double endPercentExclusive)
      Description copied from interface: Table
      Extracts a subset of a table by row percentages.

      Returns a subset of table in the range [floor(startPercentInclusive * sizeOfTable), floor(endPercentExclusive * sizeOfTable)). For example, for a table of size 10, slicePct(0.1, 0.7) will return a subset from the second row to the seventh row. Similarly, slicePct(0, 1) would return the entire table (because row positions run from 0 to size-1). The percentage arguments must be in range [0,1], otherwise the function returns an error.

      Parameters:
      startPercentInclusive - the starting percentage point for rows to include in the result, range [0, 1]
      endPercentExclusive - the ending percentage point for rows to include in the result, range [0, 1]
      Returns:
      a new Table, which is the requested subset of rows from the original table

    • head

      public Table head(long size)

    • tail

      public Table tail(long size)

    • headPct

      public Table headPct(double percent)
      Description copied from interface: Table
      Provides a head that selects a dynamic number of rows based on a percent.
      Parameters:
      percent - the fraction of the table to return between [0, 1]. The number of rows will be rounded up. For example if there are 3 rows, headPct(50) returns the first two rows. For percent values outside [0, 1], the function will throw an exception.

    • tailPct

      public Table tailPct(double percent)
      Description copied from interface: Table
      Provides a tail that selects a dynamic number of rows based on a percent.
      Parameters:
      percent - the fraction of the table to return between [0, 1]. The number of rows will be rounded up. For example if there are 3 rows, tailPct(50) returns the last two rows. For percent values outside [0, 1], the function will throw an exception.

    • exactJoin

      public Table exactJoin(Table rightTable, Collection<? extends JoinMatch> columnsToMatch, Collection<? extends JoinAddition> columnsToAdd)
      Description copied from interface: TableOperations
      Perform an exact-join with the rightTable.

      Similar to TableOperations.naturalJoin(Object, Collection, Collection), but requires that exactly one match from the rightTable.

      Parameters:
      rightTable - The right side table on the join.
      columnsToMatch - The match pair conditions.
      columnsToAdd - The columns from the right side that need to be added to the left side as a result of the match.
      Returns:
      the exact-joined table

    • aggAllBy

      public Table aggAllBy(AggSpec spec, ColumnName... groupByColumns)

    • aggBy

      public Table aggBy(Collection<? extends Aggregation> aggregations, boolean preserveEmpty, Table initialGroups, Collection<? extends ColumnName> groupByColumns)
      Description copied from interface: TableOperations
      Produce an aggregated result by grouping this according to the groupByColumns and applying aggregations to each resulting group of rows. The result table will have one row per group, ordered by the encounter order within this, thereby ensuring that the row key for a given group never changes.
      Parameters:
      aggregations - The aggregations to apply
      preserveEmpty - Whether to keep result rows for groups that are initially empty or become empty as a result of updates. Each aggregation operator defines its own value for empty groups.
      initialGroups - A table whose distinct combinations of values for the groupByColumns should be used to create an initial set of aggregation groups. All other columns are ignored. This is useful in combination with preserveEmpty == true to ensure that particular groups appear in the result table, or with preserveEmpty == false to control the encounter order for a collection of groups and thus their relative order in the result. Changes to initialGroups are not expected or handled; if initialGroups is a refreshing table, only its contents at instantiation time will be used. If initialGroups == null, the result will be the same as if a table with no rows was supplied.
      groupByColumns - The columns to group by
      Returns:
      A new table aggregating the rows of this

    • aggNoMemo

      public QueryTable aggNoMemo(@NotNull @NotNull AggregationContextFactory aggregationContextFactory, boolean preserveEmpty, @Nullable @Nullable Table initialGroups, @NotNull @NotNull Collection<? extends ColumnName> groupByColumns)

    • headBy

      public Table headBy(long nRows, String... groupByColumns)

    • tailBy

      public Table tailBy(long nRows, String... groupByColumns)

    • moveColumns

      public Table moveColumns(int index, String... columnsToMove)
      Description copied from interface: Table
      Produce a new table with the specified columns moved to the specified index. Column indices begin at 0. Columns can be renamed with the usual syntax, i.e. "NewColumnName=OldColumnName"). The renames are simultaneous and unordered, enabling direct swaps between column names. The resulting table retains the original column ordering except for the specified columns, which are inserted at the specified index, in the order of columnsToMove, after the effects of applying any renames.

      IllegalArgumentException will be thrown:

      • if a source column does not exist
      • if a source column is used more than once
      • if a destination column is used more than once

      Values of index outside the range of 0 to the number of columns in the table (exclusive) will be clamped to the nearest valid index.

      Parameters:
      index - The index to which the specified columns should be moved
      columnsToMove - The columns to move to the specified index (and, optionally, to rename)
      Returns:
      The new table, with the columns rearranged as explained above

    • where

      public Table where(Filter filter)

    • whereIn

      public Table whereIn(Table rightTable, Collection<? extends JoinMatch> columnsToMatch)
      Description copied from interface: TableOperations
      Filters this table based on the set of values in the rightTable.

      Note that when the rightTable ticks, all of the rows in this table are going to be re-evaluated, thus the intention is that the rightTable is fairly slow moving compared with this table.

      Parameters:
      rightTable - the filtering table.
      columnsToMatch - the columns to match between the two tables
      Returns:
      a new table filtered on right table

    • whereNotIn

      public Table whereNotIn(Table rightTable, Collection<? extends JoinMatch> columnsToMatch)
      Description copied from interface: TableOperations
      Filters this table based on the set of values not in the rightTable.

      Note that when the rightTable ticks, all of the rows in this table are going to be re-evaluated, thus the intention is that the rightTable is fairly slow moving compared with this table.

      Parameters:
      rightTable - the filtering table.
      columnsToMatch - the columns to match between the two tables
      Returns:
      a new table filtered on right table

    • flatten

      public Table flatten()
      Description copied from interface: Table
      Creates a version of this table with a flat RowSet.

    • setFlat

      public void setFlat()

    • isFlat

      public boolean isFlat()
      Description copied from interface: Table
      Return true if this table is guaranteed to be flat. The RowSet of a flat table will be from 0...numRows-1.

    • releaseCachedResources

      public void releaseCachedResources()
      Description copied from interface: Table
      Attempt to release cached resources held by this table. Unlike Table.close(), this must not render the table unusable for subsequent read operations. Implementations should be sure to call super.releaseCachedResources().

    • select

      public Table select(Collection<? extends Selectable> columns)

    • update

      public Table update(Collection<? extends Selectable> newColumns)

    • validateSelect

      public SelectValidationResult validateSelect(SelectColumn... selectColumns)
      This does a certain amount of validation and can be used to get confidence that the formulas are valid. If it is not valid, you will get an exception. Positive test (should pass validation): "X = 12", "Y = X + 1") Negative test (should fail validation): "X = 12", "Y = Z + 1")

      DO NOT USE -- this API is in flux and may change or disappear in the future.

    • view

      public Table view(Collection<? extends Selectable> viewColumns)

    • updateView

      public Table updateView(Collection<? extends Selectable> viewColumns)

    • lazyUpdate

      public Table lazyUpdate(Collection<? extends Selectable> newColumns)
      Description copied from interface: TableOperations
      Compute column formulas on demand.

      Lazy update defers computation until required for a set of values, and caches the results for a set of input values. This uses less RAM than an update statement when you have a smaller set of unique values. Less computation than an updateView is needed, because the results are saved in a cache.

      If you have many unique values, you should instead use an update statement, which will have more memory efficient structures. Values are never removed from the lazyUpdate cache, so it should be used judiciously on a ticking table.

      Parameters:
      newColumns - the columns to add
      Returns:
      a new Table with the columns added; to be computed on demand

    • dropColumns

      public Table dropColumns(String... columnNames)
      Description copied from interface: TableOperations
      Creates a new table without the columnNames from this.
      Parameters:
      columnNames - the columns to drop
      Returns:
      the table

    • renameColumns

      public Table renameColumns(Collection<Pair> pairs)
      Description copied from interface: Table
      Produce a new table with the specified columns renamed using the specified pairs. The renames are simultaneous and unordered, enabling direct swaps between column names. The resulting table retains the original column ordering after applying the specified renames.

      IllegalArgumentException will be thrown:

      • if a source column does not exist
      • if a source column is used more than once
      • if a destination column is used more than once
      Parameters:
      pairs - The columns to rename
      Returns:
      The new table, with the columns renamed

    • asOfJoin

      public Table asOfJoin(Table rightTable, Collection<? extends JoinMatch> exactMatches, AsOfJoinMatch asOfMatch, Collection<? extends JoinAddition> columnsToAdd)

    • naturalJoin

      public Table naturalJoin(Table rightTable, Collection<? extends JoinMatch> columnsToMatch, Collection<? extends JoinAddition> columnsToAdd)
      Description copied from interface: TableOperations
      Perform an exact-join with the rightTable.

      Requires zero or one match from the rightTable.

      Parameters:
      rightTable - The right side table on the join.
      columnsToMatch - The match pair conditions.
      columnsToAdd - The columns from the right side that need to be added to the left side as a result of the match.
      Returns:
      the natural-joined table

    • join

      public Table join(@NotNull @NotNull Table rightTable, @NotNull @NotNull Collection<? extends JoinMatch> columnsToMatch, @NotNull @NotNull Collection<? extends JoinAddition> columnsToAdd, int numRightBitsToReserve)
      Description copied from interface: TableOperations
      Perform a cross join with the rightTable.

      Returns a table that is the cartesian product of left rows X right rows, with one column for each of this table's columns, and one column corresponding to each of the rightTable's columns that are included in the columnsToAdd argument. The rows are ordered first by the this table then by the rightTable. If columnsToMatch is non-empty then the product is filtered by the supplied match conditions.

      To efficiently produce updates, the bits that represent a key for a given row are split into two. Unless specified, join reserves 16 bits to represent a right row. When there are too few bits to represent all of the right rows for a given aggregation group the table will shift a bit from the left side to the right side. The default of 16 bits was carefully chosen because it results in an efficient implementation to process live updates.

      An io.deephaven.engine.table.impl.util.OutOfKeySpaceException is thrown when the total number of bits needed to express the result table exceeds that needed to represent Long.MAX_VALUE. There are a few work arounds:

      - If the left table is sparse, consider flattening the left table.

      - If there are no key-columns and the right table is sparse, consider flattening the right table.

      - If the maximum size of a right table's group is small, you can reserve fewer bits by setting reserveBits on initialization.

      Note: If you know that a given group has at most one right-row then you should prefer using TableOperations.naturalJoin(Object, Collection, Collection).

      Parameters:
      rightTable - The right side table on the join.
      columnsToMatch - The match pair conditions.
      columnsToAdd - The columns from the right side that need to be added to the left side as a result of the match.
      numRightBitsToReserve - The number of bits to reserve for rightTable groups.
      Returns:
      a new table joined according to the specification in columnsToMatch and columnsToAdd

    • rangeJoin

      public Table rangeJoin(@NotNull @NotNull Table rightTable, @NotNull @NotNull Collection<? extends JoinMatch> exactMatches, @NotNull @NotNull RangeJoinMatch rangeMatch, @NotNull @NotNull Collection<? extends Aggregation> aggregations)
      Description copied from interface: TableOperations
      Perform a range join with rightTable. For each row in this Table, this operation joins aggregations over a range of responsive rows from rightTable according to zero-or-more exact join matches and one range join match.

      Matching Rules

      The exact join matches identify possibly-responsive rows according to exactly matching values between the left exact match columns and the right exact match columns, similar to other join operations. The range join match bounds the beginning of the responsive range for a given output row by the relationship between a left start column and a right range column, governed by the rangeStartRule, and bounds the end of the responsive range for a given output row by the relationship between a left end column and the same right range column, governed by the rangeEndRule.

      Right Table Row-Inclusion Criteria and Relative Ordering Requirements

      Rows from rightTable with null or NaN values for the right range column are discarded; that is, they are never included in the responsive range for any output row. Within a group sharing the same values for the right exact match columns, rightTable must be relatively ordered (as if sorted) according to the right range column for all rows that are not discarded.

      Special Cases

      In order to produce aggregated output, it is required that the two relative match expressions define a range of values to determine the responsive rows to aggregate. There are a few noteworthy special cases of ranges.

      empty range
      An empty range occurs for any left row with no responsive right rows. That is, no non-null, non-NaN right rows were found using the exact join matches, or none were in range according to the range join match.
      single-value ranges
      A single-value range is a range where the left row’s values for the left start column and left end column are equal and both relative matches are inclusive (<= and >=, respectively). For a single-value range, only rows within the bucket where the right range column matches the single value are included in the output aggregations.
      invalid ranges
      An invalid range occurs in two scenarios: First, when the range is inverted, i.e. when the value of the left start column is greater than the value of the left end column. Second, when either relative-match is exclusive (< or >) and the value in the left start column is equal to the value in the left end column (because value < value == false). Specifying "allow preceding" or "allow following" for either rule will not constitute an exception to either of these defined scenarios. For invalid ranges, the result row will be null for all aggregation output columns.
      undefined ranges
      An undefined range occurs when either the left start column or the left end column is NaN. For rows with an undefined range, the corresponding output values will be null (as with invalid ranges).
      Unbounded Ranges
      A partially or fully unbounded range occurs when either the left start column or the left end column is null. If the left start column value is null the range is unbounded at the beginning, and all matched right rows will be included if they respect the match rule for the left end column. If the left end column value is null the range is unbounded at the end, and all matched right rows will be included if they respect the match rule for the left start column. If both the left start column and left end column values are null the range is unbounded in both directions, and all matched right rows will be included.
      Parameters:
      rightTable - The Table to join with
      exactMatches - Possibly-empty collection of join matches that dictate exact-match criteria. That is, rows from rightTable that might be responsive to rows from this Table will have identical values for the column pairs expressed by these matches.
      rangeMatch - Specifies the range match criteria for determining the responsive rows from rightTable for each row from this Table, within the buckets created by matching on the exactMatches
      aggregations - The aggregations to perform over the responsive ranges from rightTable for each row from this Table
      Returns:
      The result Table

    • silent

      public Table silent()

    • snapshot

      public Table snapshot()
      Description copied from interface: TableOperations
      Creates a table with a single static snapshot of this.
      Returns:
      the snapshot

    • snapshotWhen

      public Table snapshotWhen(Table trigger, SnapshotWhenOptions options)
      Description copied from interface: TableOperations
      Creates a table that captures a snapshot of this whenever trigger updates.
      Parameters:
      trigger - the trigger table
      options - the snapshot options
      Returns:
      the snapshotting table
      See Also:

    • sort

      public Table sort(Collection<SortColumn> columnsToSortBy)

    • reverse

      public Table reverse()
      The reverse operation returns a new table that is the same as the original table, but the first row is last, and the last row is first. This is an internal API to be used by .raj(), but is accessible for unit tests.
      Returns:
      the reversed table

    • ungroup

      public Table ungroup(boolean nullFill, Collection<? extends ColumnName> columnsToUngroup)
      Description copied from interface: TableOperations
      Ungroups a table by expanding columns of arrays or vectors into columns of singular values, creating one row in the output table for each value in the columns to be ungrouped. Columns that are not ungrouped have their values duplicated in each output row corresponding to a given input row.
      Parameters:
      nullFill - indicates if the ungrouped table should allow disparate sized arrays filling shorter columns with null values. If set to false, then all arrays should be the same length.
      columnsToUngroup - the columns to ungroup
      Returns:
      the ungrouped table

    • selectDistinct

      public Table selectDistinct(Collection<? extends Selectable> columns)

    • propagateFlatness

      public void propagateFlatness(QueryTable result)

      If this table is flat, then set the result table flat.

      This function is for use when the result table shares a RowSet; such that if this table is flat, the result table must also be flat.

      Parameters:
      result - the table derived from this table

    • getSubTable

      public QueryTable getSubTable(@NotNull @NotNull TrackingRowSet rowSet)
      Get a Table that contains a sub-set of the rows from this. The result will share the same column sources and definition as this table.

      The result will not update on its own, the caller must also establish an appropriate listener to update rowSet and propagate updates.

      No nugget is opened for this table, to prevent operations that call this repeatedly from having an inordinate performance penalty. If callers require a nugget, they must create one in the enclosing operation.

      Parameters:
      rowSet - The result's row set
      Returns:
      A new table sharing this table's column sources with the specified row set

    • withAdditionalColumns

      public QueryTable withAdditionalColumns(@NotNull @NotNull Map<String,ColumnSource<?>> additionalSources)
      Get a Table that adds, or overwrites, columns from this. The result will share the same row set as this table.

      The result will not update on its own. The caller must also establish an appropriate listener to update the provided column sources and propagate updates.

      No attributes are propagated to the result table.

      Parameters:
      additionalSources - The additional columns to add or overwrite
      Returns:
      A new table with the additional columns

    • getSubTable

      public QueryTable getSubTable(@NotNull @NotNull TrackingRowSet rowSet, @Nullable @Nullable ModifiedColumnSet resultModifiedColumnSet, @Nullable @Nullable Map<String,Object> attributes, @NotNull @NotNull Object... parents)
      Get a Table that contains a sub-set of the rows from this. The result will share the same column sources and definition as this table.

      The result will not update on its own, the caller must also establish an appropriate listener to update rowSet and propagate updates.

      This method is intended to be used for composing alternative engine operations, in particular partitionBy(boolean, String...).

      No nugget is opened for this table, to prevent operations that call this repeatedly from having an inordinate performance penalty. If callers require a nugget, they must create one in the enclosing operation.

      Parameters:
      rowSet - The result's row set
      resultModifiedColumnSet - The result's modified column set, or null for default initialization
      attributes - The result's attributes, or null for default initialization
      parents - Parent references for the result table
      Returns:
      A new table sharing this table's column sources with the specified row set

    • copy

      public QueryTable copy()
      Copies this table, but with a new set of attributes.
      Specified by:
      copy in class LiveAttributeMap<Table,QueryTable>
      Returns:
      an identical table; but with a new set of attributes

    • copy

      public QueryTable copy(Predicate<String> shouldCopy)

    • copy

      public QueryTable copy(TableDefinition definition, Predicate<String> shouldCopy)

    • updateBy

      public Table updateBy(@NotNull @NotNull UpdateByControl control, @NotNull @NotNull Collection<? extends UpdateByOperation> ops, @NotNull @NotNull Collection<? extends ColumnName> byColumns)
      Description copied from interface: TableOperations
      Creates a table with additional columns calculated from window-based aggregations of columns in its parent. The aggregations are defined by the operations, which support incremental aggregation over the corresponding rows in the parent table. The aggregations will apply position or time-based windowing and compute the results for the row group (as determined by the byColumns).
      Parameters:
      control - the control to use when updating the table.
      ops - the operations to apply to the table.
      byColumns - the columns to group by before applying.
      Returns:
      a table with the same rowSet, with the specified operations applied to each group defined by the byColumns

    • setMemoizeResults

      @VisibleForTesting public static boolean setMemoizeResults(boolean memoizeResults)
      For unit tests, provide a method to turn memoization on or off.
      Parameters:
      memoizeResults - should results be memoized?
      Returns:
      the prior value

    • memoizeResult

      public <R> R memoizeResult(MemoizedOperationKey memoKey, Supplier<R> operation)
      Saves a weak reference to the result of the given operation.
      Parameters:
      memoKey - a complete description of the operation, if null no memoization is performed
      operation - a supplier for the result
      Returns:
      either the cached or newly generated result

    • getResult

      public <T extends DynamicNode & NotificationStepReceiver> T getResult(QueryTable.Operation<T> operation)

    • checkInitiateOperation

      public static void checkInitiateOperation(@NotNull @NotNull Table table)

    • checkInitiateBinaryOperation

      public static void checkInitiateBinaryOperation(@NotNull @NotNull Table first, @NotNull @NotNull Table second)

    • apply

      public <R> R apply(@NotNull @NotNull Function<Table,R> function)
      Description copied from interface: Table
      Applies a function to this table.

      This is useful if you have a reference to a table or a proxy and want to run a series of operations against the table without each individual operation resulting in an RMI.

      Type Parameters:
      R - the return type of function
      Parameters:
      function - the function to run, its single argument will be this table
      Returns:
      the return value of function

    • wouldMatch

      public Table wouldMatch(WouldMatchPair... matchers)
      Description copied from interface: Table
      A table operation that applies the supplied predicate to each row in the table and produces columns containing the pass/fail result of the predicate application. This is similar to TableOperations.where(String...) except that instead of selecting only rows that meet the criteria, new columns are added with the result of the comparison.
      Returns:
      a table with new columns containing the filter result for each row.

    • disableParallelWhereForThread

      public static SafeCloseable disableParallelWhereForThread()