Selection method properties

The Deephaven query engine select and update operations bring table data into memory. The default mode of operation preserves the address space of the input table. Depending on your data patterns, this address space preservation may use significant amounts of additional memory. To better understand how a query's memory is used, execute the following Groovy snippet to identify how much memory is wasted per table.

import gnu.trove.set.hash.TLongHashSet
import io.deephaven.engine.table.impl.sources.SparseArrayColumnSource
import io.deephaven.shadow.enterprise.com.illumon.iris.db.util.file.DataBufferCalculator

accumulateBlocks = { long rowKey, TLongHashSet longSet -> longSet.add(rowKey >> 10) }

checkSparsity = { bindingVarEntry, list ->
    def varName = bindingVarEntry.getKey()
    def varValue = bindingVarEntry.getValue()
    if (varValue instanceof Table) {
        def s = varValue.getColumnSources().stream();
        s = s.filter({cs -> cs instanceof SparseArrayColumnSource})
        s = s.map({cs -> cs.getType()})
        s = s.mapToInt({cst -> int result = DataBufferCalculator.getSimpleDataRecordSizeInBytes(cst); return result == 0 ? 8 : result; })
        int colByteSizeSum = s.sum()
        if (colByteSizeSum > 0) {
            def longSet = new TLongHashSet()
            long idealBlocks = ((varValue.getRowSet().size() + 1023) / 1024)
            varValue.getRowSet().forAllRowKeys({rowKey -> accumulateBlocks(rowKey, longSet)})
            long actualBlocks = longSet.size()
            long waste = (colByteSizeSum * 1024 * (actualBlocks - idealBlocks))
            println varName + ": ideal blocks=" + idealBlocks + ", actual blocks=" + actualBlocks + ", efficiency=" + (actualBlocks / idealBlocks) + ", waste=" + waste
            list.add([varName, waste])
        }
    }
}
bindingVars = getBinding().getVariables()
resultList = []
bindingVars.entrySet().forEach({e -> checkSparsity(e, resultList)})
resultList.sort({e1, e2 -> -1 * Long.compare(e1.get(1), e2.get(1))})
println resultList

import gnu.trove.set.hash.TLongHashSet
import com.illumon.iris.db.v2.sources.SparseArrayColumnSource
import com.illumon.iris.db.util.file.DataBufferCalculator
accumulateBlocks = { long idx, TLongHashSet longSet -> longSet.add(idx >> 10) }
checkSparsity = { bindingVarEntry, list ->
    def varName = bindingVarEntry.getKey()
    def varValue = bindingVarEntry.getValue()
    if (varValue instanceof Table) {
        def s = varValue.getColumnSources().stream();
        s = s.filter({cs -> cs instanceof SparseArrayColumnSource})
        s = s.map({cs -> cs.getType()})
        s = s.mapToInt({cst -> int result = DataBufferCalculator.getSimpleDataRecordSizeInBytes(cst); return result == 0 ? 8 : result; })
        int colByteSizeSum = s.sum()
        if (colByteSizeSum > 0) {
            def longSet = new TLongHashSet()
            long idealBlocks = ((varValue.getIndex().size() + 1023) / 1024)
            varValue.getIndex().forAllLongs({idx -> accumulateBlocks(idx, longSet)})
            long actualBlocks = longSet.size()
            long waste = (colByteSizeSum * 1024 * (actualBlocks - idealBlocks))
            println varName + ": ideal blocks=" + idealBlocks + ", actual blocks=" + actualBlocks + ", efficiency=" + (actualBlocks / idealBlocks) + ", waste=" + waste
            list.add([varName, waste])
        }
    }
}
bindingVars = getBinding().getVariables()
resultList = []
bindingVars.entrySet().forEach({e -> checkSparsity(e, resultList)})
resultList.sort({e1, e2 -> -1 * Long.compare(e1.get(1), e2.get(1))})
println resultList

For example, we can create three tables with different address space patterns:

First, a raw ProcessEventLog (which has a dense address space made up of one region per partition).
Second, a table filtered on Timestamp (which will have a contiguous subset of the addresses from each partition).
Third, a table using worker name (which filters rows throughout the table without any inherent adjacency).
If we called these tables pelAllSelect, pelTimestampSel and pelWorkerSel, the snippet produces output similar to the following:

pelAllSelect: ideal blocks=518, actual blocks=521, efficiency=1.0057915058, waste=196608
pelTimestampSel: ideal blocks=20, actual blocks=27, efficiency=1.35, waste=458752
pelWorkerSel: ideal blocks=19, actual blocks=59, efficiency=3.1052631579, waste=2621440
[[pelWorkerSel, 2621440], [pelTimestampSel, 458752], [pelAllSelect, 196608]]

The efficiency is ideally 1.0, and we can see the pelAllSelect table very nearly reaches 1.0 as it has a dense address space. On the other hand, the pelTimestamp table uses 40% more space than ideal, and the pelWorkerSel table uses more than three times as much memory as optimal. In the worst case, a table that uses only one index key per 1024-entry block will require 1024 times as much memory as optimal.

If we were to flatten the pelWorkerSel table and then rerun the snippet, we see that a flat table has an ideal ratio of 1.0:

pelAllSelect: ideal blocks=518, actual blocks=521, efficiency=1.0057915058, waste=196608
pelTimestampSel: ideal blocks=20, actual blocks=27, efficiency=1.35, waste=458752
pelWorkerSel: ideal blocks=19, actual blocks=59, efficiency=3.1052631579, waste=2621440
pelWorkerFlat: ideal blocks=19, actual blocks=19, efficiency=1, waste=0
[[pelWorkerSel, 2621440], [pelTimestampSel, 458752], [pelAllSelect, 196608], [pelWorkerFlat, 0]]

Preserving the original address space allows data to be read more quickly, but in workloads where it results in poor memory utilization or performance, the following optional properties for select and update may be configured:

QueryTable.redirectSelect
QueryTable.redirectUpdate

Both are false by default. If set to true, then select and update create redirected column sources. These are more memory efficient, but potentially use more CPU as they require an internal hash table look-up to read or write values.

Selection method properties

Related documentation