Package io.deephaven.engine.table.impl.sources

Class SparseArrayColumnSource<T>

java.lang.Object

io.deephaven.engine.table.impl.AbstractColumnSource<T>

io.deephaven.engine.table.impl.sources.SparseArrayColumnSource<T>

All Implemented Interfaces:

RowSetShiftCallback, ChunkSink<Values>, ChunkSource<Values>, ChunkSource.WithPrev<Values>, ColumnSource<T>, ElementSource<T>, FillContextMaker, GetContextMaker, DefaultChunkSource<Values>, DefaultChunkSource.WithPrev<Values>, FillUnordered<Values>, InMemoryColumnSource, PossiblyImmutableColumnSource, Releasable, TupleExporter<T>, TupleSource<T>, WritableColumnSource<T>, WritableSourceWithPrepareForParallelPopulation

Direct Known Subclasses:

BooleanSparseArraySource, ByteSparseArraySource, CharacterSparseArraySource, DoubleSparseArraySource, FloatSparseArraySource, IntegerSparseArraySource, LongSparseArraySource, ObjectSparseArraySource, ShortSparseArraySource

public abstract class SparseArrayColumnSource<T>
extends AbstractColumnSource<T>
implements FillUnordered<Values>, WritableColumnSource<T>, InMemoryColumnSource, PossiblyImmutableColumnSource, WritableSourceWithPrepareForParallelPopulation, RowSetShiftCallback

A column source backed by arrays that may not be filled in all blocks.

To store the blocks, we use a multi-level page table like structure. Each entry that exists is complete, i.e. we never reallocate partial blocks, we always allocate the complete block. The row key is divided as follows:

Description	Size	Bits
Block 0	19	62-44
Block 1	18	43-26
Block 2	18	25-8
Index Within Block	8	7-0

Bit 63, the sign bit, is used to indicate null (that is, all negative numbers are defined to be null)

Parallel structures are used for previous values and prevInUse. We recycle all levels of the previous blocks, so that the previous structure takes up memory only while it is in use.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected static class	SparseArrayColumnSource.FillByContext<UArray>

Nested classes/interfaces inherited from class io.deephaven.engine.table.impl.AbstractColumnSource

AbstractColumnSource.DefaultedImmutable<DATA_TYPE>, AbstractColumnSource.DefaultedMutable<DATA_TYPE>

Nested classes/interfaces inherited from interface io.deephaven.engine.table.ChunkSink

ChunkSink.FillFromContext

Nested classes/interfaces inherited from interface io.deephaven.engine.table.ChunkSource

ChunkSource.FillContext, ChunkSource.GetContext, ChunkSource.WithPrev<ATTR extends Any>

Nested classes/interfaces inherited from interface io.deephaven.engine.table.impl.DefaultChunkSource

DefaultChunkSource.SupportsContiguousGet<ATTR extends Any>, DefaultChunkSource.WithPrev<ATTR extends Any>

Nested classes/interfaces inherited from interface io.deephaven.engine.table.TupleExporter

TupleExporter.ExportElementFunction<TUPLE_TYPE>

Nested classes/interfaces inherited from interface io.deephaven.engine.table.WritableColumnSource

WritableColumnSource.ByteFiller, WritableColumnSource.CharFiller, WritableColumnSource.DoubleFiller, WritableColumnSource.FloatFiller, WritableColumnSource.IntFiller, WritableColumnSource.LongFiller, WritableColumnSource.ObjectFiller, WritableColumnSource.ShortFiller, WritableColumnSource.SinkFiller

Field Summary

Fields inherited from class io.deephaven.engine.table.impl.AbstractColumnSource

componentType, type, updateGraph, USE_RANGES_AVERAGE_RUN_LENGTH

Fields inherited from interface io.deephaven.engine.table.ChunkSink

DEFAULT_FILL_FROM_INSTANCE

Fields inherited from interface io.deephaven.engine.table.ChunkSource

DEFAULT_FILL_INSTANCE, ZERO_LENGTH_CHUNK_SOURCE_ARRAY

Fields inherited from interface io.deephaven.engine.table.ChunkSource.WithPrev

ZERO_LENGTH_CHUNK_SOURCE_WITH_PREV_ARRAY

Fields inherited from interface io.deephaven.engine.table.ColumnSource

ZERO_LENGTH_COLUMN_SOURCE_ARRAY

Fields inherited from interface io.deephaven.engine.table.impl.sources.InMemoryColumnSource

TWO_DIMENSIONAL_COLUMN_SOURCE_THRESHOLD

Method Summary

Modifier and Type	Method	Description
void	fillChunk(@NotNull ChunkSource.FillContext context, @NotNull WritableChunk<? super Values> dest, @NotNull RowSequence rowSequence)	Populates the given destination chunk with data corresponding to the keys from the given RowSequence.
void	fillChunkUnordered(@NotNull ChunkSource.FillContext context, @NotNull WritableChunk<? super Values> dest, @NotNull LongChunk<? extends RowKeys> keys)	Populates a contiguous portion of the given destination chunk with data corresponding to the keys from the given LongChunk.
void	fillFromChunk(@NotNull ChunkSink.FillFromContext context, @NotNull Chunk<? extends Values> src, @NotNull RowSequence rowSequence)	Our default, inefficient, implementation.
void	fillPrevChunkUnordered(@NotNull ChunkSource.FillContext context, @NotNull WritableChunk<? super Values> dest, @NotNull LongChunk<? extends RowKeys> keys)	Populates a contiguous portion of the given destination chunk with prev data corresponding to the keys from the given LongChunk.
static WritableColumnSource<Instant>	getInstantMemoryColumnSource(long[] data)
int	getPreferredChunkSize()	Using a preferred chunk size of BLOCK_SIZE gives us the opportunity to directly return chunks from our data structure rather than copying data.
static SparseArrayColumnSource<Byte>	getSparseMemoryColumnSource(byte[] data)
static SparseArrayColumnSource<Character>	getSparseMemoryColumnSource(char[] data)
static SparseArrayColumnSource<Double>	getSparseMemoryColumnSource(double[] data)
static SparseArrayColumnSource<Float>	getSparseMemoryColumnSource(float[] data)
static SparseArrayColumnSource<Integer>	getSparseMemoryColumnSource(int[] data)
static SparseArrayColumnSource<Long>	getSparseMemoryColumnSource(long[] data)
static <T> WritableColumnSource<T>	getSparseMemoryColumnSource(long size, Class<T> type)
static <T> WritableColumnSource<T>	getSparseMemoryColumnSource(long size, Class<T> type, @Nullable Class<?> componentType)
static SparseArrayColumnSource<Short>	getSparseMemoryColumnSource(short[] data)
static <T> WritableColumnSource<T>	getSparseMemoryColumnSource(Class<T> type)
static <T> WritableColumnSource<T>	getSparseMemoryColumnSource(Class<T> type, Class<?> componentType)
static ColumnSource<?>	getSparseMemoryColumnSource(Object dataArray)
static <T> WritableColumnSource<T>	getSparseMemoryColumnSource(Collection<T> data, Class<T> type)
boolean	isImmutable()	Determine if this column source is immutable, meaning that the values at a given row key never change.
ChunkSink.FillFromContext	makeFillFromContext(int chunkCapacity)	Provide a default, empty ChunkSink.FillFromContext for use with our default WritableColumnSource.fillFromChunk(io.deephaven.engine.table.ChunkSink.FillFromContext, io.deephaven.chunk.Chunk<? extends io.deephaven.chunk.attributes.Values>, io.deephaven.engine.rowset.RowSequence).
boolean	providesFillUnordered()	Returns true if this column source can efficiently provide an unordered fill.
void	remove(RowSet toRemove)
void	set(long key, byte value)
void	set(long key, char value)
void	set(long key, double value)
void	set(long key, float value)
void	set(long key, int value)
void	set(long key, long value)
void	set(long key, short value)
void	setImmutable()	Set this column source as having an immutable result.
void	setNull(RowSequence rowSequence)

Methods inherited from class io.deephaven.engine.table.impl.AbstractColumnSource

allowsReinterpret, defaultFillChunk, defaultFillPrevChunk, doReinterpret, fillPrevChunk, getComponentType, getPrevSource, getType, match, reinterpret

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.deephaven.engine.table.ChunkSource.WithPrev

fillPrevChunk

Methods inherited from interface io.deephaven.engine.table.ColumnSource

allowsReinterpret, cast, cast, createPreviousTuple, createTuple, createTupleFromValues, exportAllTo, exportElement, exportElement, getChunkType, getComponentType, getPrevSource, getType, isStateless, match, reinterpret, releaseCachedResources, startTrackingPrevValues, tupleLength

Methods inherited from interface io.deephaven.engine.table.impl.DefaultChunkSource

getChunk, getChunk, getChunkByFilling, makeFillContext, makeGetContext

Methods inherited from interface io.deephaven.engine.table.impl.DefaultChunkSource.WithPrev

getPrevChunk, getPrevChunk, getPrevChunkByFilling

Methods inherited from interface io.deephaven.engine.table.ElementSource

get, getBoolean, getByte, getChar, getDouble, getFloat, getInt, getLong, getPrev, getPrevBoolean, getPrevByte, getPrevChar, getPrevDouble, getPrevFloat, getPrevInt, getPrevLong, getPrevShort, getShort

Methods inherited from interface io.deephaven.engine.table.FillContextMaker

makeFillContext

Methods inherited from interface io.deephaven.engine.table.GetContextMaker

makeGetContext

Methods inherited from interface io.deephaven.engine.table.impl.sources.InMemoryColumnSource

isInMemory

Methods inherited from interface io.deephaven.engine.rowset.RowSetShiftCallback

shift

Methods inherited from interface io.deephaven.engine.table.TupleExporter

exportAllReinterpretedTo, exportAllReinterpretedTo, exportAllTo, exportElementReinterpreted

Methods inherited from interface io.deephaven.engine.table.TupleSource

createTupleFromReinterpretedValues

Methods inherited from interface io.deephaven.engine.table.WritableColumnSource

ensureCapacity, ensureCapacity, fillFromChunkUnordered, set, setNull

Methods inherited from interface io.deephaven.engine.table.WritableSourceWithPrepareForParallelPopulation

prepareForParallelPopulation

Method Details

set

public void set(long key,
 byte value)

Specified by:

set in interface WritableColumnSource<T>

set

public void set(long key,
 char value)

Specified by:

set in interface WritableColumnSource<T>

set

public void set(long key,
 double value)

Specified by:

set in interface WritableColumnSource<T>

set

public void set(long key,
 float value)

Specified by:

set in interface WritableColumnSource<T>

set

public void set(long key,
 int value)

Specified by:

set in interface WritableColumnSource<T>

set

public void set(long key,
 long value)

Specified by:

set in interface WritableColumnSource<T>

set

public void set(long key,
 short value)

Specified by:

set in interface WritableColumnSource<T>

remove

public void remove(RowSet toRemove)

getSparseMemoryColumnSource

public static <T> WritableColumnSource<T> getSparseMemoryColumnSource(Collection<T> data,
 Class<T> type)

getSparseMemoryColumnSource

public static SparseArrayColumnSource<Byte> getSparseMemoryColumnSource(byte[] data)

getSparseMemoryColumnSource

public static SparseArrayColumnSource<Character> getSparseMemoryColumnSource(char[] data)

getSparseMemoryColumnSource

public static SparseArrayColumnSource<Double> getSparseMemoryColumnSource(double[] data)

getSparseMemoryColumnSource

public static SparseArrayColumnSource<Float> getSparseMemoryColumnSource(float[] data)

getSparseMemoryColumnSource

public static SparseArrayColumnSource<Integer> getSparseMemoryColumnSource(int[] data)

getSparseMemoryColumnSource

public static SparseArrayColumnSource<Long> getSparseMemoryColumnSource(long[] data)

getInstantMemoryColumnSource

public static WritableColumnSource<Instant> getInstantMemoryColumnSource(long[] data)

getSparseMemoryColumnSource

public static SparseArrayColumnSource<Short> getSparseMemoryColumnSource(short[] data)

getSparseMemoryColumnSource

public static <T> WritableColumnSource<T> getSparseMemoryColumnSource(Class<T> type)

getSparseMemoryColumnSource

public static <T> WritableColumnSource<T> getSparseMemoryColumnSource(Class<T> type,
 Class<?> componentType)

getSparseMemoryColumnSource

public static <T> WritableColumnSource<T> getSparseMemoryColumnSource(long size,
 Class<T> type)

getSparseMemoryColumnSource

public static <T> WritableColumnSource<T> getSparseMemoryColumnSource(long size,
 Class<T> type,
 @Nullable
 @Nullable Class<?> componentType)

getSparseMemoryColumnSource

public static ColumnSource<?> getSparseMemoryColumnSource(Object dataArray)

getPreferredChunkSize

public int getPreferredChunkSize()

Using a preferred chunk size of BLOCK_SIZE gives us the opportunity to directly return chunks from our data structure rather than copying data.

fillChunk

public void fillChunk(@NotNull
 @NotNull ChunkSource.FillContext context,
 @NotNull
 @NotNull WritableChunk<? super Values> dest,
 @NotNull
 @NotNull RowSequence rowSequence)

Description copied from interface: ChunkSource

Populates the given destination chunk with data corresponding to the keys from the given RowSequence.

Specified by:

fillChunk in interface ChunkSource<T>

Overrides:

fillChunk in class AbstractColumnSource<T>

Parameters:

context - A context containing all mutable/state related data used in retrieving the Chunk.

dest - The chunk to be populated according to rowSequence. No assumptions shall be made about the size of the chunk shall be made. The chunk will be populated from position [0,rowSequence.size()).

rowSequence - An RowSequence representing the keys to be fetched

setNull

public void setNull(RowSequence rowSequence)

Specified by:

setNull in interface WritableColumnSource<T>

fillChunkUnordered

public void fillChunkUnordered(@NotNull
 @NotNull ChunkSource.FillContext context,
 @NotNull
 @NotNull WritableChunk<? super Values> dest,
 @NotNull
 @NotNull LongChunk<? extends RowKeys> keys)

Description copied from interface: FillUnordered

Populates a contiguous portion of the given destination chunk with data corresponding to the keys from the given LongChunk.

It behaves as if the following code were executed:

 destination.setSize(keys.size());
 for (int ii = 0; ii < keys.size(); ++ii) {
     destination.set(ii, get(keys.get(ii)));
 }
 

Specified by:

fillChunkUnordered in interface FillUnordered<T>

Parameters:

context - A context containing all mutable/state related data used in retrieving the Chunk.

dest - The chunk to be populated according to keys

keys - A chunk of individual, not assumed to be ordered keys to be fetched

fillPrevChunkUnordered

public void fillPrevChunkUnordered(@NotNull
 @NotNull ChunkSource.FillContext context,
 @NotNull
 @NotNull WritableChunk<? super Values> dest,
 @NotNull
 @NotNull LongChunk<? extends RowKeys> keys)

Description copied from interface: FillUnordered

Populates a contiguous portion of the given destination chunk with prev data corresponding to the keys from the given LongChunk.

It behaves as if the following code were executed:

 destination.setSize(keys.size());
 for (int ii = 0; ii < keys.size(); ++ii) {
     destination.set(ii, getPrev(keys.get(ii)));
 }
 

Specified by:

fillPrevChunkUnordered in interface FillUnordered<T>

Parameters:

context - A context containing all mutable/state related data used in retrieving the Chunk.

dest - The chunk to be populated according to keys

keys - A chunk of individual, not assumed to be ordered keys to be fetched

makeFillFromContext

public ChunkSink.FillFromContext makeFillFromContext(int chunkCapacity)

Description copied from interface: WritableColumnSource

Provide a default, empty ChunkSink.FillFromContext for use with our default WritableColumnSource.fillFromChunk(io.deephaven.engine.table.ChunkSink.FillFromContext, io.deephaven.chunk.Chunk<? extends io.deephaven.chunk.attributes.Values>, io.deephaven.engine.rowset.RowSequence).

Specified by:

makeFillFromContext in interface ChunkSink<T>

Specified by:

makeFillFromContext in interface WritableColumnSource<T>

fillFromChunk

public void fillFromChunk(@NotNull
 @NotNull ChunkSink.FillFromContext context,
 @NotNull
 @NotNull Chunk<? extends Values> src,
 @NotNull
 @NotNull RowSequence rowSequence)

Description copied from interface: WritableColumnSource

Our default, inefficient, implementation. Inheritors who care should provide a better implementation.

Specified by:

fillFromChunk in interface ChunkSink<T>

Specified by:

fillFromChunk in interface WritableColumnSource<T>

Parameters:

context - A context containing all mutable/state related data used in writing the Chunk.

src - The source of the data rowSequence

rowSequence - An RowSequence representing the keys to be written

isImmutable

public boolean isImmutable()

Description copied from interface: ColumnSource

Determine if this column source is immutable, meaning that the values at a given row key never change.

Specified by:

isImmutable in interface ColumnSource<T>

Returns:

true if the values at a given row key of the column source never change, false otherwise

setImmutable

public void setImmutable()

Description copied from interface: PossiblyImmutableColumnSource

Set this column source as having an immutable result.

Specified by:

setImmutable in interface PossiblyImmutableColumnSource

providesFillUnordered

public boolean providesFillUnordered()

Description copied from interface: FillUnordered

Returns true if this column source can efficiently provide an unordered fill. If this method returns false, then fillChunkUnordered and fillPrevChunkUnordered may throw an UnsupportedOperationException.

Specified by:

providesFillUnordered in interface FillUnordered<T>

Returns:

if this column source can provide an unordered fill