Package io.deephaven.kafka.ingest
Class KafkaStreamPublisher
- All Implemented Interfaces:
LogOutputAppendable
,LivenessManager
,LivenessNode
,LivenessReferent
,ConsumerRecordToStreamPublisherAdapter
,StreamPublisher
public class KafkaStreamPublisher
extends StreamPublisherBase
implements ConsumerRecordToStreamPublisherAdapter
An adapter that maps keys and values, possibly each with multiple fields, to single Deephaven columns. Each Kafka
record produces one Deephaven row.
-
Nested Class Summary
-
Field Summary
Fields inherited from class io.deephaven.kafka.StreamPublisherBase
chunks, consumer
-
Method Summary
Modifier and TypeMethodDescriptionlong
consumeRecords
(long receiveTime, @NotNull List<? extends org.apache.kafka.clients.consumer.ConsumerRecord<?, ?>> records) Consume a List of Kafka records, producing zero or more rows in the output.make
(@NotNull KafkaStreamPublisher.Parameters parameters, @NotNull Runnable shutdownCallback) void
propagateFailure
(@NotNull Throwable cause) Propagate a failure from the Kafka consumer to this StreamPublisher.void
shutdown()
Shutdown this StreamPublisher.Methods inherited from class io.deephaven.kafka.StreamPublisherBase
flush, getChunksToFill, register
Methods inherited from class io.deephaven.engine.liveness.ReferenceCountedLivenessNode
getWeakReference, initializeTransientFieldsForLiveness, onReferenceCountAtZero, tryManage, tryUnmanage, tryUnmanage
Methods inherited from class io.deephaven.engine.liveness.ReferenceCountedLivenessReferent
destroy, dropReference, tryRetainReference
Methods inherited from class io.deephaven.util.referencecounting.ReferenceCounted
append, decrementReferenceCount, forceReferenceCountToZero, getReferenceCountDebug, incrementReferenceCount, resetReferenceCount, toString, tryDecrementReferenceCount, tryIncrementReferenceCount
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
Methods inherited from interface io.deephaven.engine.liveness.LivenessManager
manage, unmanage, unmanage
Methods inherited from interface io.deephaven.engine.liveness.LivenessReferent
dropReference, getReferentDescription, retainReference, tryRetainReference
Methods inherited from interface io.deephaven.stream.StreamPublisher
flush, register
-
Field Details
-
NULL_COLUMN_INDEX
public static final int NULL_COLUMN_INDEX- See Also:
-
-
Method Details
-
make
public static ConsumerRecordToStreamPublisherAdapter make(@NotNull @NotNull KafkaStreamPublisher.Parameters parameters, @NotNull @NotNull Runnable shutdownCallback) -
propagateFailure
Description copied from interface:ConsumerRecordToStreamPublisherAdapter
Propagate a failure from the Kafka consumer to this StreamPublisher.- Specified by:
propagateFailure
in interfaceConsumerRecordToStreamPublisherAdapter
- Parameters:
cause
- The failure to propagate
-
consumeRecords
public long consumeRecords(long receiveTime, @NotNull @NotNull List<? extends org.apache.kafka.clients.consumer.ConsumerRecord<?, ?>> records) Description copied from interface:ConsumerRecordToStreamPublisherAdapter
Consume a List of Kafka records, producing zero or more rows in the output.- Specified by:
consumeRecords
in interfaceConsumerRecordToStreamPublisherAdapter
- Parameters:
receiveTime
- the time, in nanoseconds since the epoch, the records were received in this processrecords
- the records received fromKafkaConsumer.poll(Duration)
.- Returns:
- the number of bytes processed
-
shutdown
public void shutdown()Description copied from interface:StreamPublisher
Shutdown this StreamPublisher. Implementations should stop publishing new data and release any related resources as soon as practicable.- Specified by:
shutdown
in interfaceStreamPublisher
-