consumeToPartitionedTable

The consumeToPartitionedTable method reads a Kafka stream into an in-memory partitioned table.

Syntax

Parameters

ParameterTypeDescription
kafkaPropertiesProperties

Configuration for the associated Kafka consumer and the resulting table. Once the table-specific properties are stripped, the remaining one is used to call the constructor of org.apache.kafka.clients.consumer.KafkaConsumer; pass any KafkaConsumer-specific desired configuration here.

topicstr

The Kafka topic name.

partitionFilterIntPredicate

A predicate returning true for the partitions to consume. The convenience constant ALL_PARTITIONS is defined to facilitate requesting all partitions.

partitionToInitialOffsetIntToLongFunction

A function specifying the desired initial offset for each partition consumed.

keySpecKeyOrValueSpec

Conversion specification for Kafka record keys.

valueSpecKeyOrValueSpec

Conversion specification for Kafka record values.

tableTypeTableType

A TableType enum. The default is TableType.blink().

  • TableType.append() - Create an append-only table.
  • TableType.blink() - Create a blink table (default).
  • TableType.ring(N) - Create a ring table of size N.

Returns

An in-memory partitioned table.

Examples

In the following example, consumeToPartitionedTable is used to read the Kafka topic orders into a Deephaven table. KafkaTools.Consume.FROM_PROPERTIES allows the key and value column types to be inferred by the properties passed in.

  • The host and port for the Kafka server to use to bootstrap are specified by kafkaProps.
    • The value redpanda:9092 corresponds to the current setup for development testing with Docker images (which uses an instance of redpanda).
  • The topic name is orders.