consumeToPartitionedTable

The consumeToPartitionedTable method reads a Kafka stream into an in-memory partitioned table.

Syntax

Parameters

ParameterTypeDescription
kafkaPropertiesProperties

Configuration for the associated Kafka consumer and the resulting table. Once the table-specific properties are stripped, the remaining one is used to call the constructor of org.apache.kafka.clients.consumer.KafkaConsumer; pass any KafkaConsumer-specific desired configuration here.

topicstr

The Kafka topic name.

partitionFilterIntPredicate

A predicate returning true for the partitions to consume. The convenience constant ALL_PARTITIONS is defined to facilitate requesting all partitions.

partitionToInitialOffsetIntToLongFunction

A function specifying the desired initial offset for each partition consumed.

keySpecKeyOrValueSpec

Conversion specification for Kafka record keys.

valueSpecKeyOrValueSpec

Conversion specification for Kafka record values.

tableTypeTableType

A TableType enum. The default is TableType.blink().

  • TableType.append() - Create an append-only table.
  • TableType.blink() - Create a blink table (default).
  • TableType.ring(N) - Create a ring table of size N.

Returns

An in-memory partitioned table.

Examples

In the following example, consumeToPartitionedTable is used to read the Kafka topic orders into a Deephaven table. The example ignores the Kafka key with KafkaTools.Consume.IGNORE and reads the value as JSON with KafkaTools.Consume.jsonSpec, which maps the JSON fields jsymbol, jside, jprice, and jqty to the Symbol, Side, Price, and Qty columns.

  • The host and port for the Kafka server to use to bootstrap are specified by kafkaProps.
    • The value redpanda:9092 corresponds to the current setup for development testing with Docker images (which uses an instance of redpanda).
  • The topic name is orders.