Kafka cheat sheet
This cheat sheet shows common ways to produce Deephaven tables to Kafka and consume Kafka topics into live Deephaven tables that update as new records arrive. The main methods are:
The examples assume the following setup:
- A Kafka broker (the server that stores topics) at
redpanda:9092. See Kafka basic terms. - A schema registry at
http://redpanda:8081, for the Avro example only. - The
testTopictopic, which Simple produce writes and Simple consume and Start from the beginning of a topic read. - The
time-topic_grouptopic, which Produce JSON withlastByon key columns writes. - A
share.pricetopic with string keys and double values, supplied separately, for Define the key and value columns and Ignore the partition and key. - A
share.price.avrotopic of Avro records whose schema is registered asshare.price.record, supplied separately, for Consume Avro. - An
orderstopic of JSON records with the fieldsjsymbol,jside,jprice, andjqty, supplied separately, for Consume JSON with a field mapping.
To run a broker alongside Deephaven, see Connect to a Kafka stream.
Produce to Kafka
Simple produce
Publish column X as the message key with a simple key spec and ignore the message value. The last argument, lastByKeyColumns, is false, so the producer publishes every row.
Produce JSON with lastBy on key columns
Publish column X as a JSON key and columns X and Y as a JSON value. Setting the last argument, lastByKeyColumns, to true runs a lastBy on the key column X and publishes from the result, so each message carries the latest row for its key. Because sourceGroup keeps ticking, the producer still publishes a message each time a key's latest row changes.
Consume from Kafka
consumeToTable has no optional arguments, so every call passes all seven: the properties, the topic, the partitions, the starting offsets, the key spec, the value spec, and the table type. The consume examples on this page pass these values for three of them:
- Partitions:
KafkaTools.ALL_PARTITIONSreads all partitions. - Starting offsets:
KafkaTools.ALL_PARTITIONS_DONT_SEEKstarts each partition without seeking, so Kafka picks the starting offset. With nogroup.idset, as in these examples, it starts at the end of each partition, so the table receives only records produced after the call. Start from the beginning of a topic shows how to read existing records instead. - Table type:
KafkaTools.TableType.blinkreturns a blink table, andKafkaTools.TableType.appendreturns an append-only table.KafkaTools.TableType.ring(capacity)returns a ring table.
consumeToPartitionedTable takes the same arguments as consumeToTable and returns a partitioned table with one constituent table per Kafka partition. See its reference page for an example.
Simple consume
This example passes KafkaTools.FROM_PROPERTIES as the key and value specs, so the key and value column types come from the deephaven.key.column.type and deephaven.value.column.type properties. The properties match what Simple produce writes: an int key and no value. The KafkaKey column is typed int, and the KafkaValue column holds only nulls.
Start from the beginning of a topic
Pass KafkaTools.ALL_PARTITIONS_SEEK_TO_BEGINNING as the starting offsets to read the records already in the topic, starting at the beginning of each partition, and then the new records as they arrive.
Define the key and value columns
Read the key into a Symbol string column and the value into a Price double column.
Ignore the partition and key
Setting deephaven.partition.column.name to an empty string leaves out the KafkaPartition column, which consumeToTable adds by default to hold each record's partition number. KafkaTools.Consume.IGNORE leaves out the key.
Consume JSON with a field mapping
Parse each JSON value into four typed columns. The mapping map, passed as the second argument of jsonSpec, maps JSON field names, such as jsymbol, to column names.
Consume Avro
Parse each value with version 1 of the share.price.record Avro schema from the schema registry.