Kafka cheat sheet
This cheat sheet shows common ways to produce Deephaven tables to Kafka and consume Kafka topics into live Deephaven tables that update as new records arrive. The main methods are:
The examples assume the following setup:
- A Kafka broker (the server that stores topics) at
redpanda:9092. See Kafka basic terms. - A schema registry at
http://redpanda:8081, for the Avro example only. - The
testTopictopic, which Simple produce writes and Simple consume and Start from the beginning of a topic read. - The
time-topic_grouptopic, which Produce JSON withlast_byon key columns writes. - A
share.pricetopic with string keys and double values, supplied separately, for Define the key and value columns and Ignore the partition and key. - A
share.price.avrotopic of Avro records whose schema is registered asshare.price.record, supplied separately, for Consume Avro. - An
orderstopic of JSON records with the fieldsjsymbol,jside,jprice, andjqty, supplied separately, for Consume JSON with a field mapping.
To run a broker alongside Deephaven, see Connect to a Kafka stream.
Produce to Kafka
Simple produce
Publish column X as the message key with a simple key spec and ignore the message value.
Produce JSON with last_by on key columns
Publish column X as a JSON key and columns X and Y as a JSON value. Setting last_by_key_columns to True runs a last_by on the key column X and publishes from the result, so each message carries the latest row for its key. Because source_group keeps ticking, the producer still publishes a message each time a key's latest row changes.
Consume from Kafka
By default, consume reads all partitions and returns a blink table. It does not seek, so Kafka picks the starting offset. With no group.id set, as in these examples, it starts at the end of each partition, so the table receives only records produced after the call. Start from the beginning of a topic shows how to read existing records instead. The examples that pass table_type=TableType.append() return an append-only table instead.
consume_to_partitioned_table takes the same arguments as consume and returns a partitioned table with one constituent table per Kafka partition. See its reference page for an example.
Simple consume
Without a key or value spec, the key and value column types come from the deephaven.key.column.type and deephaven.value.column.type properties. The properties match what Simple produce writes: an int key and no value. The KafkaKey column is typed int, and the KafkaValue column holds only nulls.
Start from the beginning of a topic
Pass ALL_PARTITIONS_SEEK_TO_BEGINNING as offsets to read the records already in the topic, starting at the beginning of each partition, and then the new records as they arrive.
Define the key and value columns
Read the key into a Symbol string column and the value into a Price double column.
Ignore the partition and key
Setting deephaven.partition.column.name to an empty string leaves out the KafkaPartition column, which consume adds by default to hold each record's partition number. KeyValueSpec.IGNORE leaves out the key.
Consume JSON with a field mapping
Parse each JSON value into four typed columns. The mapping argument maps JSON field names, such as jsymbol, to column names.
Consume Avro
Parse each value with version 1 of the share.price.record Avro schema from the schema registry.