Connect to a Kafka stream
Kafka is a distributed event streaming platform that lets you read, write, store, and process events, also called records.
Deephaven turns a Kafka topic into a live table that updates as records arrive, so you can filter, join, and aggregate a stream with the same table operations you use on static data.
Kafka record keys and values come in many formats, such as simple types like strings and numbers, JSON, Avro, or Protobuf. In this guide, we show you how to read each of these formats into Deephaven tables, and how to write a Deephaven table to Kafka.
See A Kafka introduction: basic terms for a detailed discussion of Kafka topics and supported formats. See the Apache Kafka documentation for full details on how to use Kafka.
Launch Kafka with Deephaven
Deephaven has an official Docker Compose file that runs the Deephaven images alongside Redpanda, one of many Kafka-compatible event streaming platforms that work with Deephaven. Redpanda's command-line tool lets you send records to a Kafka topic from the terminal.
Save the Docker Compose file locally as docker-compose.yml, and start it with docker compose up. The Deephaven IDE is then at http://localhost:10000/ide.
Key and value
Each Kafka record has a key and a value. Kafka writes each record to a partition at an offset, with a timestamp. For example, a list of Kafka records might have a stock ticker as the key and its price as the value.
The key and the value can each be nearly any sequence of bytes.
A key spec and a value spec tell Deephaven how to turn the key and the value into table columns. You pass them to consume as its key_spec and value_spec arguments. You create them with functions from the deephaven.stream.kafka.consumer module, which the examples on this page import as kc.
The key and the value can each be:
- read as a simple type
- decoded from JSON
- decoded from Avro
- decoded from Protobuf
- parsed by an object processor, such as a Jackson JSON provider
- named and typed from consumer properties, with
kc.KeyValueSpec.FROM_PROPERTIES - ignored, with
kc.KeyValueSpec.IGNORE
You can't ignore both the key and the value.
If you omit key_spec or value_spec, Deephaven uses kc.KeyValueSpec.FROM_PROPERTIES for it, which needs a column type from the consumer properties. See Name key and value columns with properties.
Table types
Deephaven Kafka tables can be append-only, blink, or ring. Set the type with the table_type argument of consume. Its value is a TableType. If you don't pass table_type, the table is a blink table.
- Append-only tables keep every row. The table and its memory use can grow without limit. To use this type, pass
kc.TableType.append()astable_type. - Blink tables keep only the rows from the current update cycle. Each new record appears as a row for one update cycle and then disappears. To set this type explicitly, pass
kc.TableType.blink()astable_type. - Ring tables keep only the last
Nrows. When the table grows beyondNrows, it discards the oldest rows untilNremain. To use this type, passkc.TableType.ring(N)astable_type.
To keep results after blink rows disappear, see the last_by example in Ring and blink tables.
Consume a Kafka stream
In this example, we consume a Kafka topic (test.topic) as a Deephaven table. You populate the Kafka topic by entering commands in the terminal.
For demonstration purposes, we use an append-only table and ignore the Kafka key.
consume creates a Deephaven table from a Kafka topic. Its arguments do the following:
{"bootstrap.servers": "redpanda:9092"}is a dictionary that describes how to connect to Kafka.bootstrap.serverslists the initial hosts that a Kafka client connects to. The valueredpanda:9092is the address of the Redpanda service from the Docker Compose file. To connect to your own Kafka cluster, replaceredpanda:9092with your brokers'host:portlist.table_typeiskc.TableType.append(), which creates an append-only table.key_speciskc.KeyValueSpec.IGNORE, which ignores the Kafka key.value_specusessimple_specto read each record's value into aStringcolumn namedCommand.
The result_append table now reads from every partition that the test.topic topic has when the table is created.
Input Kafka data for testing
To send data to the test.topic topic that the result_append table consumes, run the following command in a terminal:
The command waits for input from the terminal and sends each line you enter to the test.topic topic as a record. Press Ctrl + D when you're done to stop producing.
Each record you send appears in the result_append table without rerunning the query.
Ring and blink tables
The following example shows how to create ring and blink tables to read from the test.topic topic:
Run docker compose exec redpanda rpk topic produce test.topic again and enter a few more lines. The result_append table contains all the records, the result_ring table contains the last three, and the result_blink table shows each new row for only one update cycle.
Because rows disappear from the result_blink table after one update cycle, let's add a table that uses last_by to keep the last row added to the result_blink table.
Read the key and choose partitions and offsets
In this example, consume reads the Kafka topic share.price into an append-only table. Unlike the previous examples, it reads both the key and the value, and it sets the partitions and the starting offsets explicitly.
When reading a Kafka topic, you can choose which partitions to read and where reading starts in each one:
- Partitions: By default, Deephaven reads all partitions.
- Starting point: Reading can start at the beginning of each partition, at the end, or at a specific offset. Kafka calls moving the read position seeking. By default, Deephaven doesn't seek, so reading starts where Kafka's consumer settings say.
Let's walk through the arguments in this query.
partitionsis set toNone, which reads all partitions.Noneis also the default when you don't passpartitions.offsetsis set tokc.ALL_PARTITIONS_DONT_SEEK, which doesn't seek. Kafka then picks the starting point:- If you set the
group.idproperty and that consumer group has a committed offset, reading starts there. - Otherwise, Kafka's
auto.offset.resetproperty decides. Its default,latest, reads only new records. This example doesn't setgroup.id, so the table shows only records sent after you create it.
- If you set the
key_specusessimple_specto read each record's key into aStringcolumn namedSymbol. To drop theSymbolcolumn, passkc.KeyValueSpec.IGNOREaskey_specinstead, as in the first example.value_specusessimple_specto read each record's value into aStringcolumn namedPrice. The column is a string becauserpk, Redpanda's command-line tool, sends135.60as text. Withdht.double, Deephaven would decode the value with Kafka'sDoubleDeserializer, which expects 8 binary bytes, so text input fails.table_typeis set tokc.TableType.append(), which creates an append-only table.
To change which partitions the example reads or where it starts, change partitions and offsets.
To read specific partitions, pass them as a list of integers, such as partitions=[1, 3, 5].
To choose a different starting point, pass one of these values as offsets:
kc.ALL_PARTITIONS_SEEK_TO_ENDalways starts with new records only.kc.ALL_PARTITIONS_SEEK_TO_BEGINNINGstarts at the beginning of every partition.- A dictionary that maps partition numbers to offsets, such as
{0: 100, 1: 250}, starts at those offsets. Partitions that the dictionary doesn't list don't seek.
Note
Starting at the beginning of a partition reads only the records Kafka still retains. Kafka's retention settings delete older records once a topic passes a configured age or size, and log compaction can remove older records for a key, keeping at least the latest one.
Now let's add some entries to our Kafka stream.
Run the following command:
Enter as many key-value pairs as you want. Put a space between each key and its value, and put each pair on its own line:
Each pair you enter appears as a row in the result table.
Read Kafka topic in JSON format
The following two examples read a Kafka topic called orders in JSON format.
The first example uses json_spec:
Here, the value_spec argument uses json_spec, which parses each record's value as JSON.
The first argument to json_spec is a table definition: a dictionary that maps each column name in the result table to its Deephaven type (for example, dht.double).
The mapping keyword argument of json_spec is a dictionary that maps JSON field names to table column names. Each column name must appear in the table definition from the first argument.
In the example, the entry "price": "Price" reads the JSON field price into the Price column of the result table. For any column that mapping doesn't mention, or for every column when you omit mapping, Deephaven reads the JSON field with the same name as the column.
The second example uses object_processor_spec with a Jackson provider:
Here, object_processor_spec takes a Jackson provider built from a dictionary that maps each JSON field name to its type. The resulting columns take the JSON field names, so the example renames them with rename_columns.
To test either example, run the following command:
Then enter the following values:
Read Kafka topic in Avro format
A schema registry stores each Kafka event schema under a subject name and tracks the subject's versions, so that producers and consumers can share them. To register a schema, see Redpanda's schema registry documentation.
In this example, consume reads the Kafka topic share.price.avro in Avro format. This example assumes that a schema with the subject name share.price.avro.record is registered in the schema registry of the Redpanda instance from the Docker Compose file.
The configuration dictionary in this query also includes a schema.registry.url entry. It gives the URL of a schema registry that has a REST API compatible with Confluent's schema registry specification.
The value_spec argument uses avro_spec, which reads each record's value as Avro.
The first positional argument in the avro_spec call specifies the Avro schema to use. In this case, avro_spec gets the schema named share.price.avro.record from the schema registry. Alternatively, the first argument can be a JSON-encoded Avro schema definition string.
avro_spec also takes three optional keyword arguments:
schema_versionis the version of the named schema to get from the schema registry. The default,latest, gets the latest available version.mappingis a dictionary that maps Avro field paths to table column names. A field inside a nested record uses its dotted path, such asparent.child. Unlessmapped_onlyisTrue, Deephaven maps each Avro field thatmappingdoesn't name to a column named with its full field path. For a nested field, that column name is the dotted path, such asparent.child, not theparent__childname Deephaven uses when you don't passmapping. A dotted path is not a valid column name, so when you passmapping, map every nested field you keep to a valid column name, or setmapped_onlytoTrueto leave out the fields you don't map.mapped_onlyis a boolean that defaults toFalse. When it isTrueandmappingis set, Deephaven leaves Avro fields thatmappingdoesn't name out of the resulting table. To keep a field inside a nested record, also name the parent record inmapping, as in{"parent": "parent", "parent.child": "Child"}. Otherwise Deephaven leaves out the whole record. Withoutmapping,mapped_onlyhas no effect.
If you don't pass mapping, Deephaven maps each top-level Avro field to a column with the same name, and each field of a nested record to a column named with the field path joined by __, such as parent__child.
Read Kafka topic in Protobuf format
In this example, consume reads the Kafka topic share.price.proto in Protobuf format. Protobuf is Google's open-source, language-neutral format for serializing structured data.
This example assumes that a schema with the subject name share.price.proto.record is registered in the schema registry of the Redpanda instance from the Docker Compose file. To register a schema, see Redpanda's schema registry documentation.
The configuration dictionary in this query also includes a schema.registry.url entry. It gives the URL of a schema registry that has a REST API compatible with Confluent's schema registry specification.
The value_spec argument uses protobuf_spec, which reads each record's value as Protobuf.
protobuf_spec takes the following arguments:
schemais the schema subject name,share.price.proto.recordin the example. When set, Deephaven fetches the Protobuf message descriptor from the schema registry. Set either this ormessage_class, but not both.message_classis the fully qualified Java class name for the Protobuf message on the current classpath, for examplecom.example.MyMessageorcom.example.OuterClass$MyMessage. When this is set, Deephaven doesn't use the schema registry. Set either this orschema, but not both.schema_versionspecifies the schema version. The default,None, fetches the latest version. We recommend setting it so that the resulting table definition doesn't change across restarts.schema_message_nameis the fully qualified Protobuf message name, for examplecom.example.MyMessage. This message's descriptor is the basis for the resulting table's definition. The default,None, uses the first message descriptor in the Protobuf schema. We recommend setting it for the same reason.includeis a list of/-separated field paths to include. End a path with/*to also include every field path that starts with it. A field is included when any path in the list matches it. The default,None, includes all fields.include=["/foo/bar"]includes the fieldbarinsidefoo, along with its parents: the top-level message andfoo.include=["/foo/bar/*"]also includes every field nested under/foo/bar, such as/foo/bar/bazand/foo/bar/baz/zap.
protocolis the wire protocol of the payload, as akc.ProtobufProtocol. The default iskc.ProtobufProtocol.serdes()whenschemais set andkc.ProtobufProtocol.raw()whenmessage_classis set.kc.ProtobufProtocol.serdes()is the Kafka Protobuf serializer/deserializer (serdes) format, which puts a magic byte, a 4-byte schema ID, and message indexes before the Protobuf data.kc.ProtobufProtocol.raw()is the Protobuf binary encoding with no header.
Each resulting column is named with the field path joined by _, such as foo_bar for the field bar inside foo. If the parser produces more than one column for a field path, the second and later columns get the suffixes _2, _3, and so on, such as foo_bar_2.
Name key and value columns with properties
Deephaven can take a key or value column's name and type from consumer properties instead of from the spec:
kc.KeyValueSpec.FROM_PROPERTIEStakes both the column name and the type from properties. Omittingkey_specorvalue_specdoes the same.simple_specwithoutdata_type, as inkc.simple_spec("Symbol"), sets the column name and takes the type from properties.
With kc.KeyValueSpec.FROM_PROPERTIES, Deephaven reads the key or value into a single column. These properties in the configuration dictionary passed to consume name it:
deephaven.key.column.namenames the key column. The default isKafkaKey.deephaven.value.column.namenames the value column. The default isKafkaValue.
Deephaven chooses the column type in this order:
- The
data_typepassed tosimple_spec, if there is one. - The
deephaven.key.column.typeordeephaven.value.column.typeproperty. It acceptsshort,int,long,float,double,byte[], orString(also accepted asstring). - The Kafka deserializer set in the
key.deserializerorvalue.deserializerconsumer property. Deephaven recognizes the numeric, byte-array,UUID,ByteBuffer, andBytesdeserializers.
For a String column, set the type property or pass dht.string to simple_spec.
If none of these sets a type, consume raises an error that names the column.
Standard data fields
Besides the key and value, each record's partition, offset, and timestamp become columns in the table that Deephaven creates when it consumes the topic. You can rename these columns, but you can't change their types.
You can also add optional columns for the receive time and for the key and value sizes. The receive time is the date-time immediately after Deephaven observes the record.
| Column | Type | Property | Default name |
|---|---|---|---|
| Partition | int | deephaven.partition.column.name | KafkaPartition |
| Offset | long | deephaven.offset.column.name | KafkaOffset |
| Kafka timestamp | Instant | deephaven.timestamp.column.name | KafkaTimestamp |
| Receive time | Instant | deephaven.receivetime.column.name | Not present |
| Key size | int (bytes) | deephaven.keybytes.column.name | Not present |
| Value size | int (bytes) | deephaven.valuebytes.column.name | Not present |
Consumer properties control these columns. Set them in the configuration dictionary passed as the first argument to consume:
- To add an optional column, set its property to the column name you want.
- To disable a column that is present by default, set its property to an empty string, not
None.
For example, you might disable the partition column when the topic has only one partition.
The result_no_partition table has KafkaOffset, KafkaTimestamp, and Command columns, and no KafkaPartition column.
Join and aggregate two Kafka streams
In this example, consume reads two Kafka topics, quotes and orders, into Deephaven as blink tables. The example uses last_by to track the latest data from each topic, natural_join to join the streams, and agg.sum_ and agg.weighted_sum to aggregate the results.
The quotes consumer doesn't pass key_spec. Instead, the deephaven.key.column.name and deephaven.key.column.type properties read each record's key into a String column named Symbol (see Name key and value columns with properties).
Next, let's add records to the two topics. In a terminal, first run the following command to start writing to the quotes topic:
Then add the following entries:
After submitting the entries to the quotes topic, use the following command to write to the orders topic:
Then add the following entries in the terminal:
The tables update as you add each entry to the Kafka streams. The final results are in the total_notional table.
Consume a Kafka stream into a partitioned table
consume_to_partitioned_table consumes a topic into a partitioned table with one subtable for each Kafka partition. Its key column, Partition, holds the Kafka partition number. See the partitioned tables guide for the operations you can run on it.
consume_to_partitioned_table takes the same arguments as consume and reads all partitions by default. To read specific partitions, pass a list such as partitions=[1, 3, 5].
Custom Kafka parser
Some use cases call for custom parsing of Kafka streams, such as when payloads use non-standard encodings or need complex transformation.
See the dedicated guide, Write your own custom parser for Kafka, for a step-by-step walkthrough and complete examples.
Write to a Kafka stream
Deephaven can write tables to Kafka streams as well. When data in a table changes with real-time updates, Deephaven also writes those changes to Kafka. The Kafka producer module defines functions to do this, including produce.
When producing, a key spec and a value spec work in the other direction. They say which table columns Deephaven writes as each record's key and value.
Kafka's default partitioner hashes a non-null key to choose a partition. All records with the same key go to the same partition as long as the topic's partition count doesn't change.
In this example, we write a time table to a topic called time-topic. The example writes column X as each record's key and ignores the value.
To see the records arrive, run docker compose exec redpanda rpk topic consume time-topic in a terminal.
produce returns a callback. Keep a reference to it, such as write_topic, for as long as you want to publish. Call write_topic() to stop publishing and release its resources.
Now we write a time table to a topic called time-topic_group. The last argument, True, sets last_by_key_columns, which tells the producer to perform a last_by on the key columns before writing to the stream.
Related documentation
- Kafka basic terminology
- Table types
- Incremental update model
- Custom parser for Kafka
- Kafka overview
- Partitioned tables
- Create a time table
- Filter table data
- Exact and relational joins
- Multi-aggregation
consumeconsume_to_partitioned_tableproducetime_tableupdatelast_bynatural_joinagg_byagg.sum_agg.weighted_sumrename_columns- Date-times
consumerPydocproducerPydocjacksonPydoc