Protocol Buffers and Remote Procedure Calls

Protobuf, short for Protocol Buffers, is a language-neutral, platform-neutral mechanism for serializing structured data. By defining how data is structured a single time, you can use Protobuf to generate source code in multiple languages, which can then be used to serialize and deserialize data consistently across different systems.

Protobuf is one of several options for users who wish to build their own Deephaven client API. It is particularly powerful when considering performance, as binary serialization is much faster than string-based serialization. This is important for Deephaven, which is specifically designed to handle large amounts of data in real time. In short, Protobuf offers a closer-to-the-hardware binary representation of data, supporting higher throughput and lower latency than text-based serialization formats like JSON or XML.

Protobuf allows you to define structured data types and services in language-neutral ways so that your code can interact with other languages and platforms. This is critical for client APIs because they need to be able to communicate with a Deephaven server regardless of the client language used. For example, the Python client API communicates with the Deephaven server despite being written largely in Java. Protobuf is the mechanism that standardizes the structures and messages sent between the two.

Deephaven defines several of its APIs using gRPC, Google's open-source Remote Procedure Call (RPC) framework. gRPC uses Protobuf as its interface description language, enabling it to let programs and/or objects written in one language communicate with those written in another language. Deephaven leverages this to facilitate communication between servers and clients in different languages. In particular, gRPC's support of bidirectional streaming is a key feature of Deephaven's architecture and design. Protobuf can be used on its own, but since Deephaven uses it in conjunction with gRPC, this guide focuses on both.

This guide provides a brief overview of what Protobuf and RPC are, what they do, and gives two examples of Deephaven .proto (Protobuf) files. For more detailed and specific guidance on Protobuf and gRPC, see:

Protobuf in Deephaven

Deephaven uses gRPC (Remote Procedure Call) to define several of its APIs. gRPC uses Protobuf as its interface description language, enabling it to let programs and/or objects written in one language communicate with those written in another language. This is particularly useful for Deephaven, which is designed to work with multiple languages and platforms. In particular, gRPC's support of bidirectional streaming is a key feature of Deephaven's architecture and design.

It's because of gRPC and Protobuf that you can use multiple client APIs to interact with a Deephaven server. For example, you can use the Java client API to connect to a Deephaven server and run queries while simultaneously using the Python client API to check the results of those queries.

Protobuf concepts

Protobuf can be used on its own for things like serializing and deserializing Kafka payloads. The concepts defined in this section are specific to Protobuf; gRPC is the mechanism that enables clients to call server functions and methods, while Protobuf describes the data passed between them.

Messages

Messages are structured data types that define schemas for serialized data. These messages contain fields with names and types in a strict, well-defined sequence.

The following example defines a message called Person with four fields: name, id, email, and phones.

Enums

Enums are a way to define a set of named values. They are useful for defining a fixed set of options for a field in a message. Enums are defined using the enum keyword, followed by the name of the enum and its values.

The following example defines an Enum called PhoneType with three values: MOBILE, HOME, and WORK.

RPC concepts

Remote Procedure Calls (RPC) allow programs to execute procedures on remote systems as if they were local. RPC allows data to be sent and received between a client and server, while Protobuf defines the structure, serializes, and deserializes it.

Services

Services are a way to define a set of RPC methods that can be called remotely. Services are defined using the service keyword, followed by the name of the service and its methods.

The following example defines a Service called Greeter, which implements two messages: HelloRequest and HelloResponse to send a hello message with a name and receive a greeting in response.

Note

It's best practice to define separate messages for requests and responses, hence the HelloRequest and HelloResponse messages in the example above. The use of separate single-argument requests and responses allows for API evolution without breaking backward compatibility. For example, extending an API by including additional data in the request or response messages won't break existing servers or clients running old code.

A simple Deephaven example

To illustrate how Deephaven uses Protobuf, consider the Protobuf definition for an Input table. Input tables are tables with the special ability to add data manually via either the UI or programmatically. In the case of a client API, data can only be added programmatically. The source code shown below can be found here.

The best way to understand this example is to break it down into its components:

Syntax, packages, options, and imports

The first line of the file specifies the Protobuf version being used. In this case, it specifies proto3, the latest version of Protobuf.

Just below, the package statement prevents name clashes between protocol message types. In this case, this prevents name clashes with other protocol message types in the io.deephaven.proto.backplane.grpc package.

The option statements dictate custom behavior to provide additional metadata. In this case, the options specify:

  • java_multiple_files: Generates separate Java files for each message type.
  • optimize_for: Optimizes the generated code for speed.
  • go_package: Specifies the Go package name for the generated code.

The import statement enables the use of definitions found in another proto file. In this case, it imports the Ticket message type from the deephaven_core/proto/ticket.proto file. This allows you to use anything defined in ticket.proto in this file. In particular, the Ticket message type is used in requests to add and delete tables from input tables.

InputTableService

The InputTableService service defines the methods that can be called remotely. In this case, it defines two methods:

  • AddTableToInputTable: Adds a table to an input table.

This accepts a request of type AddTableRequest and returns a response of type AddTableResponse.

  • DeleteTableFromInputTable: Deletes a table from an input table.

This accepts a request of type DeleteTableRequest and returns a response of type DeleteTableResponse.

Messages

The InputTableService service uses four different messages:

  • AddTableRequest: The request message for the AddTableToInputTable method. It contains two fields: input_table and table_to_add, both of type Ticket. These contain information about the input table and the table that a user wishes to add to it.
  • AddTableResponse: The response message for the AddTableToInputTable method. It does not contain any fields, as the response is empty. Empty messages are useful for indicating a request was successful.
  • DeleteTableRequest: The request message for the DeleteTableFromInputTable method. It contains two fields: input_table and table_to_remove, both of type Ticket. These contain information about the input table and the table that a user wishes to remove from it.
  • DeleteTableResponse: The response message for the DeleteTableFromInputTable method. It does not contain any fields, as the response is empty. Empty messages are useful for indicating a request was successful.

Result

This proto file defines a service that allows a client API to add and remove tables of data from input tables. It can generate standardized code in many different languages, all of which could be used to communicate with a Deephaven server to make these requests.

A more complex Deephaven example

Another illustration of Deephaven's use of Protobuf is the Console API. It defines how users running a client API can run commands remotely. The Console API is far more complex than the Input Table API - the latter is a simple request/response API for a table type meant to mimic tables in spreadsheets. The former enables interaction with the Deephaven console, which is a complex system that allows users to run commands, check server health, use autocomplete, get logs, and more. The full source code for console.proto can be found here. It is collapsed by default due to its length.

console.proto

Generate Python code from Protobuf

Once you have defined your Protobuf files, you can generate code for any of its supported languages. Doing so requires the protoc compiler, which compiles proto files into source code. Once you have protoc installed, you can generate Python code from Protobuf files via the following command:

Where:

  • <PROTO_PATH>: The path to the directory containing your proto files.
  • <OUTPUT_DIR>: The directory where you want to save the generated Python code.
  • <PROTO_FILE_1>, <PROTO_FILE_2>, ...: The proto files you want to compile.