Deephaven already offers several clients in varied languages (Python, Java, and C++ at the time of writing), but we are always looking to make the power of dynamic and streaming data available to more users. So, at the end of this month, Deephaven will also offer a native client for the Go programming language. In this article I’ll discuss the new client and what’s so special about it.
Golang, Deephaven, and streaming tables
Deephaven is a flexible data stack designed from the ground up to excel with real-time table data, both on its own and in conjunction with static (batch) data. Traditional table-based systems process data all at once, and waste resources recomputing data that hasn’t changed, making them unsuitable for use cases that Deephaven is uniquely designed for. High-throughput, low-latency, real-time applications and analytics are where Deephaven shines— and we’re excited to offer these capabilities to users building Go applications.
What can you do with the first release?
The first release of the client will enable you to perform all of the following operations in native Go:
- Uploading and downloading table data
- Using Deephaven’s rich API to manipulate updating or static tables (one at a time or in batches)
- Running Python or Groovy scripts directly on the server, including those that have embedded user-defined functions or integrate with 3rd party libraries
The Go client also supports input tables, allowing you to stream new data to the server in real time. The streamed data can be used like any other Deephaven table — filtered, sorted, truncated, updated, or projected incrementally.
Streaming table updates will be supported in the next release
Though the client will be powerful already, the second release will add the ability to receive streaming table updates from the server in real time.
Why Go?
With open-source client APIs in Python, Java, and C++ already, the Deephaven community has expressed interest in Go, Rust, and Julia as the next priorities. (Some have requested C# as well; we have an enterprise client API that we soon expect to port).
We chose to prioritize the development of the Go client for reasons that may sound familiar:
Go is popular
Go has been on the rise for a while now, and it appears in several lists of the most popular programming languages. It’s also popular among Deephaven users—it was the most-requested language for a new client.
Go is easy to learn
Go touts simplicity as one of its defining features, and it doesn’t disappoint. At the beginning of last month I had never written a single line of Go before, and was worried I would need to spend months learning Go idioms and library features. Instead, I completed the tutorial in under a day and could write code fluidly in only a few weeks.
Go is well-suited for clients
Go offers a plethora of features that make writing networked applications simple. Lightweight concurrency, request management in the standard library, and straightforward error handling all contribute to easily-digestible and robust code.
A ‘Hello, World’ from Go
Let’s roll up our sleeves and actually do some work with the API. (This code might be different from the release version, and error handling has been omitted for brevity). First, we’re going to need to have a Deephaven server running. Then, importing and installing the client is trivial with Go’s package management:
import (
"context"
"fmt"
"github.com/deephaven/deephaven-core/go/client/client"
)
Now we can connect to the server and run a few operations. Let's create a time table, filter it, and give it a name so that we can see it in the web UI.
ctx := context.Background()
myClient, err := client.NewClient(ctx, "localhost", "10000", client.WithConsole("python"))
defer myClient.Close()
timeTable, err := myClient.TimeTable(ctx, time.Second, time.Now())
err = myClient.BindToVariable(ctx, "myTimeTable", timeTable)
If we open the web UI at http://localhost:10000/ide/, we can see the time table that we just created.
A more exciting example
It wouldn’t be very exciting if making a time table was all the Go client could do. Instead, let’s walk through writing a real-time CPU usage display. Again, error handling is omitted for brevity — in a real application, you want to handle errors properly!
The Go client has support for input tables, so we can stream in our CPU data in real time. Let's create an input table to hold our raw data.
schema := arrow.NewSchema(
[]arrow.Field{
{Name: "Time", Type: arrow.PrimitiveTypes.Int64},
{Name: "CpuTime", Type: arrow.PrimitiveTypes.Float64},
{Name: "IdleTime", Type: arrow.PrimitiveTypes.Float64},
},
nil,
)
inputTable, err := myClient.NewAppendOnlyInputTableFromSchema(ctx, schema)
The input table by itself just holds the CPU statistics straight from the system. If we want to get an actual percent utilization, we have to do some processing on it. Thankfully, the Go client also supports an expressive query system, which is an easy and efficient way to perform table operations. Deephaven has rich coverage of relational and time series methods, so calculating CPU usage over a sliding window is easy.
timedQuery := inputTable.
Query().
Update("Time = nanosToTime(Time)", "PastTime = Time - 5 * SECOND")
// Here we join the current CPU stats with the stats from 5 seconds ago.
// Then we do some math to see what proportion of the time was spent idle.
joinedQuery := timedQuery.
AsOfJoin(
timedQuery,
[]string{"PastTime = Time"},
[]string{"PastTime = Time", "PastCpuTime = CpuTime", "PastIdleTime = IdleTime"},
client.MatchRuleLessThanEqual).
DropColumns("PastTime").
Update(
"CpuTimeChange = CpuTime - PastCpuTime",
"IdleTimeChange = IdleTime - PastIdleTime",
"CpuUtil = 100.0 * (1.0 - (IdleTimeChange / CpuTimeChange))")
cleanQuery := joinedQuery.
View("Time", "CpuUtil").
Tail(60)
tables, err := myClient.ExecBatch(ctx, timedQuery, joinedQuery, cleanQuery)
timedTable, joinedTable, cleanTable := tables[0], tables[1], tables[2]
err = myClient.BindToVariable(ctx, "go_cpu_usage", cleanTable)
And finally, we need to actually put some data into the input table. We're going to use the goprocinfo package to get the CPU stats, then put the stats into an Apache Arrow table. Finally, we can upload that table to the server and add it to the input. Even though we're constantly adding new data, Deephaven processes only the new rows, and thus our application remains lightning fast whether we have ten or ten billion data points.
for {
cpuStat, err := linuxproc.ReadStat("/proc/stat")
timestamp := time.Now().UnixNano()
totalTime := cpuStat.CPUStatAll.User + cpuStat.CPUStatAll.Nice +
cpuStat.CPUStatAll.System + cpuStat.CPUStatAll.Idle +
cpuStat.CPUStatAll.IOWait + cpuStat.CPUStatAll.IRQ +
cpuStat.CPUStatAll.SoftIRQ
idleTime := cpuStat.CPUStatAll.Idle
b := array.NewRecordBuilder(memory.DefaultAllocator, schema)
b.Field(0).(*array.Int64Builder).AppendValues([]int64{timestamp}, nil)
b.Field(1).(*array.Float64Builder).AppendValues([]float64{cpuTime}, nil)
b.Field(2).(*array.Float64Builder).AppendValues([]float64{idle}, nil)
newRec := b.NewRecord()
newData, err := myClient.ImportTable(ctx, newRec)
err = inputTable.AddTable(ctx, newData)
err = newData.Release(ctx)
newRec.Release()
b.Release()
time.Sleep(time.Second)
}
Then, if we open up the web UI, we can see that our table is there and is updating in real time. Neat!
Of course, this isn’t everything the Go client will be able to do. It implements nearly all of the table operations of the server-side API (for which there is a tutorial here), and once released, the client will have detailed examples for all of its functionality. Hopefully the preview above sparks ideas for potential applications of the new client. I’m looking forward to seeing what exciting things people will create with it! Feel free to share your ideas in the Deephaven Community Slack.