Use scikit-learn in Deephaven
This guide shows how to use scikit-learn in Deephaven queries.
scikit-learn is an open-source machine learning library for Python. It features a variety of methods for classification, regression, clustering, and dimensionality reduction.
This guide gives two examples. Both classify observations in the Iris dataset, which is available in Deephaven's examples repository. The first example uses scikit-learn with pandas DataFrames only, as a baseline. The second trains the same classifier on a Deephaven table and then applies it to a live, ticking table that receives new rows while the query runs.
The Iris flower dataset is a popular dataset commonly used in introductory machine learning applications. Introduced by R.A. Fisher in his 1936 paper, The use of multiple measurements in taxonomic problems, it contains 150 samples, 50 from each of three Iris species. The Class column holds the species: Iris-setosa, Iris-versicolor, or Iris-virginica. Each row also holds four measurements, in centimeters:
SepalLengthCM: sepal lengthSepalWidthCM: sepal widthPetalLengthCM: petal lengthPetalWidthCM: petal width
Predicting a flower's species from these measurements is a classification problem. This guide solves it with the K-Nearest Neighbors classifier. If you already know scikit-learn, skip to Classify the Iris dataset with Deephaven tables.
Install scikit-learn
scikit-learn is not included in Deephaven's default server Docker image. To use it within Deephaven, you can install it yourself or choose one of the Deephaven Docker deployments that include it:
- Python with scikit-learn, without example data.
- Python with scikit-learn and example data.
- Python with All AI (NLTK, PyTorch, scikit-learn, and TensorFlow), without example data.
- Python with All AI and example data.
Note
In this guide, we read data from Deephaven's examples repository. You can also load files from the /data directory in the Docker container, which Deephaven's Docker Compose deployments mount from ./data on your computer. See Docker bind mounts to learn more about the relation between locations in the container and the local file system.
Classify the Iris dataset with pandas
This first example shows how to use scikit-learn with pandas DataFrames to classify Iris flowers from measurements.
Let's first import all the packages we need.
Next, we import the data, encode the species names in the Class column as integers, and split the data into training and testing sets. The encoding maps Iris-setosa to 0, Iris-virginica to 1, and Iris-versicolor to 2.
Next, we construct a K-Nearest Neighbors classifier with 3 as the number of neighbors. We fit it to the training set.
Lastly, we use the classifier to predict the species of each flower in the testing set and count how many predictions are correct.
Around 95% correct for such a simple solution. Pretty neat! The exact figure changes from run to run, because sample shuffles the rows randomly before the code splits them.
This example follows a basic formula for using a classifier to solve a classification problem:
- Import data from an external source.
- Encode its class labels as integers.
- Split the data into training and testing sets.
- Define the classification model.
- Fit the classification model to the training data.
- Test the model on the testing data.
- Assess its accuracy.
In this case, a simple classifier is suitable for a simple dataset.
Classify the Iris dataset with Deephaven tables
So we just classified the Iris dataset with K-Nearest Neighbors. That's kind of cool, but wouldn't it be cooler to do it on a live feed of incoming data? Let's do that with Deephaven!
This section trains the same K-Nearest Neighbors classifier as the pandas example, this time on a Deephaven table, and then uses it to classify rows of a live table as they arrive. To keep the focus on how Deephaven moves data into and out of the model, this version trains on all 150 rows and skips the testing and accuracy steps.
First, we import the packages we need.
Load the data into a table
Next, we import the data with read_csv and encode the Class column as integers, as in the first example. Here the data goes into a Deephaven table, and an update operation calls a Python function to replace each species name with its integer label. The -> np.intc type hint makes the new Class column an int column.
Train and classify with learn
To train and classify inside Deephaven, we use the learn function. How to use deephaven.learn covers it in depth. learn takes these arguments:
table: the table to read from.model_func: the Python function that receives the gathered inputs for each batch of rows and runs the model on them.inputs: a list oflearn.Inputs that gather table columns into the model function's arguments.outputs: a list oflearn.Outputs that write the model function's results to new columns.batch_size: the maximum number of rows the model function receives at once.
First, we write the two model functions: one fits the classifier, and one uses it.
Next, we write the functions that move data between the table and the model. learn calls each learn.Input's function once per batch, with the batch's rows (rows) and that input's columns (cols). It calls each learn.Output's function once per row, with the model function's result (data) and the row's position in the batch (idx). It writes the returned value to the column that the learn.Output names.
table_to_array_doubleusesgather.table_to_numpy_2dto gather the measurement columns into a 2D NumPy array of doubles.table_to_array_intgathers theClasscolumn as integers in the same way, then callsnp.squeezeto flatten the one-column result into the 1D array of labels that scikit-learn'sfitexpects.get_predicted_classreturns one row's prediction as an integer.
Now, we can classify the Iris species from these measurements, with two calls to learn:
- The first call fits the model to the training data.
- The second call uses the fitted model to classify each row and writes the prediction to a new
ClassifiedClasscolumn.
Two arguments in the training call matter:
outputs=None, becausefit_knnproduces no output. Withoutputs=None,learnrunsfit_knnon every batch even though nothing reads a result.batch_size=150, the number of rows in the Iris dataset, sofit_knnsees every row in one batch.fit_knnbuilds a new classifier for each batch it receives, so a smaller batch size would leave a model trained only on the last batch.
The second learn call classifies the same rows the model was trained on, so the result shows how learn works rather than how accurate the model is. These are still static classifications on static data. That's not that exciting. We really want to perform the classification stage on a live data feed.
Classify live data
To demonstrate classification on live data, we create some fake Iris measurements. To keep each measurement in a plausible range, we draw it at random from the range of that column in the Iris dataset:
| Column | Minimum (cm) | Maximum (cm) |
|---|---|---|
SepalLengthCM | 4.3 | 7.9 |
SepalWidthCM | 2.0 | 4.4 |
PetalLengthCM | 1.0 | 6.9 |
PetalWidthCM | 0.1 | 2.5 |
A background thread adds one row of fake measurements, drawn from these ranges, to a live table once per second for one minute. As each row arrives, the model predicts which Iris species it belongs to.
First, we create an in-memory real-time table with DynamicTableWriter and write data to it. DynamicTableWriter produces an append-only table, so iris_classified_live keeps every prediction.
Note
For most other live-ingestion cases, we recommend table_publisher. It produces a blink table, which holds only the rows added in the current update cycle, so it would not keep every prediction here.
All it takes to change from static to live data is an add-only source table, such as the append-only table that DynamicTableWriter produces. See How to use deephaven.learn for the table types learn supports.
Now we use learn on the live table. The call matches the static classification above with two changes. Its table argument is now live_iris, and its output column is named LikelyClass. Here, batch_size=150 caps how many new rows use_knn receives at once.
As the thread writes each row to live_iris, iris_classified_live gains a matching row with its predicted species in LikelyClass:
Related documentation
- Docker bind mounts
- Table types
- Incremental update model
- Install and run with Docker
- How to install and use Python packages
- How to use
deephaven.learn - AI/ML workflows
- How to use PyTorch
- How to use TensorFlow
- Python functions in query strings
- How to write data to an in-memory, real-time table
DynamicTableWriterread_csvtable_publisherupdatedeephaven.learnPydocdeephaven.learn.gatherPydoc