Pulling real-time data into tables only shows you part of the story when it comes to making valuable insights about your stocks. Well-designed plots help you interpret market conditions quickly, and the faster you can make choices, the better. Deephaven not only provides a variety of convenient plotting tools but syncs your plots to your updating source data - as the data changes in real time, your plot changes as well.
We've been discussing crypto as one exemplary real-time use case over a series of blog posts. We've covered ingesting streaming crypto data, performing calculations on that data, and using sentiment analysis to make informed choices about when / what to buy and sell. In this iteration, we show you how to easily see your crypto assets in candle (AKA Open-High-Low-Close) charts, this time using a new data source (Finnhub). OHLC charts are a popular choice for showing the price fluctuations of a financial instrument over time, and Deephaven's query methods make them easy to render.
Not all crypto data is the same, and having a full picture of your assets is extremely important. We've been tracking Dogecoin throughout our series, and as Bitcoin's value plummets, we can use our charts to monitor how the market is changing as a result. Deephaven plots change dynamically, reacting immediately to these fluctuations based on the asset you pick. For this program to run, all you need to do is pick a search term - be it BTC, ETH, etc. The end result is several plots - one for each asset that has the chosen term. Crypto assets are complicated, but Deephaven is easy.
Getting started
Finnhub is a great tool for Real-Time RESTful APIs and Websocket for Stocks, Currencies, and Crypto. I like how easy it is to pull OHLC data for Crypto. In order to follow this example, you will need a free Finnhub API key.
Launch Deephaven
Once Deephaven is up and running, navigate to http://localhost:10000/ide. If you're a new user, see our Docker installation guide. Our crypto example is a great starting point, as there are no large packages needed: a simple pip-install and you're ready to work with the data.
Full code block
We'll start with the code itself. If you just want the code without the full explanations, expand the block below.
Complete program code
import os
os.system("pip install finnhub-python")
import finnhub
finnhub_client = finnhub.Client(api_key="<INPUT YOUR TOKEN HERE>")
import pandas as pd
import time
import re
from deephaven.DateTimeUtils import currentTime, secondsToTime, convertDateTime, millis, minus, convertPeriod
from deephaven import DynamicTableWriter
import deephaven.dtypes as dht
import threading
from deephaven import Plot
search_term = 'DOGE'
time_history = 7
def pull_coins(search_term):
exchanges = finnhub_client.crypto_exchanges()
temp_ids = []
for exchange in exchanges:
long_list = finnhub_client.crypto_symbols(exchange)
symbols = pd.DataFrame(long_list).loc[:,'symbol']
temp_ids.append(symbols.values.tolist())
ids = [item for sublist in temp_ids for item in sublist]
r=re.compile(".*({}).*".format(search_term))
return list(filter(r.match, ids))#[12:]
ids = pull_coins(search_term)
def thread_func(time_history, tableWriter):
for i in range(50):
for id in ids:
data = finnhub_client.crypto_candles(id, 15, int(millis(minus(currentTime(),convertPeriod("T"+str(int((24*time_history)))+"H")))/1000), int(millis(currentTime())/1000))
coin = id
if data['s'] =='ok' and len(data['s'])>0:
c = (data['c'])
h = (data['h'])
l = (data['l'])
o = (data['o'])
t = (data['t'])
v = (data['v'])
if c != None:
for i in range(len(c)):
tableWriter.logRow(coin, float(c[i]), float(h[i]), float(l[i]), float(o[i]), secondsToTime(t[i]), float(v[i]))
time.sleep(1)
coin_tableWriter = DynamicTableWriter(
["Coin", "Close", "High", "Low", "Open", "DateTime", "Volume"],
[dht.string, dht.float64, dht.float64, dht.float64, dht.float64, dht.datetime, dht.float64]
)
coin_data = coin_tableWriter.getTable().formatColumns("Close = Decimal(`#,###.##############`)")\
.formatColumns("High = Decimal(`#,###.##############`)")\
.formatColumns("Low = Decimal(`#,###.##############`)")\
.formatColumns("Open = Decimal(`#,###.##############`)")
thread = threading.Thread(target=thread_func, args=[time_history, coin_tableWriter])
thread.start()
pulled_ids = []
for value in coin_data.selectDistinct("Coin").getColumn("Coin").getDirect():
pulled_ids.append(value)
from deephaven import Plot
for i in range(len(pulled_ids)):
coin = pulled_ids[i]
globals()[f"ohlc_{pulled_ids[i]}"] = Plot.ohlcPlot("coins", coin_data.where("Coin= coin"), "DateTime", "Open", "High", "Low", "Close")\
.show()
What the code does
The Finnhub data has a lot of crypto symbols available - as of right now that count is 8,307 symbols over 14 different exchanges. With data on that scale, it's obvious how visualizing the data properly is essential to making informed decisions. This program will pull the list of available exchanges and then search each exchange for the keyword of those symbols. The end product is a table with all the symbols from each exchange and an OHLC chart for each symbol.
Install packages
First, we need to install the Finnhub package. Here, we do that in the Deephaven IDE so that this is ready to go for the rest of the program. Note that this command needs to be re-done with each new session. For more information on installing Python packages, see How to install Python packages.
import os
os.system("pip install finnhub-python")
Imports and keys
The next step is to define the imports for everything needed in the script and to initialize the Finnhub client with your API key. Take note that the Finnhub API has a rate limit for the data, but it's pretty generous. This API key can be an environmental variable or hard-coded.
import finnhub
import pandas as pd
import time
import re
from deephaven.DateTimeUtils import currentTime, secondsToTime, convertDateTime, millis, minus, convertPeriod
from deephaven import DynamicTableWriter
import deephaven.dtypes as dht
import threading
from deephaven import Plot
finnhub_api_key = os.environ.get('FINNHUB_API_KEY')
#finnhub_api_key = "<INPUT YOUR TOKEN HERE>"
finnhub_client = finnhub.Client(api_key=finnhub_api_key)
Personal settings
Next, assign a search term for that asset you want to monitor and a time frame for how many days to go back in history. Throughout our crypto-examples, we've used Dogecoin, so we've specified our search_term accordingly.
search_term = 'DOGE'
time_history = 7
Pull asset names
To get a full picture of the dynamic data set, I like to search for every exchange currently available and then limit the assets to those that match my search_term. This function returns all of the ids that I want to use to make an OHLC chart.
def pull_coins(search_term):
exchanges = finnhub_client.crypto_exchanges()
temp_ids = []
for exchange in exchanges:
long_list = finnhub_client.crypto_symbols(exchange)
symbols = pd.DataFrame(long_list).loc[:,'symbol']
temp_ids.append(symbols.values.tolist())
ids = [item for sublist in temp_ids for item in sublist]
r=re.compile(".*({}).*".format(search_term))
return list(filter(r.match, ids))#[12:]
ids = pull_coins(search_term)
Here are the current exchanges searched:
KRAKENBITFINEXOKEXFXPIGBITTREXCOINBASEHUOBIBITMEXBINANCEPOLONIEXZBHITBTCKUCOINGEMINI
Each exchange has its own set of symbols. To provide you with the type of data this pulls, we've listed all the symbols on Coinbase and Binance.
Warning! These lists are long - if you do not limit by a certain term, you will get A LOT of data and reach the rate limit.
COINBASE assets
['COINBASE:EOS-EUR', 'COINBASE:CLV-GBP', 'COINBASE:QUICK-USD', 'COINBASE:DDX-EUR', 'COINBASE:LTC-GBP', 'COINBASE:CHZ-EUR', 'COINBASE:XTZ-BTC', 'COINBASE:ZEC-BTC', 'COINBASE:ICP-GBP', 'COINBASE:ORN-USD', 'COINBASE:ANKR-EUR', 'COINBASE:ZEC-USDC', 'COINBASE:BAND-BTC', 'COINBASE:ZRX-BTC', 'COINBASE:MIR-USD', 'COINBASE:ADA-USD', 'COINBASE:USDC-GBP', 'COINBASE:ORN-USDT', 'COINBASE:RAD-USDT', 'COINBASE:OMG-GBP', 'COINBASE:LTC-BTC', 'COINBASE:RGT-USD', 'COINBASE:WLUNA-USD', 'COINBASE:USDT-USDC', 'COINBASE:1INCH-EUR', 'COINBASE:MLN-USD', 'COINBASE:FIL-GBP', 'COINBASE:AGLD-USD', 'COINBASE:USDT-EUR', 'COINBASE:YFII-USD', 'COINBASE:BCH-USD', 'COINBASE:BAND-EUR', 'COINBASE:QNT-USD', 'COINBASE:MATIC-BTC', 'COINBASE:KEEP-USD', 'COINBASE:MANA-EUR', 'COINBASE:GRT-GBP', 'COINBASE:WLUNA-BTC', 'COINBASE:UMA-BTC', 'COINBASE:WBTC-BTC', 'COINBASE:LPT-USD', 'COINBASE:DOT-EUR', 'COINBASE:PLA-USD', 'COINBASE:ETH-DAI', 'COINBASE:LTC-USD', 'COINBASE:BTC-USDC', 'COINBASE:SOL-EUR', 'COINBASE:FIL-EUR', 'COINBASE:DOT-USDT', 'COINBASE:FET-USD', 'COINBASE:AXS-USD', 'COINBASE:LRC-BTC', 'COINBASE:MANA-BTC', 'COINBASE:XYO-BTC', 'COINBASE:ATOM-BTC', 'COINBASE:MASK-EUR', 'COINBASE:REQ-USD', 'COINBASE:NU-BTC', 'COINBASE:PAX-USD', 'COINBASE:DDX-USD', 'COINBASE:SKL-BTC', 'COINBASE:RLC-USD', 'COINBASE:AXS-BTC', 'COINBASE:BAT-BTC', 'COINBASE:WLUNA-USDT', 'COINBASE:MANA-USD', 'COINBASE:RLY-USDT', 'COINBASE:FIL-USD', 'COINBASE:CLV-EUR', 'COINBASE:CTSI-BTC', 'COINBASE:XYO-USD', 'COINBASE:DOGE-GBP', 'COINBASE:MASK-USDT', 'COINBASE:DOGE-BTC', 'COINBASE:RAD-BTC', 'COINBASE:ZEN-USD', 'COINBASE:AAVE-EUR', 'COINBASE:IOTX-USD', 'COINBASE:BAT-EUR', 'COINBASE:CTSI-USD', 'COINBASE:NKN-BTC', 'COINBASE:REQ-BTC', 'COINBASE:ZEC-USD', 'COINBASE:COTI-USD', 'COINBASE:CLV-USDT', 'COINBASE:AVAX-USDT', 'COINBASE:ZEN-USDT', 'COINBASE:LINK-EUR', 'COINBASE:BNT-USD', 'COINBASE:AMP-USD', 'COINBASE:GRT-USD', 'COINBASE:GRT-EUR', 'COINBASE:ARPA-EUR', 'COINBASE:DOT-BTC', 'COINBASE:AAVE-USD', 'COINBASE:AXS-EUR', 'COINBASE:SUSHI-ETH', 'COINBASE:IOTX-BTC', 'COINBASE:REP-BTC', 'COINBASE:BAT-USD', 'COINBASE:CHZ-USDT', 'COINBASE:LINK-GBP', 'COINBASE:SOL-USD', 'COINBASE:EOS-BTC', 'COINBASE:XLM-BTC', 'COINBASE:ASM-USDT', 'COINBASE:BOND-USD', 'COINBASE:MIR-BTC', 'COINBASE:TRB-USD', 'COINBASE:RAD-EUR', 'COINBASE:ACH-USD', 'COINBASE:AUCTION-USDT', 'COINBASE:BTC-USDT', 'COINBASE:CRV-USD', 'COINBASE:ICP-EUR', 'COINBASE:ANKR-GBP', 'COINBASE:DASH-USD', 'COINBASE:AXS-USDT', 'COINBASE:BTC-EUR', 'COINBASE:NKN-GBP', 'COINBASE:CGLD-GBP', 'COINBASE:CRV-BTC', 'COINBASE:DNT-USDC', 'COINBASE:CGLD-EUR', 'COINBASE:ADA-ETH', 'COINBASE:BADGER-USD', 'COINBASE:WCFG-USD', 'COINBASE:ANKR-BTC', 'COINBASE:1INCH-GBP', 'COINBASE:GTC-USD', 'COINBASE:CGLD-BTC', 'COINBASE:SKL-USD', 'COINBASE:ENJ-USD', 'COINBASE:DOT-USD', 'COINBASE:SUSHI-EUR', 'COINBASE:CLV-USD', 'COINBASE:REQ-GBP', 'COINBASE:XTZ-EUR', 'COINBASE:SUSHI-BTC', 'COINBASE:MKR-USD', 'COINBASE:JASMY-USD', 'COINBASE:NU-EUR', 'COINBASE:MIR-GBP', 'COINBASE:ETC-BTC', 'COINBASE:ALGO-USD', 'COINBASE:DDX-USDT', 'COINBASE:SNX-GBP', 'COINBASE:XTZ-USD', 'COINBASE:SHIB-USD', 'COINBASE:OMG-BTC', 'COINBASE:RLY-GBP', 'COINBASE:PERP-EUR', 'COINBASE:USDC-EUR', 'COINBASE:ZRX-USD', 'COINBASE:WCFG-GBP', 'COINBASE:BAT-ETH', 'COINBASE:BTRST-USD', 'COINBASE:REQ-EUR', 'COINBASE:SUSHI-GBP', 'COINBASE:ETC-GBP', 'COINBASE:UMA-EUR', 'COINBASE:REN-USD', 'COINBASE:OXT-USD', 'COINBASE:WCFG-BTC', 'COINBASE:SOL-USDT', 'COINBASE:PERP-USD', 'COINBASE:LINK-ETH', 'COINBASE:CHZ-USD', 'COINBASE:SKL-GBP', 'COINBASE:BADGER-USDT', 'COINBASE:BNT-BTC', 'COINBASE:CVC-USDC', 'COINBASE:UST-USD', 'COINBASE:ETH-BTC', 'COINBASE:ASM-USD', 'COINBASE:UNI-BTC', 'COINBASE:DOT-GBP', 'COINBASE:BTC-USD', 'COINBASE:YFI-BTC', 'COINBASE:ETC-USD', 'COINBASE:LOOM-USDC', 'COINBASE:IOTX-USDT', 'COINBASE:ENJ-BTC', 'COINBASE:ARPA-USDT', 'COINBASE:SOL-GBP', 'COINBASE:FX-USD', 'COINBASE:WCFG-USDT', 'COINBASE:CRV-EUR', 'COINBASE:YFI-USD', 'COINBASE:SNX-BTC', 'COINBASE:MATIC-GBP', 'COINBASE:ALGO-EUR', 'COINBASE:BTC-GBP', 'COINBASE:MKR-BTC', 'COINBASE:ETH-USD', 'COINBASE:AGLD-USDT', 'COINBASE:USDT-USD', 'COINBASE:BADGER-EUR', 'COINBASE:ETH-USDT', 'COINBASE:NMR-USD', 'COINBASE:ETH-GBP', 'COINBASE:AVAX-EUR', 'COINBASE:ICP-USDT', 'COINBASE:MASK-USD', 'COINBASE:NMR-EUR', 'COINBASE:PAX-USDT', 'COINBASE:REQ-USDT', 'COINBASE:BTRST-EUR', 'COINBASE:MANA-ETH', 'COINBASE:LRC-USD', 'COINBASE:COMP-BTC', 'COINBASE:NMR-BTC', 'COINBASE:STORJ-BTC', 'COINBASE:MIR-EUR', 'COINBASE:REN-BTC', 'COINBASE:GNT-USDC', 'COINBASE:FORTH-USD', 'COINBASE:UST-EUR', 'COINBASE:UMA-GBP', 'COINBASE:UMA-USD', 'COINBASE:MANA-USDC', 'COINBASE:LINK-USD', 'COINBASE:BTRST-GBP', 'COINBASE:BCH-GBP', 'COINBASE:1INCH-USD', 'COINBASE:EOS-USD', 'COINBASE:DAI-USD', 'COINBASE:TRIBE-USD', 'COINBASE:RARI-USD', 'COINBASE:UNI-USD', 'COINBASE:OGN-USD', 'COINBASE:WBTC-USD', 'COINBASE:BAL-BTC', 'COINBASE:MATIC-USD', 'COINBASE:STORJ-USD', 'COINBASE:DOGE-EUR', 'COINBASE:FORTH-EUR', 'COINBASE:AUCTION-EUR', 'COINBASE:XYO-EUR', 'COINBASE:1INCH-BTC', 'COINBASE:OGN-BTC', 'COINBASE:ADA-BTC', 'COINBASE:FORTH-BTC', 'COINBASE:NU-GBP', 'COINBASE:RLY-USD', 'COINBASE:CRV-GBP', 'COINBASE:BNT-GBP', 'COINBASE:NKN-USD', 'COINBASE:ZEN-BTC', 'COINBASE:ANKR-USD', 'COINBASE:ICP-USD', 'COINBASE:BCH-EUR', 'COINBASE:DAI-USDC', 'COINBASE:RLY-EUR', 'COINBASE:BNT-EUR', 'COINBASE:NU-USD', 'COINBASE:TRU-USDT', 'COINBASE:ZRX-EUR', 'COINBASE:RAD-GBP', 'COINBASE:TRU-BTC', 'COINBASE:FIL-BTC', 'COINBASE:BAND-USD', 'COINBASE:ICP-BTC', 'COINBASE:BAT-USDC', 'COINBASE:MASK-GBP', 'COINBASE:SKL-EUR', 'COINBASE:BTRST-BTC', 'COINBASE:DOGE-USD', 'COINBASE:DOGE-USDT', 'COINBASE:NKN-EUR', 'COINBASE:NMR-GBP', 'COINBASE:CGLD-USD', 'COINBASE:KNC-USD', 'COINBASE:SUSHI-USD', 'COINBASE:SNX-USD', 'COINBASE:BTC-UST', 'COINBASE:POLY-USD', 'COINBASE:WCFG-EUR', 'COINBASE:WLUNA-EUR', 'COINBASE:FARM-USD', 'COINBASE:TRB-BTC', 'COINBASE:BAL-USD', 'COINBASE:TRU-EUR', 'COINBASE:UST-USDT', 'COINBASE:ALGO-GBP', 'COINBASE:PERP-USDT', 'COINBASE:ARPA-USD', 'COINBASE:AUCTION-USD', 'COINBASE:WLUNA-GBP', 'COINBASE:AAVE-BTC', 'COINBASE:DASH-BTC', 'COINBASE:OMG-USD', 'COINBASE:SHIB-USDT', 'COINBASE:LTC-EUR', 'COINBASE:BAND-GBP', 'COINBASE:OMG-EUR', 'COINBASE:RAI-USD', 'COINBASE:RAD-USD', 'COINBASE:ADA-USDC', 'COINBASE:MATIC-EUR', 'COINBASE:AAVE-GBP', 'COINBASE:BTRST-USDT', 'COINBASE:RLC-BTC', 'COINBASE:ETH-EUR', 'COINBASE:POLY-USDT', 'COINBASE:FORTH-GBP', 'COINBASE:XLM-EUR', 'COINBASE:GRT-BTC', 'COINBASE:ETH-USDC', 'COINBASE:USDT-GBP', 'COINBASE:XYO-USDT', 'COINBASE:XLM-USD', 'COINBASE:FET-USDT', 'COINBASE:ALGO-BTC', 'COINBASE:CHZ-GBP', 'COINBASE:KNC-BTC', 'COINBASE:AVAX-USD', 'COINBASE:ADA-GBP', 'COINBASE:LINK-BTC', 'COINBASE:TRU-USD', 'COINBASE:REP-USD', 'COINBASE:IOTX-EUR', 'COINBASE:ORN-BTC', 'COINBASE:SNX-EUR', 'COINBASE:SOL-BTC', 'COINBASE:ADA-EUR', 'COINBASE:BCH-BTC', 'COINBASE:XTZ-GBP', 'COINBASE:ETC-EUR', 'COINBASE:ATOM-USD', 'COINBASE:COMP-USD']
BINANCE assets
Pull coin data
Now that we know the assets to monitor and the right format for each asset's name, we can pull the actual data.
We want to get the time_history for each of the ids we generated above.
The resolution is the time intervals for which the data is aggregated. These are the only accepted values:
1: 1 minute intervals5: 5-minutes intervals15: 15-minutes intervals30: 30-minutes intervals60: Hourly intervalsD: Daily intervalsW: Weekly intervalsM: Monthly intervals
Notice that when the data comes in, it needs to be cleaned. Only assets with an ok status are accepted, and we want to limit this further to only the data that has values en(data['s']). These are included so that bad data is never logged to the table.
The Finnhub mapping of data is:
c: List of close prices for returned candles.h: List of high prices for returned candles.l: List of low prices for returned candles.o: List of open prices for returned candles.s: Status of the response. This field can either beokorno_data.t: List of timestamp for returned candles.v: List of volume data for returned candles.
resolution = 5 # Available values: 1, 5, 15, 30, 60, 'D', 'W', 'M'
def thread_func(time_history, tableWriter):
for id in ids:
data = finnhub_client.crypto_candles(id, resolution, int(millis(minus(currentTime(),convertPeriod("T"+str(int((24*time_history)))+"H")))/1000), int(millis(currentTime())/1000))
coin = id
if data['s'] =='ok' and len(data['s'])>0:
c = (data['c'])
h = (data['h'])
l = (data['l'])
o = (data['o'])
t = (data['t'])
v = (data['v'])
if c != None:
for i in range(len(c)):
tableWriter.logRow(coin, float(c[i]), float(h[i]), float(l[i]), float(o[i]), secondsToTime(t[i]), float(v[i]))
We define our table so the variables match our preferred data types. That is, we are mapping the data collected in the above function with these Deephaven Python Types.
coin_tableWriter = DynamicTableWriter(
["Coin", "Close", "High", "Low", "Open", "DateTime", "Volume"],
[dht.string, dht.float64, dht.float64, dht.float64, dht.float64, dht.datetime, dht.float64]
)
Finally, we get to run the code that will execute the functions. Since many crypto assets have values that are well below zero, I format the columns so that we can see all the numbers desired.
This also starts the treading function to run the prior functions.
After running these commands a table with lots of values will appear in your IDE.
coin_data = coin_tableWriter.getTable().formatColumns("Close = Decimal(`#,###.##############`)")\
.formatColumns("High = Decimal(`#,###.##############`)")\
.formatColumns("Low = Decimal(`#,###.##############`)")\
.formatColumns("Open = Decimal(`#,###.##############`)")
thread = threading.Thread(target=thread_func, args=[time_history, coin_tableWriter])
thread.start()
Plot the data
Having a table of values is useful for data processing, and there are plenty of ways to interact with it. However, I want to view the data in OHLC charts.
The query below goes through the data pulled, creates an array of pulled_ids, then creates a new chart. We create this new value of ids to be only the data that had values. Since each pulled_ids will have its own chart, this might take a few seconds depending on the number of ids.
pulled_ids = []
for value in coin_data.selectDistinct("Coin").getColumn("Coin").getDirect():
pulled_ids.append(value)
for i in range(len(pulled_ids)):
coin = pulled_ids[i]
globals()[f"ohlc_{pulled_ids[i]}"] = Plot.ohlcPlot("coins", coin_data.where("Coin= coin"), "DateTime", "Open", "High", "Low", "Close")\
.show()
Endless possibilities
We demonstrate OHLC plots here; to experiment with our other chart types and style options, check out our plotting documentation. We hope the working example above emulates the real-time data-driven problem sets you have. We encourage you to evolve the query to deliver the business logic or analysis that interests you.
This pattern and workflow serve applications well beyond stock data. IOT, weather, sports, and industrial telemetry all offer exciting and challenging use cases. Let us know how you adapt this project. Reach out on Slack or post in our Github Discussions. Our dev-rel team is here to support you.