Skip to main content

Use URLs to easily access & import publicly available data

· 4 min read
Joshua Hilgartner

The core requirement for any data project is the data itself. This means the easier it is to access the data, the sooner you can get started on the interesting part of your project. The right tools can streamline this process and, in turn, improve your productivity. By simply reading URLs, Deephaven can read in data anywhere it is publicly available. This makes accessing data easy and allows you to focus on what truly matters in your projects.

This Deephaven functionality allowed me to make progress on financial projects I've been working on. I needed access to historical stock data for a few different stocks. Yahoo Finance is a popular resource, and I was happy to learn how incredibly easy it is to import its data into the Deephaven IDE for analysis. I walk you through the steps below.

When I set out to get financial data, my first instinct was to go to Yahoo Finance's website and simply download the stocks that I wanted to analyze. I then placed them into my project directory and could read the files immediately.

from deephaven import read_csv

bitcoin = read_csv("/data/BTC.csv")

While this works, it was very inefficient. I had to navigate through Yahoo Finance's website for a bit to get what I wanted. Then I had to relocate the files to the proper place manually. So, I looked for an alternative way in Deephaven's API. I found another method that didn't require any local file downloads. All I needed was a URL. Additionally, I could edit a few values to define a specific time range. In this way, I could quickly change which stocks I was looking at and didn't need to go to Yahoo Finance's website or even leave the Deephaven IDE. Needless to say, my productivity went through the roof.

There are many different types of links that Deephaven can read. In the case of my project, the link looked like this:

Example link: https://query1.finance.yahoo.com/v7/finance/download/SYMBOL?period1=SOMETHING&period2=SOMETHING&interval=INTERVAL&events=history&includeAdjustedClose=true

The fully capitalized text is replaced with different values according to what data you want.

  • Symbol - the symbol for the stock, such as AAPL for Apple or TSLA for Tesla
  • Interval - the frequency for which you want historical data
    • 1d (Daily)
    • 1wk (Weekly)
    • 1mo (Monthly)
  • Events - what type of events you want
    • history (Historical prices)
    • div (Dividends only)
    • split (Stock splits)
    • capitalGain (Capital gain)
  • Period 1 - start date as a Unix timestamp
  • Period 2 - end date as a Unix timestamp
from deephaven import read_csv

tesla = read_csv("https://query1.finance.yahoo.com/v7/finance/download/TSLA?period1=1622583391&period2=1654119391&interval=1d&events=history&includeAdjustedClose=true")

One small complication of this method was that the link requires the start and end date to be input as a Unix timestamp. Since Unix time is not a common method of timekeeping in the real world, this makes it slightly difficult to do by hand. However, simple code can be used to convert a date into the proper form. Since Yahoo Finance doesn't use fractions of seconds, we round the exact Unix time. However, this doesn't change how easy it is to access the desired period.

from datetime import datetime

# Conversion for May 2nd, 2022 @ 13:01:30 pm UTC
timestamp = int(datetime(2022, 5, 2, 13, 1, 30).timestamp())

print(timestamp)

Near limitless integrations

As you can see, accessing and importing data with Deephaven is very easy, which makes Deephaven a super useful tool. It can access data in a multitude of forms that can come from many different places, including sites like Yahoo Finance. In addition, the data molds quickly into Deephaven's IDE, meaning you don't have to waste any time trying to upload your data. If you haven't already, I recommend integrating Deephaven into your next data project. You can try these methods for yourself on the demo system or by following the Quick Start guide to build and launch the IDE.

Want to learn more? Check out our documentation and examples or reach out to us on Slack to get started today!