Deephaven's modernized time library

note

Since publishing this blog, we have further improved our time library. See Deephaven's modernized time library -- Part 2 for more details.

Deephaven Core v0.25 introduces major changes to modernize and standardize its date-time handling. The query libraries were originally written more than a decade ago. Over that time, Java's approach to date-time data has changed, so a rewrite was in order. v0.25 brings Deephaven up to speed with current standards.

This rewrite is a breaking change. Your data is safe but many of your queries may need to be updated. A stable API is a high priority for Deephaven and our customers; we did not take this decision lightly. Instead of a series of small changes, we decided that one large change would result in less disruption for users. We appreciate the time and effort involved and thank you for going on this journey with us.

If you are having difficulty porting your code, please contact Deephaven on Slack.

Overview

• Standard Java date-time types (Instant, LocalDate, LocalTime, ZoneId, ZonedDateTime, Period, Duration) replace custom Deephaven date-time types (DateTime, TimeZone, Period).
• Functions have been thoughtfully and consistently renamed.
• Missing functions have been added.
• Rarely used functions have been removed.
• Inconsistent function behavior has been resolved.
• Functions are fully documented (Javadoc and Pydoc).
• Unit test coverage is now >90%.
• References to the Joda time library have been removed.
• ISO standard formats replace Deephaven-specific formats.
• More date-time types and formats can be parsed.
• User-configurable time zone aliases replace hard-coded time zone aliases (e.g., NY).
• The Java ZoneId default time zone replaces two default time zones.

Porting

We have renamed functions and deleted a few that are rarely used.

• See our Pydoc to reference the new Python API.
• See our Javadoc to reference the new Java API.

To aid in porting code, the following tables indicate approximate relationships between old and new function names.

Java API changes

Type	New Method / Constant	Old Method / Constant
Arithmetic	diffDays	dayDiff, diffDay
Arithmetic	diffMicros	-
Arithmetic	diffMillis	-
Arithmetic	diffMinutes	-
Arithmetic	diffNanos	diffNanos, diff
Arithmetic	diffSeconds	-
Arithmetic	diffYears365	diffYear, yearDiff
Arithmetic	diffYearsAvg	-
Arithmetic	minus	minus
Arithmetic	plus	plus
Arithmetic	-	cappedTimeOffset
Binning	lowerBin	lowerBin
Binning	upperBin	upperBin
Chronology	atMidnight	dateAtMidnight, millisToDateAtMidnight
Chronology	dayOfMonth	dayOfMonth
Chronology	dayOfWeek	dayOfWeek
Chronology	dayOfYear	dayOfYear
Chronology	hourOfDay	hourOfDay
Chronology	microsOfMilli	microsOfMilli
Chronology	microsOfSecond	-
Chronology	millisOfDay	millisOfDay
Chronology	millisOfSecond	millisOfSecond
Chronology	minuteOfDay	minuteOfDay
Chronology	minuteOfHour	minuteOfHour
Chronology	monthOfYear	monthOfYear
Chronology	nanosOfDay	nanosOfDay
Chronology	nanosOfMilli	-
Chronology	nanosOfSecond	nanosOfSecond
Chronology	secondOfDay	secondOfDay
Chronology	secondOfMinute	secondOfMinute
Chronology	year	year
Chronology	yearOfCentury	yearOfCentury
Clock	currentClock	currentClock
Clock	now	currentTime
Clock	nowMillisResolution	currentTimeMillis
Clock	nowSystem	-
Clock	nowSystemMillisResolution
Clock	setClock	-
Clock	today	currentDate
Clock	-	clock
Comparisons	isAfter	isAfter
Comparisons	isAfterOrEqual	-
Comparisons	isBefore	isBefore
Comparisons	isBeforeOrEqual	-
Constants	DAY	DAY
Constants	DAYS_PER_NANO	-
Constants	HOUR	HOUR
Constants	HOURS_PER_NANO	-
Constants	MICRO	-
Constants	MILLI	-
Constants	MINUTE	MINUTE
Constants	MINUTES_PER_NANO	-
Constants	SECOND	SECOND
Constants	SECONDS_PER_NANO	-
Constants	WEEK	WEEK
Consants	YEAR_365	YEAR
Constants	YEAR_AVG	-
Constants	YEARS_PER_NANO_365	-
Constants	YEARS_PER_NANO_AVG	-
Constants	ZERO_LENGTH_INSTANT_ARRAY	ZERO_LENGTH_DATETIME_ARRAY
Conversions: Date-time types	toDate	-
Conversions: Date-time types	toInstant	toDateTime
Conversions: Date-time types	toLocalDate	-
Conversions: Date-time types	toLocalTime	-
Conversions: Date-time types	toZonedDateTime	getZonedDateTime
Conversions: Epoch	epochAutoToEpochNanos	autoEpochToNanos
Conversions: Epoch	epochAutoToInstant	autoEpochToTime
Conversions: Epoch	epochAutoToZonedDateTime	-
Conversions: Epoch	epochMicros	-
Conversions: Epoch	epochMicrosToInstant	microsToTime
Conversions: Epoch	epochMicrosToZonedDateTime	-
Conversions: Epoch	epochMillis	millis
Conversions: Epoch	epochMillisToInstant	millisToTime
Conversions: Epoch	epochMillisToZonedDateTime	-
Conversions: Epoch	epochNanos	nanos, toEpochNano
Conversions: Epoch	epochNanosToInstant	nanosToTime, makeInstant
Conversions: Epoch	epochNanosToZonedDateTime	makeZonedDateTime
Conversions: Epoch	epochSeconds	-
Conversions: Epoch	epochSecondsToInstant	secondsToTime
Conversions: Epoch	epochSecondsToZonedDateTime	-
Conversions: Excel	excelToInstant	-
Conversions: Excel	excelToZonedDateTime	-
Conversions: Excel	toExcelTime	getExcelDateTime
Conversions: Time units	microsToMillis	-
Conversions: Time units	microsToNanos	microsToNanos
Conversions: Time units	microsToSeconds	-
Conversions: Time units	millisToMicros	-
Conversions: Time units	millisToNanos	millisToNanos
Conversions: Time units	millisToSeconds	-
Conversions: Time units	nanosToMicros	nanosToMicros
Conversions: Time units	nanosToMillis	nanosToMillis
Conversions: Time units	nanosToSeconds	-
Conversions: Time units	secondsToMicros	-
Conversions: Time units	secondsToMillis	-
Conversions: Time units	secondsToNanos	secondsToNanos
Format	formatDate	formatDate
Format	formatDateTime	format
Format	formatDurationNanos	-
Format	-	getPartitionFromTimestampMillis
Format	-	getPartitionFromTimestampMicros
Format	-	getPartitionFromTimestampNanos
Format	-	getPartitionFromTimestampSeconds
Format	-	createFormatter
Format	-	DateStyle
Overflow / Underflow	DateTimeOverflowException	DateTimeOverflowException
Parse	DateTimeParseException	-
Parse	parseDuration	convertPeriod
Parse	parseDurationNanos	expressionToNanos
Parse	parseDurationNanosQuiet	-
Parse	parseDurationQuiet	convertPeriodQuiet
Parse	parseEpochNanos	-
Parse	parseEpochNanosQuiet	-
Parse	parseInstant	convertDateTime
Parse	parseInstantQuiet	convertDateTimeQuiet
Parse	parseLocalDate	convertDate
Parse	parseLocalDateQuiet	convertDateQuiet
Parse	parseLocalTime	convertLocalTime
Parse	parseLocalTimeQuiet	convertLocalTimeQuiet
Parse	parsePeriod	convertPeriod
Parse	parsePeriodQuiet	convertPeriodQuiet
Parse	parseTimePrecision	getFinestDefinedUnit
Parse	parseTimePrecisionQuiet	-
Parse	parseTimeZone	-
Parse	parseTimeZoneQuiet	-
Parse	parseZonedDateTime	-
Parse	parseZonedDateTimeQuiet	-
Parse	-	convertTime
Parse	-	convertTimeQuiet
Time Zone	timeZone	-
Time Zone	timeZoneAliasAdd	-
Time Zone	timeZoneAliasRm	-
Query Language	TimeLiteralReplacedExpression	convertExpression

Python API changes

Type	New Method / Constant	Old Method / Constant
Arithmetic	diff_days	-
Arithmetic	diff_micros	-
Arithmetic	diff_millis	-
Arithmetic	diff_minutes	-
Arithmetic	diff_nanos	diff_nanos, minus
Arithmetic	diff_seconds	-
Arithmetic	diff_years_365	-
Arithmetic	diff_years_avg	-
Arithmetic	minus_period	minus_period, minus_nanos
Arithmetic	plus_period	plus_period, plus_nanos
Chronology	at_midnight	datetime_at_midnight
Chronology	day_of_month	day_of_month
Chronology	day_of_week	day_of_week
Chronology	day_of_year	day_of_year
Chronology	hour_of_day	hour_of_day
Chronology	micros_of_milli	microsOfMilli
Chronology	micros_of_second	-
Chronology	millis_of_day	millis_of_day
Chronology	millis_of_second	millis_of_second
Chronology	minute_of_day	minute_of_day
Chronology	minute_of_hour	minute_of_hour
Chronology	month_of_year	month_of_year
Chronology	nanos_of_day	nanos_of_day
Chronology	nanos_of_milli	-
Chronology	nanos_of_second	nanos_of_second
Chronology	second_of_day	second_of_day
Chronology	second_of_minute	second_of_minute
Chronology	year	year
Chronology	year_of_century	year_of_century
Clock	now	now
Clock	today	-
Comparisons	is_after	is_after
Comparisons	is_after_or_equal	-
Comparisons	is_before	is_before
Comparisons	is_before_or_equal	-
Constants	DAY	DAY
Constants	DAYS_PER_NANO	-
Constants	HOUR	HOUR
Constants	HOURS_PER_NANO	-
Constants	MICRO	-
Constants	MILLI	-
Constants	MINUTE	MINUTE
Constants	MINUTES_PER_NANO	-
Constants	SECOND	SECOND
Constants	SECONDS_PER_NANO	-
Constants	WEEK	WEEK
Consants	YEAR_365	YEAR
Constants	YEAR_AVG	-
Constants	YEARS_PER_NANO_365	-
Constants	YEARS_PER_NANO_AVG	-
Conversions: Date-time types	make_instant	-
Conversions: Date-time types	make_zdt	-
Conversions: Date-time types	to_instant	to_datetime
Conversions: Date-time types	to_local_date	-
Conversions: Date-time types	to_local_time	-
Conversions: Date-time types	to_zdt	-
Conversions: Epoch	epoch_auto_to_epoch_nanos	-
Conversions: Epoch	epoch_auto_to_instant	-
Conversions: Epoch	epoch_auto_to_ztd	-
Conversions: Epoch	epoch_micros	-
Conversions: Epoch	epoch_micros_to_instant	-
Conversions: Epoch	epoch_micros_to_zdt	-
Conversions: Epoch	epoch_millis	millis
Conversions: Epoch	epoch_millis_to_instant	millis_to_datetime
Conversions: Epoch	epoch_millis_to_zdt	-
Conversions: Epoch	epoch_nanos	nanos, toEpochNano
Conversions: Epoch	epoch_nanos_to_instant	nanos_to_datetime
Conversions: Epoch	epoch_nanos_to_ztd	-
Conversions: Epoch	epoch_seconds	-
Conversions: Epoch	epoch_seconds_to_instant	-
Conversions: Epoch	epoch_seconds_to_zdt	-
Conversions: Excel	excel_to_instant	-
Conversions: Excel	excel_to_ztd	-
Conversions: Excel	to_excel_time	-
Conversions: Time units	micros_to_millis	-
Conversions: Time units	micros_to_nanos	-
Conversions: Time units	micros_to_seconds	-
Conversions: Time units	millis_to_micros	-
Conversions: Time units	millis_to_nanos	millis_to_nanos
Conversions: Time units	millis_to_seconds	-
Conversions: Time units	nanos_to_micros	-
Conversions: Time units	nanos_to_millis	nanos_to_millis
Conversions: Time units	nanos_to_seconds	-
Conversions: Time units	seconds_to_micros	-
Conversions: Time units	seconds_to_millis	-
Conversions: Time units	seconds_to_nanos	-
Format	format_date	format_date
Format	format_datetime	format
Format	format_duration_nanos	format_nanos
Parse	parse_duration	to_period
Parse	parse_duration_nanos	to_nanos
Parse	parse_epoch_nanos	-
Parse	parse_instant	convertDateTime
Parse	parse_local_date
Parse	parse_local_time	-
Parse	parse_period	to_period
Parse	parse_time_precision	getFinestDefinedUnit
Parse	parse_time_zone	-
Parse	parse_ztd	-
Parse	-	convertTime
Parse	-	convertTimeQuiet
Table operation	time_table	time_table
Time Zone	time_zone	-
Time Zone	time_zone_alias_add	-
Time Zone	time_zone_alias_rm	-
Time Zone	-	TimeZone

Type Changes

Deephaven date-time types (DateTime, TimeZone, Period) have been replaced with standard Java date-time types (Instant, LocalDate, LocalTime, ZoneId, ZonedDateTime, Period, Duration).

In most cases, type changes will require few or no code changes for Python and Groovy users. If your code directly interacts with one of the old date-time types, it will need to be ported to the new types.

In the new library, the key types are:

• Instant - An instant in time. This is analogous to DateTime in the old date-time library.
• LocalDate - A date with no time zone. No analog in the old date-time library.
• LocalTime - A time with no date or time zone. No analog in the old date-time library.
• ZoneId - A time zone identifier. This is analogous to TimeZone in the old date-time library.
• ZonedDateTime - An instant with a time zone. No analog in the old date-time library.
• Duration - A time period expressed in clock time (e.g., 1 second). This is analogous to Period in the old date-time library.
• Period - A time period expressed in calendar time (e.g., 1 month). This is analogous to Period in the old date-time library.

Note that the old Period has been replaced by both Duration and Period. Durations are elapsed clock time (hours, minutes, seconds, nanoseconds), while Periods are elapsed calendar time (days, weeks, months, years). If you are using an old Period that contains both clock time and calendar time, it will need to be broken into both a period and a duration. For example:

from deephaven.time import parse_period, parse_duration

print("original period: 1YT21H5M")

period = parse_period("P1Y")
duration = parse_duration("PT21H5M")
print(period)
print(duration)

If you are using an old Period to express clock time, that Period will need to be converted to a Duration.

tip

In old Period strings, the T character divided what is now a Period and Duration.

Literal changes

As part of the transition to ISO formats, the literal values of some types have changed.

• Existing DateTime literal formats can still be parsed as Instant or ZonedDateTime. You will need to change the name of the parsing method.
  • Python:
    • OLD: to_datetime("2021-01-23T12:34 MT")
    • NEW: parse_instant("2021-01-23T12:34 MT")
  • Java / Groovy:
    • OLD: convertDateTime("2021-01-23T12:34 MT")
    • NEW: parseInstant("2021-01-23T12:34 MT")
• Old Period now becomes Duration and/or Period. Formatting has changed to be ISO-compliant. Details on the new literal formats can be found in the Oracle docs for Duration and Period. In most cases, you will simply need to select between Duration and Period and then place the appropriate "P" or "PT" prefix at the front of the literal string.
  • Python:
    • Duration
      • OLD: to_period("T12h34m")
      • NEW: parse_duration("PT12h34m")
    • Period
      • OLD: to_period("1y2d")
      • NEW: parse_period("P1y2d")
  • Java / Groovy:
    • Duration
      • OLD: convertPeriod("T12h34m")
      • NEW: parseDuration("PT12h34m")
    • Period
      • OLD: convertPeriod("1y2d")
      • NEW: parsePeriod("P1y2d")
• Nanosecond duration literals now start with PT to be consistent with other durations.
  • OLD: 2:15:23
  • NEW: PT2:15:23
• Nanosecond duration literals no longer support days.
  • OLD: "2DT01:02:13"
  • NEW: "PT49:02:13"
• Local time literals follow the ISO standard and no longer need an L prefix. - OLD: 'L12:43:15' - NEW: '12:43:15'

Time zones

Time zone handling has been changed to be more flexible and customizable.

Full-length time zone names are now valid; e.g., "2021-04-23T11:23 America/New_York".

Hard-coded time zones (e.g., TZ_NY) have been removed. To access a time zone, use the time_zone(name) or timeZone(name) functions in Python and Java, respectively.

• Python:
  • OLD: tz = TZ_NY
  • NEW: tz = time_zone(“America/New_York”)
• Java / Groovy:
  • OLD: tz = TZ_NY
  • NEW: tz = timeZone(“America/New_York”)

Time zone aliases have replaced a specified list of hard-coded time zones (e.g., NY). Some of the most common aliases are provided in the default configuration. If you are using any of the existing short time zone names not in the default configuration, you will need to either (1) change to full-length time zone names or (2) add the short name to the alias list. To specify an alias list, use the property timezone.aliases=/path_to_your_file.csv. The alias file is a CSV where the first column is the alias, and the second column is the full-length time zone name.

Default aliases can be found at in our GitHub repo.

The default time zone is used for formatting output when no time zone is provided. The default is now configurable via the user.timezone=<full_length_timezone> property. The default is set to Eastern Standard Time (America/New_York).