Repository navigation
Timestamps: read ISO 8601 strings into timestamp columns, write them back - #3
Merged
Merged
Conversation
…back A timestamp field (type::timestamp, make_timestamp; Arrow formats tss:, tsm:, tsu: and tsn:, with or without a time zone) reads ISO 8601 (RFC 3339) strings: a date, then optionally a time with up to 9 digits of fraction, then optionally Z or an offset; a time without either is UTC. An integer is a count of units. The writer writes timestamps in ISO 8601 (Z with a time zone), and also date32 and date64 columns. The builders (make_field, ...) no longer use aggregate initialization, which -Wmissing-field-initializers would flag with the two new members of field.
… in Arrow; the hour alone is a time Arrow's JSON reader rejects 2015-01-01T15:00:00.1234 in milliseconds rather than drop digits; we do the same. It accepts 2015-01-01T15 too.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A
timestampfield reads ISO 8601 (RFC 3339) strings, such as thecreated_atof GH Archive events, into an Arrow timestamp column, and the writer writes timestamps back as ISO 8601 strings.type::timestamp, with aunit(seconds to nanoseconds) and atimezone(empty for none), imported from and exported to the Arrow formatstss:,tsm:,tsu:andtsn:, with or without a time zone.2015-01-01), then optionallyT(ort, or a space) and a time (15,15:00,15:00:00, or15:00:00.123456with.or,), then optionallyZor an offset (+05:00,+0500,+05). A time withoutZor an offset is UTC. An integer is a count of units. Anything else is an error: an impossible date (2015-02-29,24:00), a fraction with more digits than the unit (as Arrow's JSON reader does: no silent loss of precision), or a value out of the range of the unit. The message names the value:invalid timestamp "2015-02-30" (row 0, field /t)."2015-01-01T15:00:00", withZwhen the timestamp has a time zone, and the fraction (3, 6 or 9 digits, by unit) when it is not zero. The writer also writesdate32anddate64columns, as"2015-01-01".The parsing and formatting are in
src/timestamp.h(the date arithmetic follows Howard Hinnant'sdays_from_civil), which the amalgamation now includes.Results
From Python (fastsimdjson with this branch vendored),
loads_arrowreturns the same table aspyarrow.json.read_jsonfor the four units, with and without a time zone, on GH Archive events and on 2,000 random timestamps per unit (offsets, fractions, the date alone), and both reject the same invalid ones.GH Archive (886 MB), the
ghschema withcreated_atas a string, then astimestamp[us, UTC], GB/s, Xeon Gold 6548N, best of 3:bench_ndjson, GCC 16, one threadbench_ndjson, clang 21, one threadloads_arrow(manylinux wheel, Python 3.14), one threadpyarrow.json.read_json, one threadpolars.read_ndjson, one threadReading the timestamps costs next to nothing; with them,
loads_arrowis 4.7 times as fast as pyarrow and 10 times as fast as polars on one thread.benchmarks/schemas.hhas agh_tsschema (bench_ndjson gharchive.ndjson gh_ts).Tests
New tests: the parser (valid and invalid timestamps, offsets, fractions, leap years, the limits of nanoseconds), the formatter, 200,000 random round trips in every unit, reading (strings, integers, nulls, error messages), schema import and export, writing (with and without a time zone), and date32 and date64. They pass with Apple clang (also with ASan and UBSan), and with GCC 16 and clang 21 on Linux, without new warnings under
-Wall -Wextra.