Skip to content

Timestamps: read ISO 8601 strings into timestamp columns, write them back - #3

Merged
lemire merged 3 commits into
mainfrom
timestamps
Nov 12, 2026
Merged

lemire merged 3 commits into
mainfrom
timestamps

Conversation

@lemire

@lemire lemire commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

A timestamp field reads ISO 8601 (RFC 3339) strings, such as the created_at of GH Archive events, into an Arrow timestamp column, and the writer writes timestamps back as ISO 8601 strings.

make_timestamp("created_at", time_unit::microsecond, "UTC")  // Arrow format "tsu:UTC"
  • Types: type::timestamp, with a unit (seconds to nanoseconds) and a timezone (empty for none), imported from and exported to the Arrow formats tss:, tsm:, tsu: and tsn:, with or without a time zone.
  • Reading: a date (2015-01-01), then optionally T (or t, or a space) and a time (15, 15:00, 15:00:00, or 15:00:00.123456 with . or ,), then optionally Z or an offset (+05:00, +0500, +05). A time without Z or an offset is UTC. An integer is a count of units. Anything else is an error: an impossible date (2015-02-29, 24:00), a fraction with more digits than the unit (as Arrow's JSON reader does: no silent loss of precision), or a value out of the range of the unit. The message names the value: invalid timestamp "2015-02-30" (row 0, field /t).
  • Writing: "2015-01-01T15:00:00", with Z when the timestamp has a time zone, and the fraction (3, 6 or 9 digits, by unit) when it is not zero. The writer also writes date32 and date64 columns, as "2015-01-01".
  • Not changed: schema inference never infers timestamps (strings stay utf8).

The parsing and formatting are in src/timestamp.h (the date arithmetic follows Howard Hinnant's days_from_civil), which the amalgamation now includes.

Results

From Python (fastsimdjson with this branch vendored), loads_arrow returns the same table as pyarrow.json.read_json for the four units, with and without a time zone, on GH Archive events and on 2,000 random timestamps per unit (offsets, fractions, the date alone), and both reject the same invalid ones.

GH Archive (886 MB), the gh schema with created_at as a string, then as timestamp[us, UTC], GB/s, Xeon Gold 6548N, best of 3:

string timestamp
bench_ndjson, GCC 16, one thread 1.986 1.982
bench_ndjson, clang 21, one thread 1.935 1.941
fastsimdjson loads_arrow (manylinux wheel, Python 3.14), one thread 2.11 2.11
the same, 32 threads 29.7 28.3
pyarrow.json.read_json, one thread 0.45 0.45
polars.read_ndjson, one thread 0.55 0.21

Reading the timestamps costs next to nothing; with them, loads_arrow is 4.7 times as fast as pyarrow and 10 times as fast as polars on one thread.

benchmarks/schemas.h has a gh_ts schema (bench_ndjson gharchive.ndjson gh_ts).

Tests

New tests: the parser (valid and invalid timestamps, offsets, fractions, leap years, the limits of nanoseconds), the formatter, 200,000 random round trips in every unit, reading (strings, integers, nulls, error messages), schema import and export, writing (with and without a time zone), and date32 and date64. They pass with Apple clang (also with ASan and UBSan), and with GCC 16 and clang 21 on Linux, without new warnings under -Wall -Wextra.

…back

A timestamp field (type::timestamp, make_timestamp; Arrow formats tss:, tsm:,
tsu: and tsn:, with or without a time zone) reads ISO 8601 (RFC 3339)
strings: a date, then optionally a time with up to 9 digits of fraction, then
optionally Z or an offset; a time without either is UTC. An integer is a
count of units. The writer writes timestamps in ISO 8601 (Z with a time
zone), and also date32 and date64 columns.

The builders (make_field, ...) no longer use aggregate initialization, which
-Wmissing-field-initializers would flag with the two new members of field.
… in Arrow; the hour alone is a time

Arrow's JSON reader rejects 2015-01-01T15:00:00.1234 in milliseconds rather
than drop digits; we do the same. It accepts 2015-01-01T15 too.
@lemire
lemire merged commit f012628 into main Nov 12, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant