Skip to content

Return tz-aware datetimes for DateTime/DateTime64 columns with a timezone - #146

Open
Maksim-Burtsev wants to merge 2 commits into
maximdanilchenko:masterfrom
Maksim-Burtsev:datetime-tz
Open

Return tz-aware datetimes for DateTime/DateTime64 columns with a timezone#146
Maksim-Burtsev wants to merge 2 commits into
maximdanilchenko:masterfrom
Maksim-Burtsev:datetime-tz

Conversation

@Maksim-Burtsev

Copy link
Copy Markdown
Contributor

Part of #136 (the "DateTime64 with timezone" item, tail end of #91 / #119).

Problem

The client already parses the timezone out of a column type like DateTime64(3, 'Europe/Moscow'), but then throws it away. So for the same column:

# SELECT toDateTime64('2019-01-01 00:00:00.123', 3, 'Europe/Moscow')
tsv / binary  -> datetime(2019, 1, 1, 0, 0, 0, 123000)     # naive, Moscow wall-clock
native        -> datetime(2018, 12, 31, 21, 0, 0, 123000)  # naive, UTC

Three engines, two different answers, and none of them tells you which zone the time is in.

Writing was worse: inserting a tz-aware datetime through the binary/native engines either re-zoned it silently (value.replace(tzinfo=column_zone), so a UTC value landed 3 hours off in a Moscow column) or failed with TypeError: can't subtract offset-naive and offset-aware datetimes for a column without a zone.

Change

One rule, applied to every engine:

  • a column with a timezone decodes to a tz-aware datetime in that zone (zoneinfo.ZoneInfo);
  • a column without one stays naive — nothing changes for existing users of plain DateTime/DateTime64.
# after, all three engines:
datetime(2019, 1, 1, 0, 0, 0, 123000, tzinfo=ZoneInfo('Europe/Moscow'))

On INSERT a naive datetime is still taken as wall-clock time in the column's zone (as before), and an aware datetime of any zone is stored as the instant it denotes. Works on all three engines — TSV too, since ClickHouse accepts the +HH:MM offset that str(datetime) produces (the compiled unconvert already did this; the pure-Python one now matches).

The naive/aware epoch arithmetic that was duplicated across four write branches is now a single to_epoch_micros helper (in both types.py and _types.pyx).

One server quirk

The Native format header ships a DateTime('TZ') column as plain DateTime (checked on a real table with CH 26.7: DateTime64(3, 'Europe/Moscow') comes through verbatim, DateTime('Europe/Moscow') comes through as DateTime). There is nothing to decode the zone from, so that one column stays naive UTC on native=True. Documented in the README; DateTime64 is correct everywhere.

Breaking?

Only for columns declared with a timezone: they now return aware datetimes instead of naive wall-clock ones. Comparing them to naive values raises/returns False, so callers doing that will notice. Timezone-less columns are untouched.

Tests

  • TestDateTimeTimezone.test_round_trip: aware and naive values written through each of TSV / RowBinary / Native into DateTime('TZ'), DateTime64('TZ'), their tz-less twins, Nullable(DateTime64('UTC')) and Array(DateTime64(6,'UTC')); the server-side unix timestamps are asserted, then the rows are read back through all three engines.
  • NATIVE_UNSUPPORTED = {"datetime64"} in the cross-engine type matrix is gone — the native engine now agrees with the others.
  • Suite is green in all four CI configurations (plain / ciso8601 / cython / cython+ciso8601).

Maksim-Burtsev and others added 2 commits August 22, 2026 00:38
ClickHouse 26.7 made `explain_query_plan_default = 'pretty'` the default,
and the pretty plan carries a blank line between the output columns and
the plan tree. A blank TSV line decodes to an empty Record (that is what
the `WITH TOTALS` separator looks like), so indexing every row of an
EXPLAIN result stopped working.

EXPLAIN SYNTAX also renders operators as function calls now
("SELECT plus(1, 1)"), so the test no longer asserts the exact spelling.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A DateTime('TZ') / DateTime64(P, 'TZ') column now decodes to a datetime
carrying that zone (zoneinfo) on every engine; a column without a zone
stays naive. Writers accept naive values (wall-clock in the column zone,
as before) and aware values of any zone (stored as the instant they
denote) — previously an aware value was silently re-zoned or raised
TypeError.

One server quirk remains: the Native header ships DateTime('TZ') as plain
DateTime (DateTime64 keeps its zone), so that column is naive UTC on the
native engine only.

Closes the DateTime64-with-timezone item of maximdanilchenko#136.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant