Return tz-aware datetimes for DateTime/DateTime64 columns with a timezone - #146
Open
Maksim-Burtsev wants to merge 2 commits into
Open
Return tz-aware datetimes for DateTime/DateTime64 columns with a timezone#146Maksim-Burtsev wants to merge 2 commits into
Maksim-Burtsev wants to merge 2 commits into
Conversation
ClickHouse 26.7 made `explain_query_plan_default = 'pretty'` the default,
and the pretty plan carries a blank line between the output columns and
the plan tree. A blank TSV line decodes to an empty Record (that is what
the `WITH TOTALS` separator looks like), so indexing every row of an
EXPLAIN result stopped working.
EXPLAIN SYNTAX also renders operators as function calls now
("SELECT plus(1, 1)"), so the test no longer asserts the exact spelling.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A DateTime('TZ') / DateTime64(P, 'TZ') column now decodes to a datetime
carrying that zone (zoneinfo) on every engine; a column without a zone
stays naive. Writers accept naive values (wall-clock in the column zone,
as before) and aware values of any zone (stored as the instant they
denote) — previously an aware value was silently re-zoned or raised
TypeError.
One server quirk remains: the Native header ships DateTime('TZ') as plain
DateTime (DateTime64 keeps its zone), so that column is naive UTC on the
native engine only.
Closes the DateTime64-with-timezone item of maximdanilchenko#136.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #136 (the "DateTime64 with timezone" item, tail end of #91 / #119).
Problem
The client already parses the timezone out of a column type like
DateTime64(3, 'Europe/Moscow'), but then throws it away. So for the same column:Three engines, two different answers, and none of them tells you which zone the time is in.
Writing was worse: inserting a tz-aware
datetimethrough the binary/native engines either re-zoned it silently (value.replace(tzinfo=column_zone), so a UTC value landed 3 hours off in a Moscow column) or failed withTypeError: can't subtract offset-naive and offset-aware datetimesfor a column without a zone.Change
One rule, applied to every engine:
datetimein that zone (zoneinfo.ZoneInfo);DateTime/DateTime64.On INSERT a naive
datetimeis still taken as wall-clock time in the column's zone (as before), and an awaredatetimeof any zone is stored as the instant it denotes. Works on all three engines — TSV too, since ClickHouse accepts the+HH:MMoffset thatstr(datetime)produces (the compiledunconvertalready did this; the pure-Python one now matches).The naive/aware epoch arithmetic that was duplicated across four
writebranches is now a singleto_epoch_microshelper (in bothtypes.pyand_types.pyx).One server quirk
The Native format header ships a
DateTime('TZ')column as plainDateTime(checked on a real table with CH 26.7:DateTime64(3, 'Europe/Moscow')comes through verbatim,DateTime('Europe/Moscow')comes through asDateTime). There is nothing to decode the zone from, so that one column stays naive UTC onnative=True. Documented in the README; DateTime64 is correct everywhere.Breaking?
Only for columns declared with a timezone: they now return aware datetimes instead of naive wall-clock ones. Comparing them to naive values raises/returns False, so callers doing that will notice. Timezone-less columns are untouched.
Tests
TestDateTimeTimezone.test_round_trip: aware and naive values written through each of TSV / RowBinary / Native intoDateTime('TZ'),DateTime64('TZ'), their tz-less twins,Nullable(DateTime64('UTC'))andArray(DateTime64(6,'UTC')); the server-side unix timestamps are asserted, then the rows are read back through all three engines.NATIVE_UNSUPPORTED = {"datetime64"}in the cross-engine type matrix is gone — the native engine now agrees with the others.