Skip to content

Fix non-UTC ORC timestamp read correctness - #15450

Draft
res-life wants to merge 2 commits into
NVIDIA:release/26.08from
res-life:fix-orc-non-utc-correctness
Draft

Fix non-UTC ORC timestamp read correctness#15450
res-life wants to merge 2 commits into
NVIDIA:release/26.08from
res-life:fix-orc-non-utc-correctness

Conversation

@res-life

@res-life res-life commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Fixes #15449.

Depends on NVIDIA/cudf-spark-jni#4917.

Description

GPU ORC reads could return incorrect timestamps in non-UTC timezones for both
physical timestamp columns and schema-evolution casts. Symptoms included
one-hour DST shifts, NULL for valid double-to-timestamp values, and historical
offset mismatches before a timezone's first recorded transition.

Spark delegates these conversions to Apache ORC. ORC uses
java.util.TimeZone rules from the writer footer and reader timezone, applies
timezone conversion before millisecond rounding and overflow handling, and
skips cross-timezone conversion when writer and reader have the same rules.
The GPU path differed in all three areas: it compared timezone identities
instead of rules, performed parts of schema-evolution rounding/overflow before
timezone conversion, and used transition data that did not reproduce
java.util.TimeZone behavior before the first recorded transition.

This change aligns the GPU path with Spark/ORC and adds deterministic regression
coverage in three layers:

  • a small boundary corpus across canonical, alias, fixed-offset, DST, and
    non-DST timezone pairs, including Europe/Paris;
  • a table-driven timestamp, bigint, float, and double to timestamp
    schema-evolution matrix;
  • pinned historical, DST transition -1us/at/+1us, integer 514952012,
    zero, rounding, and positive/negative overflow values.

The previous broad randomized tests sampled a large date range but were very
unlikely to hit exact DST boundaries, pre-first-transition values, zero and
overflow behavior, or the relevant same-zone/alias/directional combinations.

Validation

  • A/B with the original two fixed seeds:
    • baseline: all 7 reported cases reproduced (3 failed in
      Asia/Shanghai, 4 failed in America/New_York);
    • fixed build: the same 7 cases passed.
  • OrcTimezoneSuite: 46 succeeded, 0 failed.
  • Targeted Python regressions:
    • America/New_York: 3 passed;
    • Asia/Shanghai: 3 passed.
  • Spark 3.4.0 / Scala 2.12 package: BUILD SUCCESS.

Checklists

Documentation

  • Updated for new or modified user-facing features or behaviors
  • No user-facing change

Testing

  • Added or modified tests to cover new code paths
  • Covered by existing tests
  • Not required

Performance

  • Tests ran and results are added in the PR description
  • Issue filed with a link in the PR description
  • Not required

Match Apache ORC timezone, rounding, and historical offset semantics for physical timestamps and schema-evolution casts.

Fixes NVIDIA#15449.

Signed-off-by: Chong Gao <chongg@nvidia.com>
@res-life
res-life force-pushed the fix-orc-non-utc-correctness branch from 0baee14 to d9dec29 Compare July 31, 2026 00:45
@res-life
res-life changed the base branch from main to release/26.08 July 31, 2026 00:45
Signed-off-by: Chong Gao <chongg@nvidia.com>
@res-life res-life changed the title Fix non-UTC ORC timestamp reads [databricks] Fix non-UTC ORC timestamp read correctness Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] ORC timestamp reads produce incorrect results in non-UTC DST timezones

2 participants