[FLINK-40491][core] Extend BinaryVariant with TIME, TIMESTAMP_NS, TIMESTAMP_LTZ_NS primitives by manner · Pull Request #29050 · apache/flink

manner · 2026-08-31T13:04:58Z

What is the purpose of the change

The variant binary encoding spec (see Variant Encoding) defines primitive type codes 17-20 for TIME, TIMESTAMP_LTZ_NS, TIMESTAMP_NS, and UUID that are currently missing in Flink.
This PR adds support for codes 17-19 (TIME, TIMESTAMP_LTZ_NS, TIMESTAMP_NS) to BinaryVariant, so that a LocalTime and a nanosecond-precision Instant/LocalDateTime can be represented in a Variant without lossy truncation to microseconds.
UUID (code 20) is intentionally out of scope here and will be added in a separate ticket.

Brief change log

Added BinaryVariantUtil constants for primitive codes 17-19, extended getType()/valueSize()/getLong() to handle them, and added a TIME_FORMATTER for JSON rendering
Added Variant.Type.TIME/TIMESTAMP_NS/TIMESTAMP_LTZ_NS and the corresponding getTime()/getDateTimeNanos()/getInstantNanos() accessors to the Variant interface
Added VariantBuilder.of(LocalTime); made of(Instant)/of(LocalDateTime) precision-aware so a value with no sub-microsecond component keeps using the existing compact micros encoding, and only switches to the new nanosecond encoding when the value actually needs it
Implemented appendTime/appendTimestampNanos/appendTimestampLtzNanos in BinaryVariantInternalBuilder, and the matching read/get()/toJson() support in BinaryVariant

Verifying this change

This change added tests and can be verified as follows:

Round-trip and generic get() dispatch tests for the new types (BinaryVariantTest#testScalarVariant)
Dedicated precision-dispatch tests verifying Instant/LocalDateTime pick the existing TIMESTAMP_LTZ/TIMESTAMP encoding for microsecond-aligned values and the new TIMESTAMP_LTZ_NS/TIMESTAMP_NS encoding otherwise, including that the mismatched accessor throws VariantTypeException (BinaryVariantTest#testNanosecondPrecisionVariant)
A test verifying that LocalTime silently truncates below microsecond precision, since TIME has no nanosecond-precision counterpart in the variant spec (BinaryVariantTest#testTimeSubMicrosecondTruncation)
JSON rendering assertions for TIME/TIMESTAMP_NS/TIMESTAMP_LTZ_NS (BinaryVariantTest#testToJsonScalar)

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): yes (Variant and VariantBuilder are @PublicEvolving; this adds new enum constants and new interface methods)
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented? not documented (documentation is tracked separately in FLINK-40494)

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

Generated-by: Claude Sonnet 5 (Claude Code)

…ESTAMP_LTZ_NS primitives

flinkbot · 2026-08-31T13:10:21Z

CI report:

c7dc74a Azure: PENDING
058498a Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

raminqaf

Thanks for the PR @manner! Left two comments

snuyanzin · 2026-08-31T19:21:57Z

            case DATE:
                return readLong(value, pos + 1, 4);
            case INT8:
+            case TIME:


is it TIME with micro/nanos or not?

TIME is using micros:

https://parquet.apache.org/docs/file-format/types/variantencoding/

snuyanzin · 2026-08-31T19:23:25Z

+            case TIMESTAMP_LTZ_NS:
+            case TIMESTAMP_NS:


why do we think 8 bytes is enough here?

The Variant specification states 8 bytes for nanosecond timestamps as well. The downside for using the same 8 bytes for the higher precision values is the smaller range of timestamps that can be used (+/- 292 years around unix epoch)

https://parquet.apache.org/docs/file-format/types/variantencoding/

ok, thanks for clarification
I tend to think we need to explicitly mention such limitation in docs

and by the way what will happen with timestamp_NS -300 years?
will it fail (user friendly message?)
or produce some wrong result?

You're right, this should be properly documented. I think documentation will mostly happen in this ticket:
https://issues.apache.org/jira/browse/FLINK-40494

When converting goes wrong it will now throw a VariantTypeException with a helpful error message:
https://github.com/apache/flink/pull/29050/changes#diff-2a1f60d1bf5ca4585c1c2d08c51866702d17d70306ee16ce096b238377b82770R137-R148

There's a test for this here:
https://github.com/apache/flink/pull/29050/changes#diff-662023392a0949caa3a0814536613caf4fb2a17c1e6f3d7217e829bee0d50390R142-R154

raminqaf · 2026-09-01T09:56:06Z

    // Get a long value from variant value `value[pos...]`.
-    // It is only legal to call it if `getType` returns one of `Type.LONG/DATE/TIMESTAMP/
-    // TIMESTAMP_LTZ`. If the type is `DATE`, the return value is guaranteed to fit into an int and
-    // represents the number of days from the Unix epoch.
+    // It is only legal to call it if `getType` returns one of `Type.LONG/DATE/TIME/TIMESTAMP/
+    // TIMESTAMP_LTZ/TIMESTAMP_NS/TIMESTAMP_LTZ_NS`.
+    // If the type is `DATE`, the return value is
+    // guaranteed to fit into an int and represents the number of days from the Unix epoch.
+    // If the type is `TIME`, the return value represents the number of microseconds since
+    // midnight.
    // If the type is `TIMESTAMP/TIMESTAMP_LTZ`, the return value represents the number of
    // microseconds from the Unix epoch.
+    // If the type is `TIMESTAMP_NS/TIMESTAMP_LTZ_NS`, the return value represents the number of
+    // nanoseconds from the Unix epoch.


nit: Should we change this into proper JavaDocs instead of line comments?

raminqaf · 2026-09-01T09:59:54Z

        Instant instant = Instant.EPOCH;
        LocalDateTime localDateTime = LocalDateTime.of(2000, 1, 1, 0, 0);
        LocalDate localDate = LocalDate.of(2000, 1, 1);
+        LocalTime localTime = LocalTime.of(13, 45, 30, 123456000);


Let's keep the nano precision with non-zero values for the time here to be sure that the truncation works as expected

Suggested change

LocalTime localTime = LocalTime.of(13, 45, 30, 123456000);

LocalTime localTime = LocalTime.of(13, 45, 30, 123456789);

raminqaf

LGTM @manner! Thanks for the contribution! 🚀

[FLINK-40491][core] Extend BinaryVariant with TIME, TIMESTAMP_NS, TIM…

5b4a097

…ESTAMP_LTZ_NS primitives

[FLINK-40491][core] Fix formatting

c7dc74a

raminqaf reviewed Aug 31, 2026

View reviewed changes

Comment thread flink-core/src/test/java/org/apache/flink/types/variant/BinaryVariantTest.java

Comment thread flink-core/src/main/java/org/apache/flink/types/variant/VariantBuilder.java Outdated

manner added 2 commits August 31, 2026 17:47

[FLINK-40491][core] Add comment about truncating LocalTime

ffd3b20

[FLINK-40491][core] Reject out-of-range nanosecond-precision timestamps

058498a

manner requested a review from raminqaf August 31, 2026 15:53

github-actions Bot added the community-reviewed PR has been reviewed by the community. label Aug 31, 2026

snuyanzin reviewed Aug 31, 2026

View reviewed changes

raminqaf reviewed Sep 1, 2026

View reviewed changes

manner added 2 commits September 1, 2026 12:06

[FLINK-40491][core] Add nano precision to Time to check truncation

19c6ec8

[FLINK-40491][core] Refactor comments to Javadoc and fix inconsistencies

5eea899

raminqaf approved these changes Sep 1, 2026

View reviewed changes

Navigation Menu

Uh oh!

[FLINK-40491][core] Extend BinaryVariant with TIME, TIMESTAMP_NS, TIMESTAMP_LTZ_NS primitives - #29050

[FLINK-40491][core] Extend BinaryVariant with TIME, TIMESTAMP_NS, TIMESTAMP_LTZ_NS primitives#29050
manner wants to merge 6 commits into
apache:masterfrom
manner:FLINK-40491

manner commented Aug 31, 2026

Uh oh!

flinkbot commented Aug 31, 2026 •

edited

Loading

Uh oh!

raminqaf left a comment

Uh oh!

Uh oh!

Uh oh!

snuyanzin Aug 31, 2026

Uh oh!

manner Sep 1, 2026

Uh oh!

snuyanzin Aug 31, 2026 •

edited

Loading

Uh oh!

manner Sep 1, 2026

Uh oh!

snuyanzin Sep 1, 2026

Uh oh!

snuyanzin Sep 1, 2026 •

edited

Loading

Uh oh!

manner Sep 1, 2026

Uh oh!

raminqaf Sep 1, 2026

Uh oh!

raminqaf Sep 1, 2026

Uh oh!

raminqaf left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	LocalTime localTime = LocalTime.of(13, 45, 30, 123456000);
	LocalTime localTime = LocalTime.of(13, 45, 30, 123456789);

Uh oh!

Conversation

manner commented Aug 31, 2026

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Was generative AI tooling used to co-author this PR?

Uh oh!

flinkbot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

raminqaf left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

snuyanzin Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

manner Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

snuyanzin Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

manner Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

snuyanzin Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

snuyanzin Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

manner Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

raminqaf Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

raminqaf Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

raminqaf left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

flinkbot commented Aug 31, 2026 •

edited

Loading

snuyanzin Aug 31, 2026 •

edited

Loading

snuyanzin Sep 1, 2026 •

edited

Loading