Skip to content

v2: split timestamps are days, recorded as dates in the manifest - #168

Merged
kvignesh1420 merged 1 commit into
v2from
v2-date-splits
Nov 16, 2026
Merged

kvignesh1420 merged 1 commit into
v2from
v2-date-splits

Conversation

@kvignesh1420

Copy link
Copy Markdown
Collaborator

Running the RT README's preprocessing on a v2 database, on the ILC cluster with relational-transformer 1.11.0, panicked:

failed to parse val_timestamp "2018-01-09 09:45:04.423059456": trailing input

relational-transformer's preprocessor (rustler/src/pre.rs) reads val_timestamp as a %Y-%m-%d date. v2 wrote the exact quantile time, which is valid relbench but not readable by relational-transformer. The published stanford-star/plurel corpus records dates, for example val_timestamp: '2017-12-27'.

  • split_timestamps returns the midnight before each quantile. The split moves by less than a day.
  • write_database records a midnight as a date, any other time in full, and a static database's NaT as before.

Draws and fingerprints do not move. Lint, pyright, 362 tests, README blocks and recipes pass. The new test checks the manifest records both split days as dates.

Checked end to end on the cluster at 45c6ebc: scripts/synthetic_gen.py wrote plurel-0 (20 tables, 112,484 rows) with val_timestamp as a date. The RT README's one(...) block, run verbatim, then completed:

  • rustler graph and column stats up to val_timestamp;
  • 224 texts embedded with all-MiniLM-L12-v2;
  • all 8 output files written (meta.json, nodes.rkyv, offsets.rkyv, p2f_adj.rkyv, …).

Relational Transformer's preprocessor reads val_timestamp as a %Y-%m-%d date and panicked on
the full quantile times v2 wrote. split_timestamps now returns the midnight before each
quantile, and the manifest records a midnight as a date, as the published corpus does.
@kvignesh1420
kvignesh1420 merged commit 90458b8 into v2 Nov 16, 2026
5 checks passed
@kvignesh1420
kvignesh1420 deleted the v2-date-splits branch November 16, 2026 23:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant