Repository navigation
Conversation
RyanPCo
force-pushed
the
ryanco/lance-migrate-pilot
branch
from
October 23, 2026 19:36
66443e0 to
de074f0
Compare
RyanPCo
force-pushed
the
ryanco/lance-annotations-stage
branch
from
October 23, 2026 19:36
879a5a7 to
6c62f33
Compare
The schema is moving to `lab` = collecting lab (old `site`), a required
`end_effector` column, and steps tables keyed steps/{lab}/{embodiment}/{end_effector}.lance.
The pilot roots were written with lab = source format and steps/{aria,eva}.lance,
so they need a one-off copy into the new layout before the new readers can use them.
The script merges one or more old roots (--src, repeatable) into a new root, never in
place: it splits every old steps table by its recordings' new key in source row
order, unions tasks/aliases/annotations/changes, fails on recording_id or alias
episode_hash collisions across sources, drops the legacy tasks.annotations column
(dry-run counts the non-empty rows), copies videos and other files as is (video_path
is root-relative), maps changes rows to the new key in one function, refuses null
sites unless --lab-for-null-site is given, and verifies row counts and recording
placement after writing. Lance history is not carried over; new tables are tagged
instead. NEW_RECORDINGS_COLUMNS must match schema.RECORDINGS once that commit lands.
Claude-Session: https://claude.ai/code/session_01HjCQBqQCr2igFEMR7yUovw
The schema commit landed, so the migration takes the recordings columns from schema.RECORDINGS (null content_digest for roots that predate it), end_effector values from END_EFFECTOR and table paths from steps_path instead of local copies. Changes rows follow the landed design: one row per old row, the old `lab` becomes `source`, and `steps_tables` lists the steps tables its recordings live in ([] for annotation commits). Claude-Session: https://claude.ai/code/session_01HjCQBqQCr2igFEMR7yUovw
The aria and eva pilot roots were merged into /coc/flash7/scratch/lance_pilot/lance_v2 with the new lab/end_effector schema and per-key steps tables; the old roots use the pre-migration schema the readers no longer accept. Claude-Session: https://claude.ai/code/session_01HjCQBqQCr2igFEMR7yUovw
The old roots keep the pre-migration schema, which the stack no longer reads. Claude-Session: https://claude.ai/code/session_01HjCQBqQCr2igFEMR7yUovw
RyanPCo
force-pushed
the
ryanco/lance-annotations-stage
branch
from
October 23, 2026 20:03
6c62f33 to
5c415cf
Compare
RyanPCo
force-pushed
the
ryanco/lance-migrate-pilot
branch
from
October 23, 2026 20:03
de074f0 to
546c4fa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The pilot roots were written with
lab= source format ("aria"/"eva"),site=collecting lab, and steps/{aria,eva}.lance. The tables now have one
labcolumn (thecollecting lab), a required
end_effector, and steps tables atsteps/{lab}/{embodiment}/{end_effector}.lance, so the pilot roots need a one-off copy
into the new layout before the new readers can use them.
egomimic/scripts/lance_pilot/migrate_lab_end_effector.py merges one or more old roots
(--src, repeatable) into a new root (--dst), never in place:
sitebecomeslab, end_effector comes from the oldlab(aria ->hand, eva -> gripper). Null sites fail unless --lab-for-null-site is given.
end_effector) in source row order. Steps rows of recordings missing from
recordings.lance need --drop-orphan-steps.
alias episode_hash found twice fails the run. recording_id is unchanged (its hash
input stays "aria/").
labbecomessource, andsteps_tableslists the steps tableseach batch's recordings now live in ([] for annotation commits).
one.
Lance history is not carried over: every new table starts at version 1 and is tagged
--tag (default migrate-lab-end-effector). Old tags are listed and dropped; changes.tag
keeps naming them.
Run on the two pilot roots (/coc/flash7/scratch/lance_pilot/lance and lance_eva) into
/coc/flash7/scratch/lance_pilot/lance_v2 (5.5G); the source roots are unchanged:
same as the old steps/aria.lance and steps/eva.lance.
The branch also points aria_pilot_lance.yaml, eva_pilot_lance.yaml and the lance_pilot
bench/check scripts at lance_v2; the old roots keep the old schema, which the stack no
longer reads.
Testing: on a compute node, test_lance_migrate_lab_end_effector, test_lance_commit and the
pilot cases of test_data_configs_compose pass (25 passed). LanceStreamingResolver reads
frames from lance_v2 for 34 aria human_bimanual and 49 eva_bimanual recordings.
https://claude.ai/code/session_01HjCQBqQCr2igFEMR7yUovw