Repository navigation
Conversation
RyanPCo
force-pushed
the
ryanco/lance-s3-committer
branch
2 times, most recently
from
October 23, 2026 19:28
5950ffc to
f4da635
Compare
RyanPCo
force-pushed
the
ryanco/lance-mp4-ranges
branch
from
October 23, 2026 19:36
07c145d to
12ee83c
Compare
RyanPCo
force-pushed
the
ryanco/lance-s3-committer
branch
from
October 23, 2026 19:36
f4da635 to
13c02e0
Compare
The committer (egomimic/scripts/lance_commit.py, moved from lance_pilot/commit.py) is the only writer of the Lance tables. Converters stage deliveries under <staging>/<lab>/<vendor id>/ with meta.json uploaded last (commit.stage; --stage-to on aria_to_lance, eva_to_lance and the Ray driver, which no longer commits). Each run validates complete deliveries, commits them in batches, records each batch in a new changes table and tags every table <lab>-<UTC time>, then deletes their staged files. A delivery that fails validation gets REJECTED.json with its errors and a digest of its files and is skipped until they change; one already committed waits for --replace. commit.py works on local paths and s3:// roots alike (R2 credentials from the environment). Frame counts come from the MP4 index, so the committer no longer needs torchcodec. Six concurrent appends on R2 all land with the expected version count (tests/integration/test_lance_r2). Claude-Session: https://claude.ai/code/session_01Kt1E8T2x8kdV9akAvbJyVd
…bles
Staging prefixes, tag prefixes and the committer's --lab named the converter
(aria, eva), which is no longer what lab means in the tables. They are now
--source / <staging>/<source>/. The changes table drops its lab column for
source (the converter or "annotations") and steps_tables, the
steps/{lab}/{embodiment}/{end_effector}.lance tables the batch wrote, since one
batch can span several. Staged deliveries carry lab and end_effector.
Claude-Session: https://claude.ai/code/session_01HjCQBqQCr2igFEMR7yUovw
RyanPCo
force-pushed
the
ryanco/lance-mp4-ranges
branch
from
October 23, 2026 20:03
12ee83c to
ffa99f1
Compare
RyanPCo
force-pushed
the
ryanco/lance-s3-committer
branch
from
October 23, 2026 20:03
13c02e0 to
9928b36
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The committer (egomimic/scripts/lance_commit.py, moved from// with-, then deletes their staged files. A
lance_pilot/commit.py) is the only writer of the Lance tables.
Converters stage deliveries under /
meta.json uploaded last (commit.stage; --stage-to on aria_to_lance,
eva_to_lance and the Ray driver, which no longer commits). Each run
(lance_commit --source aria|eva) validates complete deliveries, commits
them in batches, records each batch in a new changes table and tags
every table
delivery that fails validation gets REJECTED.json with its errors and a
digest of its files and is skipped until they change; one already
committed waits for --replace.
A changes row names the converter in
sourceand lists the steps tablesthe batch wrote in
steps_tables, since one batch can span several(lab, embodiment, end_effector) tables.
commit.py works on local paths and s3:// roots alike (R2 credentials
from the environment). Frame counts come from the MP4 index, so the
committer no longer needs torchcodec. Six concurrent appends on R2 all
land with the expected version count (tests/integration/test_lance_r2).
https://claude.ai/code/session_01HjCQBqQCr2igFEMR7yUovw