zarr_to_lance: vendor and robot zarr episodes into Lance, with their annotations by RyanPCo · Pull Request #751 · GaTech-RL2/EgoVerse

RyanPCo · 2026-10-30T19:04:32Z

Converts the vendor and robot zarr episodes that have no raw converter into committer
deliveries, so the migration can move them into Lance. Aria and EVA migrate from raw
(aria_to_lance, eva_to_lance), so their zarrs are refused here; the aria/eva-zarr path and
the aria parity configs from the prototype are gone.

egomimic/scripts/zarr_to_lance.py, convert_zarr_episode(zarr, out_dir, metadata):

Formats: mecka, scale, microagi v2 (human_hand, HUMAN_STEPS), yam (EVA_STEPS) and
dexmate (DEXMATE_STEPS, tactile images as extra videos), detected from the zarr attrs
and keys. Microagi v1, trace, aria and eva fail with a clear NotImplementedError.
Steps rows are exactly the zarr frames; nothing is filled back in. Validity flags mark
sentinel values, untracked (all-zero) hands and zero robot poses (forward-filled as
eva_to_zarr does). Timestamps are obs_rgb_timestamps_ns, else the hash's start + i/fps.
JPEGs are re-encoded to AV1 at short side 360, plus a full-size front_1_<short> stream
for human front cameras; intrinsics are scaled to match.
Vendor-prefixed embodiments (MECKA_BIMANUAL, SCALE_) map onto human_ via
canonical_embodiment_name.
recording_id("zarr", episode_hash), not keyed by lab. lab, task_name and operator come
from metadata (the SQL row) over the zarr attrs and are never waived; a blank or
placeholder metadata value (required_metadata.is_placeholder) falls back to the zarr
attr; scene may be null. Without per-frame times, meta.json provenance says clock "fps".
Annotations: the zarr annotations array is where vendor language spans live (checked on
mecka, scale, microagi v2, sharpa_dex and yam episodes; SQL segments is only filled for
some mecka rows and matched the zarr). Spans are mapped onto the recording clock (frame i
is steps row i, so episodes without obs_rgb_timestamps_ns map by index; an end past the
last frame is clamped, as yam writes them) and written to meta.json annotations.

lance_commit:

--source zarr shares load_meta_deliveries with EVA; a zarr delivery may have no
capture time (abc's UUID hashes).
After each committed batch, the batch's delivery annotations are reviewed like a change
file and committed as one annotation change (annotation_vendor = lab). A span that fails
review is reported, never the recording. Lance training samples over task segments
(rldb/lance: one leaf per recording, sampled over its task segments #722), so this keeps vendor spans usable after the migration. With --replace, a
replaced recording's committed spans from the delivery's vendors that the new delivery
no longer carries are deleted.
egomimic/rldb/lance/metadata.py holds the value helpers the converters, planner and
committer share (text_or_none, placeholder, objects_list, read_manifest), and
required_metadata.UNWAIVABLE_FIELDS the always-required fields.

egomimic/scripts/lance_pilot/check_zarr_to_lance.py compares a converted episode with its
zarr: steps values on valid frames, decoded-video PSNR/SSIM, and loader samples under the
format's training keymap and transforms.

Testing: on a compute node, tests/unit/test_zarr_to_lance.py (new: a human zarr with
dropped frames and sentinels committed end to end, one-arm embodiments, required metadata,
unknown and aria zarrs refused, a round trip per vendor/robot format against the
ZarrDataset twin, robot zero-pose dropout, untracked hands, annotations committed on the
recording clock with and without per-frame times) with test_lance_committer and
test_lance_annotations: 36 passed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HjCQBqQCr2igFEMR7yUovw

RyanPCo

Automated adversarial review (read-only; findings were checked by a second skeptical pass before posting): 3 medium, 3 low, 11 nit.

RyanPCo · 2026-10-30T19:20:41Z

+SHORT_SIDE = 360
+IDENTITY_POSE = np.array([0, 0, 0, 1, 0, 0, 0], dtype=np.float32)
+REQUIRED = ("lab", "task_name", "operator")
+PLACEHOLDERS = {"", "none", "null", "nan", "unknown", "debug"}


medium (duplication): PLACEHOLDERS is a smaller copy of required_metadata._PLACEHOLDER_VALUES, so a lab, operator or task_name of 'n/a', 'tbd' or '-' gets through here even though ZarrWriter and the validators reject it. Drop PLACEHOLDERS and have _placeholder return _text(v) is None or is_placeholder(str(v)), using egomimic.rldb.zarr.required_metadata.is_placeholder.

RyanPCo · 2026-10-30T19:20:41Z

+def detect_format(attrs: dict, keys: set[str]) -> str:
+    """The FORMATS entry of the converter that wrote the episode; fails on unknown ones."""
+    emb = str(attrs.get("embodiment", "")).lower()
+    kind = emb.split("_")[0]


medium (correctness): Vendor-tagged embodiments are rejected. detect_format splits the raw name, so MECKA_BIMANUAL and SCALE_LEFT_ARM zarrs from the vendor-split registry raise NotImplementedError. ARIA_* passes detection here but then fails in steps_schema at line 446, and the raw name is also what ends up in meta["embodiment"]. Normalize once with canonical_embodiment_name(attrs["embodiment"]).lower() and use that for detection, arm layout, schema and meta. Add a test with a MECKA_BIMANUAL zarr.

RyanPCo · 2026-10-30T19:20:41Z

+    out = commit.commit_annotations(
+        lance_root,
+        rv.adds,
+        [],


medium (data-integrity): With --replace, annotations the earlier delivery committed for a recording are never removed. commit.replace rewrites steps and the catalog but not annotations.lance, and annotation_id hashes the content. If a re-conversion moves a span (a new clock, clamping or text), the old row stays beside the new one and may fall outside the replaced task bounds. For recordings in result["replaced"], delete the committed annotation_ids for that recording and annotation_vendor that are missing from the new set. If keeping them is intended, document it.

RyanPCo · 2026-10-30T19:20:41Z

+SENTINEL = 1e8
+SHORT_SIDE = 360
+IDENTITY_POSE = np.array([0, 0, 0, 1, 0, 0, 0], dtype=np.float32)
+REQUIRED = ("lab", "task_name", "operator")


low (duplication): REQUIRED here and lance_commit.REQUIRED_META are two more literal copies of the unwaivable fields, which mcap/spec.py already derives as UNWAIVABLE_FIELDS from REQUIRED_METADATA_FIELDS and WAIVABLE_FIELDS. Move UNWAIVABLE_FIELDS into rldb/zarr/required_metadata.py and import it in spec.py, zarr_to_lance and lance_commit, so a change to the rule cannot leave one copy behind.

RyanPCo · 2026-10-30T19:20:41Z

+def episode_metadata(attrs: dict, metadata: dict | None) -> dict:
+    """Zarr attrs overlaid with `metadata`; rejects missing or placeholder lab/task_name/operator."""
+    m = {k: attrs.get(k) for k in ("task_name", "task_description", "scene", "objects")}
+    over = {k: v for k, v in (metadata or {}).items() if _text(v) is not None}


low (correctness): Blank manifest cells override real zarr attrs. csv.DictReader returns '' for an empty or NULL cell, and _text('') is not None, so task='' becomes task_name='' and the episode is rejected even when the zarr has a task_name. A blank task_description likewise replaces the zarr's. Drop empty or placeholder values from the overlay, for example with if not _placeholder(v).

RyanPCo · 2026-10-30T19:20:42Z

+                worst[key] = f"shape {tuple(v.shape)} vs {tuple(w.shape)}"
+                continue
+            d = (v.float() - w.float()).abs()
+            if "images" in key:


nit (simplification): Both branches set worst[key] the same way. Write if 'images' in key: img.append(float(d.mean())) and put a single worst[key] = max(...) after it.

RyanPCo · 2026-10-30T19:20:42Z

+    ap.add_argument("--samples", type=int, default=200)
+    args = ap.parse_args()
+    (name,) = [p.name for p in args.zarr_dir.iterdir()]
+    h = name[: -len(".zarr")] if name.endswith(".zarr") else name


nit (duplication): This line repeats zarr_to_lance.episode_hash_of. Add it to the existing import and write h = episode_hash_of(name).

RyanPCo · 2026-10-30T19:20:42Z

+
+
+def _dexmate():
+    # No Dexmate embodiment class on this branch: every numeric column, untransformed.


nit (stale-comment): 'on this branch' will read wrong after merge. Reword to 'No Dexmate embodiment class yet: every numeric column, untransformed.'

RyanPCo · 2026-10-30T19:20:42Z

+    if fmt == "microagi_v2":
+        assert task["objects"] == ["iron"]
+    videos = set(dict(rec["video_path"]))
+    assert {f"{c}" for c in spec.get("cams", ("front_1",))} <= videos


nit (test-quality): {f"{c}" for c in spec.get('cams', ('front_1',))} is just set(spec.get('cams', ('front_1',))). At line 274 the identity-pose literal could be zarr_to_lance.IDENTITY_POSE, or a one-line list that black keeps on one line.

RyanPCo · 2026-10-30T19:20:42Z

+For the vendor and robot zarrs that have no raw converter (FORMATS); Aria and EVA migrate
+from raw (aria_to_lance, eva_to_lance), so their zarrs are refused. Steps rows are exactly
+the zarr frames: the zarr converters already deleted the frames they could not use
+(untracked hands, missing poses), so nothing is filled back in and a gap in `timestamp_ns`


nit (correctness): "a gap in timestamp_ns is where zarr dropped frames" is true only for zarrs with obs_rgb_timestamps_ns. For mecka, scale, yam and dexmate the clock is start + i/fps over the surviving frames, so dropped frames leave no gap and gaps is always 0. Narrow the docstring, and consider recording in meta that the clock is synthetic.

RyanPCo · 2026-10-30T19:41:54Z

Review follow-up (force-pushed). All 17 findings fixed:

Vendor-prefixed embodiments (MECKA_BIMANUAL, SCALE_, ARIA_) are normalized with canonical_embodiment_name before detection and storage (new test).
With --replace, a replaced recording's committed annotations from the delivery's vendors that the new delivery no longer carries are deleted in the same annotation change (new test). This only applies to deliveries whose meta.json has annotations, so aria/eva replaces never touch annotations.
Blank or placeholder manifest cells no longer override zarr attrs (new test).
New egomimic/rldb/lance/metadata.py (text_or_none, placeholder via required_metadata.is_placeholder, objects_list, read_manifest) replaces the copies in zarr_to_lance and lance_commit. PLACEHOLDERS, _text, _placeholder and _objects are gone.
UNWAIVABLE_FIELDS moved to required_metadata. zarr_to_lance and lance_commit use it, and REQUIRED/REQUIRED_META are dropped. spec.py no longer defines it, and test_mcap_validate imports it from required_metadata.
episode_rules.episode_hash_start_ns is new; hash_start_ns uses it and episode_hash_start_ms derives from it.
ARMS replaced by embodiment_error + arm_sides.
Dead code removed: unused fmt param (and a docstring added), unused log.
Renamed jpeg_keys, added READ_BATCH, dropped the vendor alias and provenance.episode_hash.
Clock docstring fixed. provenance records clock ("obs_rgb_timestamps_ns" or "fps").
check_zarr_to_lance: the duplicated branch merged, episode_hash_of reused, comment reworded. Head pose is compared on valid["head"], and the fps reference starts at hash_start_ns, not Lance's own ts[0].
Test: IDENTITY_POSE replaces the literal, and set(...) replaces the f-string set.

https://claude.ai/code/session_01HjCQBqQCr2igFEMR7yUovw

zarr_to_lance converts one zarr episode (mecka, scale, microagi v2, yam, dexmate) into a delivery: steps rows are exactly the zarr frames, validity flags mark sentinels, untracked hands and zero robot poses, and the JPEGs are re-encoded as AV1 MP4s. The recording id is recording_id("zarr", episode_hash). Vendor-prefixed embodiments (MECKA_BIMANUAL) map onto human_*. lab, task_name and operator come from the SQL row passed as metadata and are never waived; a blank or placeholder metadata value falls back to the zarr attr. Aria and EVA zarrs are refused: they migrate from raw. Without per-frame times the clock is start + i / fps, which meta.json's provenance records as clock "fps". The zarr's annotations array is carried in meta.json on the recording clock (frame i is steps row i, so episodes without obs_rgb_timestamps_ns map by index), and lance_commit commits a batch's annotations as one annotation change right after the batch. Replacing a recording deletes its committed annotations from the delivery's vendors that the new delivery no longer carries. lance_commit gains the zarr source, sharing load_meta_deliveries with EVA; a zarr delivery may have no capture time. rldb/lance/metadata.py holds the value helpers (text_or_none, placeholder, objects_list, read_manifest) the converters and committer share, and required_metadata.UNWAIVABLE_FIELDS the always-required fields. check_zarr_to_lance compares a converted episode with its zarr: steps values, decoded video PSNR/SSIM, and loader samples under the format's training config. Claude-Session: https://claude.ai/code/session_01HjCQBqQCr2igFEMR7yUovw

RyanPCo commented Oct 30, 2026

View reviewed changes

RyanPCo force-pushed the ryanco/zarr-to-lance branch from eab64d3 to d5864a2 Compare October 30, 2026 19:41

RyanPCo force-pushed the ryanco/lance-direct-videos branch from 60d159f to 73bd06a Compare October 30, 2026 19:41

RyanPCo mentioned this pull request Oct 30, 2026

lance: migration plan, planner and Ray driver #752

Open

RyanPCo force-pushed the ryanco/lance-direct-videos branch from 73bd06a to a03459a Compare October 30, 2026 23:05

RyanPCo force-pushed the ryanco/zarr-to-lance branch from d5864a2 to 853c113 Compare October 30, 2026 23:05

RyanPCo mentioned this pull request Oct 30, 2026

lance: annotations go through the committer; provenance in changes #734

Draft

Navigation Menu

Repository navigation

zarr_to_lance: vendor and robot zarr episodes into Lance, with their annotations - #751

zarr_to_lance: vendor and robot zarr episodes into Lance, with their annotations#751
RyanPCo wants to merge 1 commit into
ryanco/lance-direct-videosfrom
ryanco/zarr-to-lance

RyanPCo commented Oct 30, 2026 •

edited

Loading

Uh oh!

RyanPCo left a comment

Uh oh!

RyanPCo Oct 30, 2026

Uh oh!

RyanPCo Oct 30, 2026

Uh oh!

RyanPCo Oct 30, 2026

Uh oh!

RyanPCo Oct 30, 2026

Uh oh!

RyanPCo Oct 30, 2026

Uh oh!

RyanPCo Oct 30, 2026

Uh oh!

RyanPCo Oct 30, 2026

Uh oh!

RyanPCo Oct 30, 2026

Uh oh!

RyanPCo Oct 30, 2026

Uh oh!

RyanPCo Oct 30, 2026

Uh oh!

RyanPCo commented Oct 30, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant



		def _dexmate():
		# No Dexmate embodiment class on this branch: every numeric column, untransformed.

Conversation

RyanPCo commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RyanPCo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RyanPCo commented Oct 30, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

RyanPCo commented Oct 30, 2026 •

edited

Loading