diff --git a/CONTRIBUTING_DATA.md b/CONTRIBUTING_DATA.md index b182a33638..68c406cdbf 100644 --- a/CONTRIBUTING_DATA.md +++ b/CONTRIBUTING_DATA.md @@ -2,6 +2,8 @@ *For new labs and companies contributing egocentric human demonstration data to the EgoVerse consortium.* +RL2 team members: ingest, the episode database and the internal converters are documented in [INTERNAL_DATA.md](INTERNAL_DATA.md). + --- ## Table of Contents @@ -10,7 +12,7 @@ 2. [Dataset Practices](#2-dataset-practices) 3. [Prerequisites](#3-prerequisites) 4. [Episode Hash Convention](#4-episode-hash-convention) -5. [Database Registry](#5-database-registry) +5. [Episode Registry (handled by RL2)](#5-episode-registry-handled-by-rl2) 6. [Zarr v3 Episode Format](#6-zarr-v3-episode-format) 7. [Coordinate Frame Conventions](#7-coordinate-frame-conventions) 8. [Language Annotations](#8-language-annotations) @@ -32,17 +34,18 @@ Every contributed episode must satisfy these check lists: |---|---| | **File format** | Zarr v3 store with specific key names, dtypes, and shapes | | **Coordinate frame** | All poses expressed in a consistent reference frame | -| **Database record** | One row per episode in the PostgreSQL episode registry (added by the RL2 team on ingest — external contributors skip this, see §5) | +| **Metadata** | Per-episode `task_name`, `operator`, `lab`, `scene`, `objects` in `zarr.attrs` (§6.3); RL2 registers the episode on ingest (§5) | | **[Dataset Practices](#2-dataset-practices)** | Example: reducing idle times, check for data flaws | The pipeline at a glance: ``` Your raw data - └─► Convert to Zarr v3 (this guide) - └─► Register row in app.episodes DB - └─► Upload to s3://rldb/processed_v3//.zarr/ - └─► Available for download dynamically through S3MultiDataset + └─► Convert to Zarr v3 (start from the reference converter, §6.0) + └─► Validate every episode (§11) + └─► Upload to s3://rldb/processed_v3//.zarr/ (§10) + └─► Upload the raw recordings to s3://rldb/raw/// (§6.6) + └─► RL2 registers the episodes; they become available through S3MultiDataset ``` --- @@ -119,12 +122,12 @@ EgoVerse is hardware-agnostic. Any egocentric camera with a SLAM system that pro **Minimum viable setup (no robot):** egocentric camera + SLAM + hand tracking → contributes `images.front_1`, `obs_head_pose`, `left/right.obs_ee_pose`, `left/right.obs_wrist_pose`, `left/right.obs_keypoints`. -**If your SLAM system does not run at 30 fps**, ensure you upsample or interpolate pose tracks to match the RGB frame rate before writing. The training pipeline assumes all arrays are frame-aligned. +**If your SLAM system does not run at the RGB frame rate**, upsample or interpolate pose tracks to match it before writing. The training pipeline assumes all arrays are frame-aligned. The RGB stream must be 30 or 60 fps; the validator rejects any other rate (§11.1). ### 3.2 Software ```bash -# Clone and install EgoVerse +# Clone and install EgoVerse (or clone over HTTPS: https://github.com/GaTech-RL2/EgoVerse.git) git clone git@github.com:GaTech-RL2/EgoVerse.git cd EgoVerse uv venv --python 3.11 @@ -132,53 +135,41 @@ source .venv/bin/activate uv pip install -e . ``` +The reference converter writes an MP4 preview per episode (§6.5), which needs +[`ffmpeg`](https://ffmpeg.org/) on your `PATH` (e.g. `apt install ffmpeg` or +`brew install ffmpeg`). `s5cmd`, used for uploads (§10), is installed with the +Python environment. The `aws` CLI is optional. + ### 3.3 Credentials -**External contributors (partner labs / vendors): you only need Cloudflare R2 -credentials for the data bucket — nothing else.** An RL2 team member will send -them to you directly (DM), scoped to your upload prefix. You do NOT need AWS -account keys, Secrets Manager, or database access (episode registration is -handled by the RL2 team, see §5). +You only need Cloudflare R2 credentials for the data bucket. An RL2 team member +sends them to you directly. Upload only under your `lab` prefixes (§10.1). You do not need +AWS account keys or database access: RL2 registers your episodes (§5). -Set the credentials in your environment: +Set them in your environment. `s5cmd` reads the `AWS_*` names; the Python +upload helper (§10.2) reads the `R2_*` names, so set both: ```bash export AWS_ACCESS_KEY_ID= export AWS_SECRET_ACCESS_KEY= export AWS_ENDPOINT_URL_S3= export AWS_DEFAULT_REGION=auto +export R2_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" +export R2_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" +export R2_ENDPOINT_URL="$AWS_ENDPOINT_URL_S3" ``` -Verify (should list your prefix, or return empty without erroring): +Instead of exporting the `R2_*` names, you can put those three `KEY=value` lines +in `~/.egoverse_env`; the Python helpers load that file. -```bash -aws s3 ls --endpoint-url "$AWS_ENDPOINT_URL_S3" s3://rldb/processed_v3// -``` - -Note the endpoint: plain `aws s3` against AWS servers will return AccessDenied — -the bucket lives on R2, so every call needs `--endpoint-url`. - -**RL2-internal members** additionally need AWS credentials for the episode -registry (PostgreSQL via Secrets Manager): +Verify (should list your prefix, or return empty without erroring): ```bash -aws configure -# AccessKeyId / SecretAccessKey: ask the consortium lead -# Default region: us-east-2 -bash egomimic/utils/aws/setup_secret.sh -# Writes ~/.egoverse_env with R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY, -# AWS_ENDPOINT_URL_S3, SECRETS_ARN, etc. +s5cmd --endpoint-url "$AWS_ENDPOINT_URL_S3" ls "s3://rldb/processed_v3//*" ``` -Internal verification: - -```python -from egomimic.utils.aws.aws_data_utils import load_env -from egomimic.utils.aws.aws_sql import create_default_engine - -load_env() -engine = create_default_engine() # should print: Tables in schema 'app': ['episodes'] -``` +Every call needs the R2 endpoint (`--endpoint-url`); against AWS's own servers +the bucket returns AccessDenied. --- @@ -199,9 +190,10 @@ Examples: ``` **Rules:** -- The episode hash is the **primary key** in the database. It must be globally unique. +- The episode hash is the episode's ID in the EgoVerse registry. It must be globally unique, so never reuse one across your deliveries. - Use the UTC wall-clock time at the **start of the recording** as the hash. - If your hardware does not produce a UTC timestamp natively, convert from device clock using a synchronized offset. +- The helpers take **milliseconds** since the Unix epoch. Passing seconds gives a 1970 hash; passing nanoseconds fails. - The `.zarr` directory on S3 is named exactly `.zarr`. **Python helpers:** @@ -209,83 +201,44 @@ Examples: from egomimic.utils.aws.aws_sql import episode_hash_to_timestamp_ms, timestamp_ms_to_episode_hash # Convert a UTC epoch millisecond integer to an episode hash string -hash_str = timestamp_ms_to_episode_hash(1736651249664) +hash_str = timestamp_ms_to_episode_hash(1768189649664) # → "2026-01-12-03-47-29-664000" # Convert back ts_ms = episode_hash_to_timestamp_ms("2026-01-12-03-47-29-664000") -# → 1736651249664 +# → 1768189649664 ``` --- -## 5. Database Registry - -Every episode is registered in the PostgreSQL `app.episodes` table, the authoritative -index used by all download and training tooling. **Registration is performed by the RL2 -team — external contributors do NOT need database access and should skip this section.** -Deliver your episodes (`.zarr` stores + `.mp4` previews, with the attributes from §6) and -we register them on ingest. The schema below is documented for RL2-internal use. +## 5. Episode Registry (handled by RL2) -### 5.1 Schema +Every episode is listed in an internal registry that the download and training +tooling reads. **RL2 adds your episodes to it on ingest; you do not need database +access.** Upload the `.zarr` stores and `.mp4` previews with the attributes from +§6.3 (§10), and the registry row is built from them. The internal schema and +ingest steps are in [INTERNAL_DATA.md](INTERNAL_DATA.md). -The authoritative schema is the `TableRow` dataclass defined in [egomimic/utils/aws/aws_sql.py](egomimic/utils/aws/aws_sql.py). Refer to that file for the exact set of fields, defaults, and types — this guide may drift if the schema changes. - -Key field notes: -- `episode_hash`: PRIMARY KEY, must match the `.zarr` directory name exactly (see §4). -- `operator`: **hashed** operator ID (e.g. SHA-256 hex digest). MUST be hashed before insertion — never store raw names/emails. -- `lab`: short, stable, lowercase string. Once set, do not change it (used in filters). -- `task`: high-level `task_name` that groups related episodes. Before inventing a new name, check the existing tasks in the episode registry via [`sql_tutorial.ipynb`](egomimic/scripts/tutorials/sql_tutorial.ipynb) (`df.groupby("task").size()`) and reuse one if your episode fits. If no existing task matches, canonicalize your new `task_name` to a short, stable, lowercase string that names a semantically meaningful category (e.g. `fold_clothes`, `object_in_container`) — not a one-off trial description. Put trial-specific detail in `task_description`, `scene`, and `objects`. -- `embodiment`: must be one of the strings in §9. -- `robot_name`: the same `human_*` / `eva_*` string as `embodiment` (there is no per-platform variant — record your lab/hardware in the `lab` field, not here). +--- -### 5.2 Inserting a Row +## 6. Zarr v3 Episode Format -```python -from egomimic.utils.aws.aws_sql import TableRow, add_episode, create_default_engine -from egomimic.utils.aws.aws_data_utils import load_env - -load_env() -engine = create_default_engine() - -# IMPORTANT: hash the operator identifier before inserting. Do not store raw -# names, emails, or any PII in the `operator` column. -import hashlib -operator_hash = hashlib.sha256(b"jane_doe").hexdigest() - -row = TableRow( - episode_hash = "2026-03-15-14-22-10-000000", - operator = operator_hash, - lab = "rl2", - task = "fold_clothes", - embodiment = "human_bimanual", - robot_name = "human_bimanual", - task_description = "folding a 2T baby shirt on a blue table", - scene = "kitchen_A", - objects = "baby_shirt_2T", - num_frames = 2712, -) +Each episode is a **Zarr v3 group** (a directory ending in `.zarr`) containing arrays and top-level attributes. -add_episode(engine, row) -``` +### 6.0 Writing Your Converter -`add_episode` raises `RuntimeError` on a duplicate `episode_hash`. Check for collisions before inserting. +Start from the reference converter, [`egomimic/scripts/convert_template/template_to_zarr.py`](egomimic/scripts/convert_template/template_to_zarr.py). Copy it and replace two functions: -### 5.3 Updating a Row After Upload +- `find_raw_episodes(raw_root)` — one entry per raw episode in your layout. +- `load_episode(raw_path)` — return a `RawEpisode`: the image stream, head and hand poses as `(T, 7)` xyz + wxyz quaternion in your SLAM world frame (§7), optional `(T, 63)` keypoints, the camera K, the recording start time in UTC epoch milliseconds (§4), **that episode's** `task_name` / `operator` / `lab` / `scene` / `objects` (§6.3) read from your own capture logs, and the `source_recording_id` of the raw recording it was cut from (§6.6). -```python -from egomimic.utils.aws.aws_sql import update_episode +The rest is handled for you: the `.zarr` name, the `ZarrWriter` call with the correct `human_*` embodiment and intrinsics, validation of every episode right after it is written (§11.1 — a failing episode is deleted and reported, the batch keeps going), and the MP4 preview (§6.5). Run it on a folder, or call `convert_all(paths, output_dir, workers=N)` from your own batch script: -row.zarr_processed_path = "s3://rldb/processed_v3/human/2026-03-15-14-22-10-000000.zarr" -row.num_frames = 2712 -update_episode(engine, row) +```bash +python egomimic/scripts/convert_template/template_to_zarr.py --raw-root /raw --output-dir /out --workers 8 ``` ---- - -## 6. Zarr v3 Episode Format - -Each episode is a **Zarr v3 group** (a directory ending in `.zarr`) containing arrays and top-level attributes. +As shipped, `load_episode` returns synthetic data with `lab="template"`, which the validator rejects; set `lab` to your own id and you can run the template end to end before writing the rest of your loader. Re-running into an `--output-dir` that already holds an episode fails that episode (`FileExistsError`), so write each batch to a fresh directory. If an episode has `scene` or `objects` set to `MissingMetadata()` (§6.3), add `--allow-missing`; otherwise it is deleted. The MP4 preview needs `ffmpeg`; if it fails, the episode is kept and a warning is logged (`--no-mp4` skips previews). The sections below are the spec it implements. ### 6.1 Directory Structure @@ -339,7 +292,7 @@ boundary — e.g. `ZarrWriter` pads to `chunk_timesteps`); consumers must slice | Key | Shape | Dtype | Frame | Notes | |---|---|---|---|---| -| `obs_head_pose` | `(T, 7)` | `float64` | SLAM world frame | 6-DOF pose of the egocentric camera/device as XYZWXYZ. This is the pivot used at training time to re-express all other poses into head-relative coordinates. **Required for all contributors.** | +| `obs_head_pose` | `(T, 7)` | `float64` | SLAM world frame | 6-DOF pose of the egocentric camera/device as xyz + wxyz. This is the pivot used at training time to re-express all other poses into head-relative coordinates. **Required for all contributors.** | #### Hand and Wrist Poses (if hand tracking is available) @@ -347,10 +300,10 @@ Provide these if your setup produces 3D hand estimates. Omit the entire key (do | Key | Shape | Dtype | Frame | Notes | |---|---|---|---|---| -| `left.obs_ee_pose` | `(T, 7)` | `float64` | SLAM world frame | Left hand end-effector (fingertip centroid or palm center) pose as XYZWXYZ. If you have keypoints, derive this from them — see [Deriving `obs_ee_pose` from MANO keypoints](#deriving-obs_ee_pose-from-mano-keypoints-recommended) | -| `right.obs_ee_pose` | `(T, 7)` | `float64` | SLAM world frame | Right hand end-effector pose as XYZWXYZ (same derivation guidance as the left) | -| `left.obs_wrist_pose` | `(T, 7)` | `float64` | SLAM world frame | Left wrist origin pose as XYZWXYZ | -| `right.obs_wrist_pose` | `(T, 7)` | `float64` | SLAM world frame | Right wrist origin pose as XYZWXYZ | +| `left.obs_ee_pose` | `(T, 7)` | `float64` | SLAM world frame | Left hand end-effector (fingertip centroid or palm center) pose as xyz + wxyz. If you have keypoints, derive this from them — see [Deriving `obs_ee_pose` from MANO keypoints](#deriving-obs_ee_pose-from-mano-keypoints-recommended) | +| `right.obs_ee_pose` | `(T, 7)` | `float64` | SLAM world frame | Right hand end-effector pose as xyz + wxyz (same derivation guidance as the left) | +| `left.obs_wrist_pose` | `(T, 7)` | `float64` | SLAM world frame | Left wrist origin pose as xyz + wxyz | +| `right.obs_wrist_pose` | `(T, 7)` | `float64` | SLAM world frame | Right wrist origin pose as xyz + wxyz | | `left.obs_keypoints` | `(T, 63)` | `float64` | SLAM world frame | 21 hand landmarks × 3 (x, y, z); flattened row-major (see ordering below) | | `right.obs_keypoints` | `(T, 63)` | `float64` | SLAM world frame | 21 hand landmarks × 3 (x, y, z); flattened row-major | @@ -373,8 +326,10 @@ centroid of the wrist and the index/middle/ring MCPs (joints `0, 5, 9, 13`) — and the orientation is built from the wrist→palm direction plus a palm normal taken as `(index_MCP − wrist) × (pinky_MCP − wrist)`, sign-flipped for the left hand so the normal points out of the palm on both sides. Frames with missing or -degenerate keypoints carry the `1e9` sentinel. Output is `(T, 7)` `XYZWXYZ`, -the same layout as every other pose key. EgoVerse provides this as +degenerate keypoints carry the `1e9` sentinel. Output is `(T, 7)` xyz + wxyz, +the same layout as every other pose key. **Trim sentinel frames before writing** +(§2.4): a single `1e9` frame has a non-unit quaternion, which fails the whole +episode in the validator (§11.1). NaN or inf values fail it too. EgoVerse provides this as `mano_keypoints_to_cartesian` in [`egomimic/scripts/aria_process/aria_utils.py`](egomimic/scripts/aria_process/aria_utils.py): @@ -391,8 +346,8 @@ ee_pose = mano_keypoints_to_cartesian(mano_kp, is_rhand=True) # -> (T, 7) | Key | Shape | Dtype | Notes | |---|---|---|---| -| `left.obs_ee_pose` | `(T, 7)` | `float64` | Left arm EEF pose as XYZWXYZ in robot base frame | -| `right.obs_ee_pose` | `(T, 7)` | `float64` | Right arm EEF pose as XYZWXYZ in robot base frame | +| `left.obs_ee_pose` | `(T, 7)` | `float64` | Left arm EEF pose as xyz + wxyz in robot base frame | +| `right.obs_ee_pose` | `(T, 7)` | `float64` | Right arm EEF pose as xyz + wxyz in robot base frame | | `left.obs_gripper` | `(T, 1)` | `float64` | Left gripper aperture in [0, 1] (0 = fully closed) | | `right.obs_gripper` | `(T, 1)` | `float64` | Right gripper aperture in [0, 1] | | `left.cmd_ee_pose` | `(T, 7)` | `float64` | Commanded left EEF pose (if available) | @@ -413,11 +368,17 @@ The root group's `.attrs` dictionary is the **episode metadata**. It is written ```python { - "embodiment": str, # e.g. "human_bimanual" (must match DB row) + "schema_version": int, # set by ZarrWriter (currently 1); do not set it yourself + "embodiment": str, # e.g. "human_bimanual" (§9) "total_frames": int, # number of valid frames (not padded) - "fps": int, # capture frame rate (typically 30) - "task_name": str, # e.g. "fold_clothes" (must match DB row) + "fps": int, # capture frame rate: 30 or 60 + "task_name": str, # e.g. "fold_clothes" (you propose it, see below) "task_description": str, # free-text description of the trial + "operator": str, # per-episode ID of the person who collected it + "lab": str, # who the data came from: short lowercase id, e.g. "mecka" + "scene": str, # scene / location label + "objects": list[str], # objects manipulated + "source_recording_id": str, # REQUIRED: raw recording this episode was cut from (§6.6) "intrinsics": dict, # MANDATORY: {camera_key: 3x4 K matrix} dict (single-camera = # one entry, e.g. {"front_1": K}; 3x4 = the 3x3 pinhole K # with a zero last column). Projection uses the "front" entry. @@ -441,12 +402,34 @@ The root group's `.attrs` dictionary is the **episode metadata**. It is written **Rules:** - `total_frames` must equal `len(store["images.front_1"])` and every other non-padded array. -- `fps` must be the actual capture rate of `images.front_1`. Do not set to a target rate if the actual rate differs. +- `fps` must be the actual capture rate of `images.front_1`, and must be 30 or 60. Do not set it to a target rate if the actual rate differs. - `features` must have one entry per array key present in the store. -- `embodiment` and `task_name` must exactly match the values in the DB row for this episode. +- `source_recording_id` is **required**: it is how we find the raw recording again (§6.6), and with `lab` it gives the recording's ID downstream. Pass it as `ZarrWriter.create_and_write(..., source_recording_id=...)`; the reference converter does this from `RawEpisode.source_recording_id`. It must be a single path component (no `/`), with no placeholder or surrounding whitespace. - `intrinsics` is **mandatory** and is always a `{camera_key: 3×4 K matrix}` dict in `zarr.attrs` (single-camera = one entry, e.g. `{"front_1": K}`). `ZarrWriter.create_and_write` raises if it is not a non-empty dict. - `extrinsics` is **either `None` or a non-empty `dict`** of 4×4 world↔cam transforms (robots key per-arm, e.g. `{"left": T, "right": T}`); egocentric human contributors omit it (`None`). `ZarrWriter.create_and_write` raises if it is anything other than `None` or a non-empty dict. +#### Required metadata fields (`task_name`, `operator`, `lab`, `scene`, `objects`) + +These are what let people filter the data and train on it — per-operator splits, held-out scenes, object-conditioned subsets. **`task_name`, `operator` and `lab` are always required and cannot be waived.** We heavily recommend providing `scene` and `objects` for every episode too. `ZarrWriter.create_and_write` takes each as a required keyword argument with **no default**, and raises `ValueError` on an empty string, a placeholder (`"unknown"`, `"none"`, `"n/a"`, `"tbd"`, `"debug"`, `"test"`, …), or a value with leading or trailing whitespace. + +- `task_name`: propose your own short, stable, lowercase name for the task category, e.g. `fold_clothes` or `object_in_container` — not a one-off trial description. RL2 maps it to the existing task list on ingest. Put trial-specific detail in `task_description`, `scene` and `objects`. +- `operator` is per-episode — use your own stable collector ID (a pseudonymous ID is fine). +- `lab` names who the data came from. It must be a full match of `^[a-z0-9][a-z0-9_-]*$`, is the same string as your S3 prefix (§10.1), and must never change once chosen. +- `objects` is a list of names, or a comma-separated string (`"shirt, basket"`), which is stored as a list. Every entry is checked the same way as the other fields. + +If your source data genuinely has no `scene` or `objects`, you must say so explicitly: + +```python +from egomimic.rldb.zarr.required_metadata import MissingMetadata + +ZarrWriter.create_and_write( + ..., + scene=MissingMetadata(), +) +``` + +Only do this if absolutely necessary; it is rejected for `task_name`, `operator` and `lab`. The field is then stored as `null`, and the validator (§11.1) fails the episode unless it is run with `--allow-missing`. With the reference converter (`template_to_zarr.py`), pass `--allow-missing` (or `allow_missing=True` to `convert_all`); otherwise it deletes the episode. Episodes converted this way are accepted, and the `null` fields are visible in `zarr.json`. + ### 6.4 Storage / Chunking > # ⚠️ **USE THE [`ZarrWriter`](egomimic/rldb/zarr/zarr_writer.py) CLASS** ⚠️ @@ -457,7 +440,7 @@ The root group's `.attrs` dictionary is the **episode metadata**. It is written - **Annotation arrays**: chunk shape `(N,)`, sharded to `(N,)`. - **Zarr format version**: always **v3** (`zarr_format=3`). -See example usage in eva_to_zarr.py and aria_to_zarr.py. +See the reference converter (§6.0) for a complete example. #### Camera intrinsics & extrinsics — **how to store them** @@ -485,9 +468,13 @@ ZarrWriter.create_and_write( numeric_data=numeric_arrays, # left/right.obs_ee_pose, obs_head_pose, ... image_data=image_arrays, # images.front_1, ... intrinsics={"front_1": K_front}, # REQUIRED — always a {camera_key: 3x4} dict - # extrinsics=..., # REQUIRED for robot embodiments only + # extrinsics=..., # optional; robots key it per arm fps=30, - task_name="...", + task_name="...", # REQUIRED metadata fields, see §6.3 + operator="...", + lab="...", + scene="...", + objects=["..."], task_description="...", ) ``` @@ -496,7 +483,33 @@ ZarrWriter.create_and_write( Alongside each `.zarr` store, write a preview video of the egocentric RGB stream named **`.mp4`** (e.g. `2026-03-15-14-22-10-000000.mp4`). The Mecka AI dataset viz looks previews up by this exact filename, so any deviation from the `.mp4` convention will break it. -Any standard MP4 encoder works. If it's convenient, the [`save_preview_mp4`](egomimic/utils/video_utils.py) helper is available — `aria_to_zarr.py` and `eva_to_zarr.py` use it (via the `--save-mp4` flag) and emit the file next to the `.zarr` directory. +Any standard MP4 encoder works. The reference converter (§6.0) writes it for you with the [`save_preview_mp4`](egomimic/utils/video_utils.py) helper (needs `ffmpeg`, §3.2), next to the `.zarr` directory. Upload it next to the zarr as well: `s3://rldb/processed_v3//.mp4` (§10). + +### 6.6 Raw Data (alongside the Zarr) + +Zarr is what we train on, so deliver it first. Also upload the **raw recordings** each +episode was converted from, so we can rebuild the data later (new formats, higher +resolution, different processing) without coming back to you. + +- **Any format.** Send raw data the way your pipeline stores it. Use the raw-data + reference, [`egomimic/rldb/mcap/SPEC.md`](egomimic/rldb/mcap/SPEC.md), as the checklist + of what it should contain: per-camera calibration, capture timestamps on every + stream, raw IMU, head / hand (or arm) poses with tracking validity, transforms between + frames, and the §6.3 metadata (`task_name`, `operator`, `lab`, `scene`, `objects`). +- **Include video**, as the device stores it or lightly compressed (e.g. H.264 / H.265 at + a high-quality setting). Keep the original resolution and frame rate: no downscaling or + cropping, and encode at most once. +- **One folder per raw recording session**, at + `s3://rldb/raw///`. `source_recording_id` is your own stable + ID for the session; keep it the same across re-exports and corrections. +- **Link each zarr episode to its session** with the `source_recording_id` attribute + (§6.3). One session can be trimmed into several zarr episodes (§2.4); each of them + carries the same `source_recording_id`. This link is required and validated (§11.1). +- **If you use MCAP**, follow SPEC.md exactly (its episode info uses the same + `source_recording_id` key) and check every episode with + `python -m egomimic.rldb.mcap.validate episode_/ [...]`; `scene` / `objects` waived in + `missing_metadata` fail unless `--allow-missing`, as for Zarr; `task_name`, `operator` + and `lab` cannot be waived. --- @@ -562,7 +575,7 @@ The `annotations` array in the Zarr store contains `N` entries, where `N` is the |---|---|---| | `text` | `str` | Natural-language description of what is happening during `[start_idx, end_idx)` | | `start_idx` | `int` | First frame index (inclusive) | -| `end_idx` | `int` | Last frame index (exclusive). Must satisfy `0 <= start_idx < end_idx <= total_frames`. | +| `end_idx` | `int` | One past the last frame (exclusive). Must satisfy `0 <= start_idx <= end_idx <= total_frames`; a span covering the whole episode is `(0, total_frames)`. | **Rules:** - Spans may overlap. @@ -598,35 +611,36 @@ annotations = [ ("folding the bottom half up", 910, 1200), ] -writer = ZarrWriter( +ZarrWriter.create_and_write( episode_path="path/to/.zarr", embodiment="human_bimanual", fps=30, - task_name="fold_clothes", task_description="folding a 2T baby shirt", annotations=annotations, + intrinsics={"front_1": K_front}, + task_name="fold_clothes", + operator="...", lab="...", scene="...", objects=["baby_shirt_2T"], # §6.3 + # numeric_data=..., image_data=... ) ``` -To append annotations to an **existing** Zarr store: +To write annotations into an **existing** Zarr store (keeps `features` in sync): ```python -writer = ZarrWriter(episode_path="path/to/.zarr") -writer.append_annotations( +from egomimic.rldb.zarr.zarr_writer import write_annotations + +write_annotations( + "path/to/.zarr", + annotations, annotation_key="annotations", - annotations=annotations, - mode="w", # "w" = overwrite existing, "a" = append + mode="w", # "w" = replace an existing key, "a" = add a new key only ) ``` -### 8.4 Scale AI Annotation Format - -If you are delivering data through Scale AI, annotations are generated via the Scale annotation API. The `ScaleAnnotationDatasetFilter` class can be used to filter episodes to only those with completed Scale annotations. Set `SCALE_API_KEY` in your environment. - --- ## 9. Embodiment Identifiers -The `embodiment` field in the DB row and in `zarr.attrs` must be exactly one of the strings below. **All human demonstration data is a single `human` embodiment — there is no per-vendor or per-hardware embodiment.** The lab / hardware that produced the data is recorded separately in the SQL `lab` field, never in `embodiment`. Only the robot **Eva** is a distinct non-human embodiment. +The `embodiment` field in `zarr.attrs` must be exactly one of the strings below. **All human demonstration data is a single `human` embodiment — there is no per-vendor or per-hardware embodiment.** The lab / hardware that produced the data is recorded separately in the `lab` field (`zarr.attrs`, §6.3), never in `embodiment`. Only the robot **Eva** is a distinct non-human embodiment. | `embodiment` string | Integer id | Description | |---|---|---| @@ -641,20 +655,12 @@ If you are contributing egocentric human data, you use `human_bimanual` (or the ### 9.1 Using the `Human` embodiment (no subclass, no per-vendor identifier) -There is a **single concrete `Human` embodiment class** ([`egomimic/rldb/embodiment/human.py`](egomimic/rldb/embodiment/human.py)) shared by all human data. You do **not** write a per-vendor subclass, and there is **no per-vendor embodiment identifier** — every human contributor uses `human_*` and records their source in the `lab` field. Camera intrinsics travel with the data (`zarr.attrs`, §6.3 / §6.4); per-vendor structural choices are passed as explicit arguments from the data config: - -- `Human.get_keymap(keymap_mode="cartesian"|"keypoints", has_head_pose=, include_aria_keypoints=)` -- `Human.get_transform_list(mode="cartesian_6d"|"keypoints_headframe_ypr"|..., stride=)` +There is a **single `Human` embodiment class** ([`egomimic/rldb/embodiment/human.py`](egomimic/rldb/embodiment/human.py)) shared by all human data. You do **not** write a per-vendor subclass or register anything in code: -Onboarding human data is just two steps: +1. Write `embodiment="human_bimanual"` (or `human_left_arm` / `human_right_arm`) in `zarr.attrs` — the reference converter picks it from which hands you track — and record your source in the `lab` field. +2. Put your camera intrinsics in `zarr.attrs` as a `{camera_key: 3x4}` dict (§6.4). They travel with the data; there is no `INTRINSICS` constant in code. -1. Write `embodiment="human_bimanual"` (or `human_left_arm` / `human_right_arm`) in the DB row and in `zarr.attrs`; record your lab/hardware in the `lab` field. -2. Add a data config under [`egomimic/hydra_configs/data/`](egomimic/hydra_configs/data/) whose `key_map` / `transform_list` point at `Human.get_keymap` / `Human.get_transform_list` with the args your data needs. Copy `aria.yaml` (head-mounted, `stride: 3`) or `scale.yaml` (`stride: 1`). - -Notes: -- Camera **intrinsics are MANDATORY** and live in `zarr.attrs` as a `{camera_key: 3x4}` dict (§6.4). You no longer declare an `INTRINSICS` constant in code. -- `has_head_pose=False` if your data has no `obs_head_pose`; `stride` is the action-chunk stride (`3` for ~30 fps egocentric, `1` for already-downsampled data). -- **Robots** subclass `Embodiment` directly and keep their own intrinsics/extrinsics + pipeline — see `Eva` in [`egomimic/rldb/embodiment/eva.py`](egomimic/rldb/embodiment/eva.py). +RL2 adds the training data config for your data on ingest. --- @@ -662,272 +668,77 @@ Notes: ### 10.1 S3 Path Convention -**External contributors (partner labs / vendors): upload everything under your -assigned company prefix —** +Upload everything under the prefix named by your `lab` (§6.3). Your RL2 contact +confirms the `lab` string when they send your credentials. Write only to these +prefixes; never write to or modify another lab's data. ``` -s3://rldb/processed_v3//.zarr/ +s3://rldb/processed_v3//.zarr/ # the episode (§6) +s3://rldb/processed_v3//.mp4 # its preview (§6.5) +s3://rldb/raw/// # raw recordings (§6.6) ``` -`` is the short lowercase name your RL2 contact gives you along -with your credentials (which are scoped to exactly this prefix — uploads -anywhere else will be denied). Batch deliveries (e.g. a `.tar.zst` of many -episodes) also go under this prefix. - Example: ``` s3://rldb/processed_v3/acmecorp/2026-03-15-14-22-10-000000.zarr/ +s3://rldb/processed_v3/acmecorp/2026-03-15-14-22-10-000000.mp4 +s3://rldb/raw/acmecorp/session_0412/ ``` -**RL2-internal uploads** use the embodiment prefix instead: - -| Embodiment | `` | -|---|---| -| `human_*` | `human` | -| `eva_*` | `eva` | - -``` -s3://rldb/processed_v3/human/2026-03-15-14-22-10-000000.zarr/ -s3://rldb/processed_v3/eva/2025-11-04-09-30-00-000000.zarr/ -``` +Upload `.zarr` directories as they are (not archived), so they can be read +directly from the bucket. ### 10.2 Upload with `s5cmd` `s5cmd` is the recommended upload tool (installed as part of the Python environment). ```bash -# Upload a local .zarr directory -s5cmd --endpoint-url $AWS_ENDPOINT_URL_S3 \ - sync "/local/processed/2026-03-15-14-22-10-000000.zarr/*" \ - "s3://rldb/processed_v3/human/2026-03-15-14-22-10-000000.zarr/" +# Every validated episode and preview in your output directory +s5cmd --endpoint-url "$AWS_ENDPOINT_URL_S3" \ + sync "/local/out/*" "s3://rldb/processed_v3//" + +# One raw recording session +s5cmd --endpoint-url "$AWS_ENDPOINT_URL_S3" \ + sync "/local/raw/session_0412/*" "s3://rldb/raw//session_0412/" ``` -Or using the Python utility: +Or using the Python helper (reads the `R2_*` variables, §3.3): ```python -from egomimic.utils.aws.aws_data_utils import upload_dir_to_s3, load_env +from egomimic.utils.aws.aws_data_utils import upload_dir_to_s3 -load_env() upload_dir_to_s3( - local_dir = "/local/processed/2026-03-15-14-22-10-000000.zarr", + local_dir = "/local/out/2026-03-15-14-22-10-000000.zarr", bucket = "rldb", - prefix = "processed_v3/human/2026-03-15-14-22-10-000000.zarr", + prefix = "processed_v3//2026-03-15-14-22-10-000000.zarr", ) ``` -### 10.3 Bulk Upload with Ray - -For batch uploads of many episodes, use Ray to parallelize: - -```python -import ray -from egomimic.utils.aws.aws_data_utils import upload_dir_to_s3, load_env - -ray.init() - -@ray.remote -def upload_one(local_zarr_path: str, s3_prefix: str): - load_env() - upload_dir_to_s3(local_zarr_path, bucket="rldb", prefix=s3_prefix) - -tasks = [ - upload_one.remote( - f"/local/processed/{h}.zarr", - f"processed_v3/human/{h}.zarr" - ) - for h in episode_hashes -] -ray.get(tasks) -``` - --- ## 11. Validation and Verification ### 11.1 Automated Checks -Run these checks on every episode before uploading: +Run the validator on every episode before uploading: -```python -import zarr, numpy as np -import json -from pathlib import Path -from egomimic.rldb.zarr.zarr_dataset_multi import ZarrEpisode -import simplejpeg - -def validate_episode(zarr_path: str) -> tuple[list[str], list[str]]: - """Returns (errors, successes). Empty errors list = pass.""" - errors: list[str] = [] - successes: list[str] = [] - ep = ZarrEpisode(zarr_path) - meta = ep.metadata - T = meta["total_frames"] - store = zarr.open(zarr_path, mode="r") - - # ── Metadata ──────────────────────────────────────────────────────────── - for field in ("embodiment", "total_frames", "fps", "task_name", "features"): - if field not in meta: - errors.append(f"Missing metadata field: {field}") - else: - successes.append(f"metadata field present: {field}") - - if meta.get("fps", 0) not in (30, 60): - errors.append(f"Unexpected fps={meta['fps']}. Expected 30 or 60.") - else: - successes.append(f"fps={meta['fps']} is valid") - - # ── Embodiment identifier (must resolve to a valid id; see §9) ────────── - from egomimic.rldb.embodiment.embodiment import get_embodiment_id - try: - get_embodiment_id(meta.get("embodiment", "")) - successes.append(f"embodiment={meta.get('embodiment')} is a valid identifier") - except (KeyError, AttributeError): - errors.append(f"embodiment={meta.get('embodiment')!r} is not a valid identifier (see §9)") - - # ── Camera intrinsics (MANDATORY; {camera_key: 3x4 K matrix} dict) ────── - intr = meta.get("intrinsics") - if not isinstance(intr, dict) or not intr: - errors.append("intrinsics: missing or not a non-empty {camera_key: 3x4} dict") - else: - if not any("front" in str(k).lower() for k in intr): - errors.append(f"intrinsics: no front-camera entry (keys: {list(intr)})") - for cam, K in intr.items(): - if np.asarray(K, dtype=float).shape != (3, 4): - errors.append(f"intrinsics['{cam}']: expected 3x4 K, got shape {np.asarray(K).shape}") - else: - successes.append(f"intrinsics['{cam}']: 3x4 OK") - - # ── Camera extrinsics (OPTIONAL; None, or a non-empty dict of transforms) ─ - if "extrinsics" in meta and meta["extrinsics"] is not None: - extr = meta["extrinsics"] - if not isinstance(extr, dict) or not extr: - errors.append("extrinsics: present but not a non-empty dict (must be None or a dict)") - else: - successes.append(f"extrinsics: non-empty dict OK (keys: {list(extr)})") - - # ── Frame counts ──────────────────────────────────────────────────────── - features = meta.get("features", {}) - for key in store.keys(): - node = store[key] - if not isinstance(node, zarr.Array): - continue - if features.get(key, {}).get("dtype") == "json": - continue - arr_len = node.shape[0] - if arr_len < T: - errors.append(f"{key}: array length {arr_len} < total_frames {T}") - else: - successes.append(f"{key}: frame count OK ({arr_len} >= {T})") - - # ── Required keys ─────────────────────────────────────────────────────── - required = ["images.front_1", "left.obs_ee_pose", "right.obs_ee_pose"] - for key in required: - if key not in store: - errors.append(f"Missing required key: {key}") - else: - successes.append(f"required key present: {key}") - - # ── Pose shapes and norms ─────────────────────────────────────────────── - required_poses = ("left.obs_ee_pose", "right.obs_ee_pose") - optional_poses = ("left.obs_wrist_pose", "right.obs_wrist_pose", "obs_head_pose", "left.cmd_ee_pose", "right.cmd_ee_pose") - for key in required_poses + optional_poses: - if key in store: - arr = store[key][:] - if arr.shape != (T, 7) and arr.shape[0] >= T: - arr = arr[:T] - if arr.shape[-1] != 7: - errors.append(f"{key}: expected shape (T, 7), got {arr.shape}") - continue - else: - successes.append(f"{key}: shape OK (T, 7)") - quat = arr[:, 3:7] - norms = np.linalg.norm(quat, axis=1) - if not np.allclose(norms, 1.0, atol=1e-4): - bad = np.where(np.abs(norms - 1.0) > 1e-4)[0] - errors.append(f"{key}: {len(bad)} frames with non-unit quaternions (e.g. frame {bad[0]}, norm={norms[bad[0]]:.6f})") - else: - successes.append(f"{key}: all quaternions unit-norm") - - # ── Gripper shapes (optional) ─────────────────────────────────────────── - for key in ("left.obs_gripper", "right.obs_gripper", "left.gripper", "right.gripper"): - if key in store: - arr = store[key][:] - if arr.shape[0] < T: - errors.append(f"{key}: array length {arr.shape[0]} < total_frames {T}") - continue - if arr.ndim != 2 or arr.shape[-1] != 1: - errors.append(f"{key}: expected shape (T, 1), got {arr.shape}") - else: - successes.append(f"{key}: gripper shape OK (T, 1)") - - # ── Keypoint shapes ───────────────────────────────────────────────────── - for key in ("left.obs_keypoints", "right.obs_keypoints"): - if key in store: - arr = store[key][:] - if arr.shape[-1] != 63: - errors.append(f"{key}: expected last dim 63 (21×3), got {arr.shape[-1]}") - else: - successes.append(f"{key}: keypoint shape OK (last dim = 63)") - - # ── Annotation format (JSON-encoded records) ──────────────────────────── - annotation_keys = [k for k, f in features.items() if f.get("dtype") == "json" and k in store] - for key in annotation_keys: - node = store[key] - n = node.shape[0] - bad = 0 - first_err = None - for i in range(n): - raw = node[i] - # Unwrap any nested 0-d object/bytes ndarrays down to raw bytes. - while isinstance(raw, np.ndarray): - raw = raw.item() if raw.shape == () else raw.flat[0] - if isinstance(raw, np.bytes_): - raw = bytes(raw) - try: - rec = json.loads(raw.decode("utf-8") if isinstance(raw, (bytes, bytearray)) else raw) - if not isinstance(rec, dict): - raise ValueError(f"record is {type(rec).__name__}, expected dict") - for field, expected in (("text", str), ("start_idx", int), ("end_idx", int)): - if field not in rec: - raise ValueError(f"missing field '{field}'") - if not isinstance(rec[field], expected): - raise ValueError(f"field '{field}' is {type(rec[field]).__name__}, expected {expected.__name__}") - if not (0 <= rec["start_idx"] <= rec["end_idx"] <= T): - raise ValueError(f"index range invalid: start={rec['start_idx']}, end={rec['end_idx']}, T={T}") - except Exception as e: - bad += 1 - if first_err is None: - first_err = (i, str(e)) - if bad: - errors.append(f"{key}: {bad}/{n} annotations malformed (e.g. index {first_err[0]}: {first_err[1]})") - else: - successes.append(f"{key}: all {n} annotations well-formed") - - # ── Image decodability (spot-check first frame of each JPEG key) ──────── - jpeg_keys = [k for k, f in features.items() if f.get("dtype") == "jpeg" and k in store] - for key in jpeg_keys: - data = ep.read({key: (0, None)}) - try: - frame = simplejpeg.decode_jpeg(bytes(data[key]), colorspace="RGB") - if frame.ndim != 3 or frame.shape[2] != 3: - errors.append(f"{key}: decoded frame has unexpected shape {frame.shape}") - else: - successes.append(f"{key}: frame 0 decoded OK, shape={frame.shape}") - except Exception as e: - errors.append(f"{key}: failed to decode frame 0: {e}") - - return errors, successes - -# Usage -errors, successes = validate_episode("/storage/project/r-dxu345-0/shared/pick_place/2026-03-17-18-09-03-000000") -for s in successes: - print("OK:", s) -if errors: - for e in errors: - print("ERROR:", e) -else: - print("All checks passed.") +```bash +python -m egomimic.rldb.zarr.validate /local/out/*.zarr ``` +Pass the `.zarr` directories themselves, not the folder that contains them. It prints `PASS`/`FAIL` per episode with every problem found, and exits nonzero if any episode fails. Add `-v` to also list the checks that passed. It checks: + +- metadata (§6.3): `embodiment`, `total_frames`, `fps` (30 or 60), `features`, and the required metadata fields `task_name`/`operator`/`lab`/`scene`/`objects` and `source_recording_id`, with the same rules as `create_and_write` +- `embodiment` is a valid identifier (§9) +- the `.zarr` name decodes to a recording start between 2015 and now (§4); a start time in seconds or nanoseconds instead of milliseconds fails +- `intrinsics` is a non-empty `{camera_key: 3x4}` dict with a front camera; `extrinsics` is `None` or a dict of 4x4 matrices +- every array has a `features` entry (and vice versa) and at least `total_frames` rows +- required keys: `images.front_1`, `obs_head_pose` (human embodiments), plus `left`/`right.obs_ee_pose` for the arms in the embodiment +- pose arrays are `(T, 7)` with unit quaternions; grippers `(T, 1)`; keypoints last dim 63; no NaN or inf in any of them +- annotations are well-formed `annotation_v1` records within `[0, T]` +- frame 0 of every JPEG key decodes to an RGB image + +`scene` or `objects` written as `MissingMetadata()` (stored as `null`) fails the check by default; a `null` `task_name`, `operator` or `lab` always fails. `--allow-missing` downgrades it to a warning; only use it once you have confirmed the source data really has no value for that field. Episodes that pass with `--allow-missing` are accepted. From Python, `validate_episode(path)` returns a `ValidationReport` with `.errors`, `.warnings` and `.ok`. + ### 11.2 End-to-End Load Test Verify the episode loads correctly through the full training pipeline before uploading: @@ -939,11 +750,8 @@ from egomimic.rldb.filters import DatasetFilter from egomimic.rldb.embodiment.human import Human import torch -# cartesian mode re-expresses every pose relative to obs_head_pose, so the -# cartesian transform REQUIRES a head pose. Every current vendor export (Aria, -# Mecka, Scale) writes obs_head_pose. Only a contributor without one should set -# has_head_pose=False, and then transform_list must be None (validate the raw -# episode with the §11.1 checks instead). +# cartesian mode re-expresses every pose relative to obs_head_pose, which every +# human episode must have (§6.2, checked by §11.1). HAS_HEAD_POSE = True key_map = Human.get_keymap(keymap_mode="cartesian", has_head_pose=HAS_HEAD_POSE) transform_list = ( @@ -969,14 +777,14 @@ for batch in loader: pass ``` -Expected output for a valid human bimanual episode in cartesian mode: -- `actions_cartesian`: `(B, 100, 12)` — 100-step action chunk, 6 DOF × 2 arms -- `observations.state.ee_pose`: `(B, 12)` — current EEF poses, 6 DOF × 2 arms +Expected output for a valid human bimanual episode in `cartesian_6d` mode: +- `actions_cartesian`: `(B, 100, 18)` — 100-step action chunk, (3 xyz + 6 rotation) × 2 arms +- `observations.state.ee_pose`: `(B, 18)` — current EEF poses, (3 xyz + 6 rotation) × 2 arms - `observations.images.front_img_1`: `(B, 3, H, W)` — normalized RGB in `[0, 1]` ### 11.3 Visual Verification -After the load test passes, render a quick trajectory overlay on a local episode — it projects the action chunk onto the egocentric image using the per-episode intrinsics from `zarr.attrs`, so a wrong K matrix or coordinate frame shows up immediately as an overlay floating off the hands. (This step uses the cartesian `actions_cartesian` chunk, so it applies to head-mounted data with a head pose; head-pose-less data has no cartesian chunk to project — rely on §11.1 + the §11.2 raw load for those.) +After the load test passes, render a quick trajectory overlay on a local episode (the same check as [`egomimic/scripts/tutorials/zarr_data_viz.py`](egomimic/scripts/tutorials/zarr_data_viz.py)) — it projects the action chunk onto the egocentric image using the per-episode intrinsics from `zarr.attrs`, so a wrong K matrix or coordinate frame shows up immediately as an overlay floating off the hands. (This step uses the cartesian `actions_cartesian` chunk, so it applies to head-mounted data with a head pose; head-pose-less data has no cartesian chunk to project — rely on §11.1 + the §11.2 raw load for those.) ```python import imageio, torch @@ -987,7 +795,11 @@ from egomimic.rldb.filters import DatasetFilter resolver = LocalEpisodeResolver( folder_path = "/local/processed", key_map = Human.get_keymap(keymap_mode="cartesian"), - transform_list = Human.get_transform_list(mode="cartesian_6d", stride=3), + # The overlay draws xyz + yaw/pitch/roll, so it reads the plain cartesian + # chunk; training uses cartesian_6d (§11.2). + transform_list = Human.get_transform_list( + mode="cartesian", stride=3, allow_legacy_rotation=True + ), ) filters = DatasetFilter(filter_lambdas=[ "lambda row: row['episode_hash'] == '2026-03-15-14-22-10-000000'" @@ -1017,25 +829,26 @@ Do not upload an episode whose visualization is visibly misaligned. Complete every item before considering an episode ready for upload. **Episode hash** -- [ ] Episode hash is a valid UTC timestamp string (`YYYY-MM-DD-HH-MM-SS-ffffff`). -- [ ] Episode hash is unique — not already in the DB (`episode_hash_to_table_row(engine, hash)` returns `None`). +- [ ] Episode hash is a valid UTC timestamp string (`YYYY-MM-DD-HH-MM-SS-ffffff`) from the recording start in epoch milliseconds (§4). +- [ ] Episode hash is unique across all your deliveries. **Zarr format** - [ ] `obs_head_pose` is present (required for all contributors). - [ ] `left.obs_ee_pose` and `right.obs_ee_pose` are present if hand tracking is available. -- [ ] All `obs_ee_pose` arrays have shape `(T, 7)` and unit-norm quaternions. +- [ ] All `obs_ee_pose` arrays have shape `(T, 7)` and unit-norm quaternions; no sentinel (`1e9`), NaN or inf frames. - [ ] All `obs_keypoints` arrays have shape `(T, 63)`. - [ ] `features` dict in `zarr.attrs` has one entry per array key. -- [ ] `embodiment` and `task_name` in `zarr.attrs` match the DB row values. +- [ ] `fps` is 30 or 60. +- [ ] `task_name`, `operator`, `lab`, `scene` and `objects` are set for every episode (§6.3). +- [ ] `source_recording_id` names the raw recording each episode was cut from (§6.6). - [ ] `intrinsics` is present in `zarr.attrs` as a `{camera_key: 3×4}` dict (single-camera = one entry) — **mandatory**. - [ ] `extrinsics` is present in `zarr.attrs` for robot embodiments. -- [ ] All episode succeeds on zarr validation check code -- [ ] An embodiment class is registered in `egomimic/rldb/embodiment/` (§9.1). -- [ ] A sample episode has been visually verified via `zarr_data_viz.ipynb` (§11.3). +- [ ] `python -m egomimic.rldb.zarr.validate` passes on every episode (§11.1); `--allow-missing` only for `scene` / `objects` you set to `MissingMetadata()`. +- [ ] A sample episode has been visually verified (§11.3). **Coordinate frames** - [ ] All poses are in the SLAM world frame (not head frame, not camera frame). -- [ ] Quaternion is stored in XYZWXYZ order: `[tx, ty, tz, qw, qx, qy, qz]`. +- [ ] Poses are stored as xyz + wxyz: `[tx, ty, tz, qw, qx, qy, qz]`. - [ ] Translation units are **meters**. **Images** @@ -1045,18 +858,15 @@ Complete every item before considering an episode ready for upload. **Annotations** - [ ] `annotations` key is present (may be empty array if no annotations available). -- [ ] All `(start_idx, end_idx)` spans satisfy `0 <= start_idx < end_idx <= total_frames`. +- [ ] All `(start_idx, end_idx)` spans satisfy `0 <= start_idx <= end_idx <= total_frames` (`end_idx` exclusive). - [ ] Annotation text is in English, imperative or present-continuous form. -**Database** -- [ ] DB row inserted before upload. -- [ ] `zarr_processed_path` updated to the correct S3 path after upload. -- [ ] `num_frames` in DB row matches `total_frames` in `zarr.attrs`. -- [ ] `embodiment` in DB row exactly matches the embodiment enum string (§9). +**Raw data** +- [ ] The raw recording of every episode is uploaded to `s3://rldb/raw///`, video included, at original resolution (§6.6). +- [ ] It contains what [`SPEC.md`](egomimic/rldb/mcap/SPEC.md) lists (calibration, timestamps, IMU, poses, §6.3 metadata); MCAP deliveries pass `python -m egomimic.rldb.mcap.validate`. **Upload** -- [ ] Episode is accessible at `s3://rldb/processed_v3//.zarr/`. -- [ ] `sync_s3.py` with an appropriate filter can download and open the episode. +- [ ] Episode and preview are at `s3://rldb/processed_v3//.zarr/` and `.mp4` (§10.1). --- @@ -1064,10 +874,10 @@ Complete every item before considering an episode ready for upload. ### Access Request -To get credentials for the EgoVerse data bucket and episode registry: +To get credentials for the EgoVerse data bucket: 1. Email the consortium leads with your lab name, GitHub handle, and a brief description of the data you intend to contribute. -2. You will receive AWS credentials (for Secrets Manager access) and instructions to run `setup_secret.sh`. +2. An RL2 team member confirms your `lab` string and sends you R2 credentials; upload only to your prefixes (§3.3, §10.1). ### Consortium Leads @@ -1093,4 +903,4 @@ To get credentials for the EgoVerse data bucket and episode registry: If you encounter processing errors, S3 permission issues, or schema questions, post in `#egoverse-onboarding` with: - Your episode hash(es) - The error message or symptom -- The output of `validate_episode()` for the affected episode +- The output of `python -m egomimic.rldb.zarr.validate` (or `egomimic.rldb.mcap.validate`) for the affected episode diff --git a/INTERNAL_DATA.md b/INTERNAL_DATA.md new file mode 100644 index 0000000000..e6b1854959 --- /dev/null +++ b/INTERNAL_DATA.md @@ -0,0 +1,245 @@ +# EgoVerse Data: Internal Notes (RL2) + +*For RL2 team members: ingest, the episode database, and the internal converters.* + +External contributors follow [CONTRIBUTING_DATA.md](CONTRIBUTING_DATA.md), the +vendor guide. Everything in that guide (episode hash, zarr format, metadata +rules, validation) applies to internal data too; this file covers only what +vendors cannot or need not do. A bare `§N` refers to a section of this file. + +--- + +## Table of Contents + +1. [Credentials](#1-credentials) +2. [Episode Database](#2-episode-database) +3. [Ingesting a Vendor Delivery](#3-ingesting-a-vendor-delivery) +4. [Internal Conversion](#4-internal-conversion) +5. [Training Data Configs](#5-training-data-configs) +6. [Internal S3 Paths and Uploads](#6-internal-s3-paths-and-uploads) +7. [Internal Checklist](#7-internal-checklist) + +--- + +## 1. Credentials + +RL2 members need AWS credentials for the episode registry (PostgreSQL via +Secrets Manager) in addition to the R2 bucket credentials: + +```bash +aws configure +# AccessKeyId / SecretAccessKey: ask the consortium lead +# Default region: us-east-2 +bash egomimic/utils/aws/setup_secret.sh +# Writes ~/.egoverse_env with R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY, +# AWS_ENDPOINT_URL_S3, SECRETS_ARN, etc. +``` + +Verify: + +```python +from egomimic.utils.aws.aws_data_utils import load_env +from egomimic.utils.aws.aws_sql import create_default_engine + +load_env() +engine = create_default_engine() # should print: Tables in schema 'app': ['episodes'] +``` + +--- + +## 2. Episode Database + +Every episode is registered in the PostgreSQL `app.episodes` table, the +authoritative index used by all download and training tooling. + +### 2.1 Schema + +The authoritative schema is the `TableRow` dataclass in +[egomimic/utils/aws/aws_sql.py](egomimic/utils/aws/aws_sql.py). Refer to that +file for the exact fields, defaults, and types. + +Key field notes: +- `episode_hash`: PRIMARY KEY, must match the `.zarr` directory name exactly (CONTRIBUTING_DATA.md §4). +- `operator`: stable per-collector ID; a pseudonymous ID is fine (CONTRIBUTING_DATA.md §6.3). +- `lab`: who the data came from; a full match of `^[a-z0-9][a-z0-9_-]*$`. Once set, do not change it (used in filters and partitioning). Rows that fail this rule are rejected by `run_conversion` (§4.2). +- `task`: the `task_name` that groups related episodes. Before adding a new name, check the existing tasks via [`sql_tutorial.ipynb`](egomimic/scripts/tutorials/sql_tutorial.ipynb) (`df.groupby("task").size()`) and reuse one if the episode fits. A new name is a short, stable, lowercase category (e.g. `fold_clothes`, `object_in_container`), not a one-off trial description. +- `objects`: stored as text, usually a Postgres array literal (`{"pink cup",bag}`); `run_conversion` parses it into a list (§4.2). +- `embodiment`: one of the strings in CONTRIBUTING_DATA.md §9. +- `rig_name`: the physical capture rig (e.g. `aria_gen1`, `eva`). Optional. + +### 2.2 Inserting a Row + +```python +from egomimic.utils.aws.aws_sql import TableRow, add_episode, create_default_engine +from egomimic.utils.aws.aws_data_utils import load_env + +load_env() +engine = create_default_engine() + +row = TableRow( + episode_hash = "2026-03-15-14-22-10-000000", + operator = "op_7", + lab = "rl2", + task = "fold_clothes", + embodiment = "human_bimanual", + rig_name = "aria_gen1", + task_description = "folding a 2T baby shirt on a blue table", + scene = "kitchen_A", + objects = "baby_shirt_2T", + num_frames = 2712, +) + +add_episode(engine, row) +``` + +`add_episode` raises `RuntimeError` on a duplicate `episode_hash`. Check for +collisions (`episode_hash_to_table_row(engine, hash)` returns `None`) before +inserting. + +### 2.3 Updating a Row After Upload + +```python +from egomimic.utils.aws.aws_sql import update_episode + +row.zarr_processed_path = "s3://rldb/processed_v3/aria/2026-03-15-14-22-10-000000.zarr" +row.num_frames = 2712 +update_episode(engine, row) +``` + +--- + +## 3. Ingesting a Vendor Delivery + +Vendors upload validated zarr episodes and previews to +`s3://rldb/processed_v3//` and raw recordings to +`s3://rldb/raw///` (CONTRIBUTING_DATA.md §10.1). They send no delivery note +or manifest. On ingest: + +1. Run `python -m egomimic.rldb.zarr.validate` on the episodes. Episodes with + `scene` or `objects` stored as `null` (waived with `MissingMetadata()`) pass + with `--allow-missing` and are accepted. Episodes without a `schema_version` + (written before the required metadata) only get warnings for metadata. +2. Map each vendor-proposed `task_name` to an existing task (§2.1), or add a new + task name if none fits. +3. Insert one DB row per episode (§2.2), with `lab`, `operator`, `scene`, + `objects` and the mapped `task` taken from the episode's `zarr.attrs`, and + `zarr_processed_path` / `zarr_mp4_path` set to the uploaded paths. +4. Add a training data config for the vendor if it is new (§5). + +--- + +## 4. Internal Conversion + +### 4.1 Converter CLIs + +`aria_to_zarr.py` and `eva_to_zarr.py` take the required metadata on the command +line through `egomimic.rldb.zarr.required_metadata.add_required_metadata_args`: +`--task-name`, `--operator`, `--lab`, `--scene`, `--objects`, and a repeatable +`--missing-metadata FIELD` to write `scene` or `objects` as `MissingMetadata()` +(`task_name`, `operator` and `lab` cannot be waived). None of them +has a usable default: a missing flag without a waiver fails when the episode is +written. `--save-mp4` writes the CONTRIBUTING_DATA.md §6.5 preview next to the `.zarr`. + +### 4.2 Batch conversion from the DB (`run_conversion.py`) + +```bash +python egomimic/scripts/run_conversion.py --embodiment aria # or eva +``` + +It walks the raw bundles, looks up each episode's DB row, and converts it on Ray. +The required metadata comes from the row: + +- `task` → `task_name`; `operator`, `lab`, `scene` as stored (stripped). +- `objects` is parsed from the Postgres array literal or comma-separated text; + `None` / `NULL` entries are dropped. +- An empty `scene` or `objects` is written as `MissingMetadata()`. An empty + `task`, `operator` or `lab` is a metadata error (below). +- A row whose metadata the writer would reject (a placeholder such as + `"unknown"` or `"debug"`, a non-identifier `lab`, ...) is skipped before + extraction, and `zarr_processing_error` is set to `Metadata: `. Fix + the row and clear `zarr_processing_error` to have it picked up again; rows + with a non-empty `zarr_processing_error` are always skipped. + +### 4.3 Disabled converters + +`egomimic/scripts/mecka_process/mecka_to_zarr.py` (`MeckaDatasetConverter` and +its CLI), `external/scale/scripts/sfs_to_egoverse_zarr.py` +(`convert_task_to_zarr`) and `external/scale/scripts/sfs_zarr_pipeline.py` raise +`NotImplementedError`. Tracking: #738 (mecka), #739 (Scale). They predate the required metadata and mandatory +intrinsics. Re-enabling either one means passing `intrinsics={"front_1": 3x4 K}`, +`task_name`, `operator`, `lab`, `scene` and `objects` to +`ZarrWriter.create_and_write` as real arguments (`metadata_override` may not +set keys the writer owns). Reconverted mecka zarrs also need the fixed +left-wrist convention (`Human.get_transform_list(fix_left_wrist_convention=...)`). +The mecka pose helpers (`left_wrist_convention` and friends) still work and are +used by training. + +--- + +## 5. Training Data Configs + +Every human contributor uses the single `Human` embodiment (CONTRIBUTING_DATA.md §9.1). Its per-vendor +structural choices are explicit arguments from the data config: + +- `Human.get_keymap(keymap_mode="cartesian"|"keypoints", has_head_pose=, include_aria_keypoints=)` +- `Human.get_transform_list(mode="cartesian_6d"|"keypoints_headframe_ypr"|..., stride=)` + +To onboard a new vendor, add a data config under +[`egomimic/hydra_configs/data/`](egomimic/hydra_configs/data/) whose `key_map` / +`transform_list` point at those with the args the data needs. Copy `aria.yaml` +(head-mounted, `stride: 3`) or `scale.yaml` (`stride: 1`). + +- `has_head_pose=False` if the data has no `obs_head_pose`; `stride` is the + action-chunk stride (`3` for ~30 fps egocentric, `1` for already-downsampled data). +- **Robots** subclass `Embodiment` directly and keep their own + intrinsics/extrinsics and pipeline — see `Eva` in + [`egomimic/rldb/embodiment/eva.py`](egomimic/rldb/embodiment/eva.py). +- For data annotated through Scale AI, `ScaleAnnotationDatasetFilter` + ([`egomimic/rldb/filters.py`](egomimic/rldb/filters.py)) keeps only episodes + with completed Scale annotations; it needs `SCALE_API_KEY`. + +--- + +## 6. Internal S3 Paths and Uploads + +Internal conversions upload under an embodiment prefix, set with +`PROCESSED_REMOTE_PREFIX` (defaults in `ray_helper.py`: +`s3://rldb/processed_v3/aria` and `s3://rldb/processed_v3/eva`). The preview MP4 +goes next to the zarr under the same prefix. + +Bulk upload with Ray: + +```python +import ray +from egomimic.utils.aws.aws_data_utils import upload_dir_to_s3, load_env + +ray.init() + +@ray.remote +def upload_one(local_zarr_path: str, s3_prefix: str): + load_env() + upload_dir_to_s3(local_zarr_path, bucket="rldb", prefix=s3_prefix) + +tasks = [ + upload_one.remote(f"/local/processed/{h}.zarr", f"processed_v3/aria/{h}.zarr") + for h in episode_hashes +] +ray.get(tasks) +``` + +Download for local work with +[`egomimic/scripts/data_download/sync_s3.py`](egomimic/scripts/data_download/sync_s3.py) +(`--local-dir`, optional named `--filters`). + +--- + +## 7. Internal Checklist + +In addition to the vendor checklist (CONTRIBUTING_DATA.md §12): + +- [ ] Episode hash is not already in the DB (`episode_hash_to_table_row(engine, hash)` returns `None`). +- [ ] DB row inserted, with `task` mapped to an existing task name (§3). +- [ ] `embodiment` and `task` in the DB row match `zarr.attrs` (`embodiment`, mapped `task_name`). +- [ ] `zarr_processed_path` and `zarr_mp4_path` point at the uploaded files. +- [ ] `num_frames` in the DB row matches `total_frames` in `zarr.attrs`. +- [ ] `sync_s3.py` with an appropriate filter can download and open the episode. diff --git a/README.md b/README.md index 6573e5419f..ea399a3751 100644 --- a/README.md +++ b/README.md @@ -166,4 +166,4 @@ python egomimic/trainHydra.py --config-name=train_zarr_cartesian For full instructions on training see [``training.md``](./training.md) ### Converting your own data -See [``embodiment_tutorial.ipynb``](./egomimic/scripts/tutorials/embodiment_tutorial.ipynb) as reference to write a conversion script for your own data. +Start from the reference converter [``template_to_zarr.py``](./egomimic/scripts/convert_template/template_to_zarr.py): copy it and fill in ``find_raw_episodes`` and ``load_episode`` for your raw data. It writes, validates and previews every episode. The format, required metadata, uploads and raw-data delivery are specified in [``CONTRIBUTING_DATA.md``](./CONTRIBUTING_DATA.md) (vendor guide); RL2 ingest and internal conversion are in [``INTERNAL_DATA.md``](./INTERNAL_DATA.md). diff --git a/egomimic/rldb/mcap/SPEC.md b/egomimic/rldb/mcap/SPEC.md new file mode 100644 index 0000000000..c5366cec7b --- /dev/null +++ b/egomimic/rldb/mcap/SPEC.md @@ -0,0 +1,295 @@ +# EgoVerse raw-data spec 0.1 (MCAP) + +The reference for **raw data**: what a raw recording should contain so we can rebuild any +training format from it later (Lance tables, training videos, new formats) without asking +the vendor anything. Vendors deliver **Zarr first**, which is what we train on +([CONTRIBUTING_DATA.md §6](../../../CONTRIBUTING_DATA.md)), and the raw recordings +alongside it (§6.6 there). Raw data may come in any format; this document, written as +an MCAP layout, lists the information and fields it should carry. A vendor who does +deliver MCAP follows it exactly: one raw, self-contained MCAP recording per episode. + +- **Raw, not processed.** Original sensor data at its own rates: no resampling, no + aligning streams to one clock rate, no undistorting, cropping or downscaling video. We + convert on our side. +- **Self-contained.** Calibration, camera positions and episode info live inside the + file. No side JSONs, no constants in our converters. +- **Checked on arrival.** Every MCAP episode is run through the validator + (`python -m egomimic.rldb.mcap.validate`), which lists what is missing or wrong. + +The machine-readable version of this document is [`spec.py`](spec.py); the message +definitions are [`proto/egoverse.proto`](proto/egoverse.proto) (generated from +[`schemas.py`](schemas.py)). The three change together. Proto field numbers are never +changed or reused; a new field gets a new number. + +Spec 0.1 covers **human (egocentric) recordings** and **Eva robot recordings with +grippers**. Dexterous hands, tactile, humanoid / auxiliary joints (e.g. Sharpa) are not +in 0.1. + +## 1. What arrives per episode + +``` +episode_/ +├── episode.mcap # everything: video, sensors, poses, calibration, episode info +└── annotation.mcap # optional: /task spans and /subtask-annotation labels +``` + +- **An episode is one whole recording session, never a pre-cut clip.** It becomes one + row of `recordings`; the tasks performed in it become `tasks` rows. +- The label topics may instead be inline in `episode.mcap` (as ABC-130k does), but not in + both files. `annotation.mcap` holds nothing but label topics: no metadata records + (the episode info lives in `episode.mcap` only). +- MCAP with protobuf encoding; any chunk compression (zstd, lz4) is fine. + +## 2. Clock + +- **Every message's MCAP log time is its capture time in absolute Unix nanoseconds**, in + both files. Devices that stamp on their own clock (e.g. Aria device time) convert + with their device-to-UTC offset before writing. +- Within a topic, log times never decrease; each camera's strictly increase (every + frame has its own time). +- `start_time_unix` / `end_time_unix` in the episode info are **milliseconds** and must + match the recorded data within 1 s. +- **Steps clock.** The main camera (`/main-camera`, human) or top camera (`/top-camera`, + robot) defines the Lance `steps` rows: one row per decoded frame. Every other stream is + aligned to those frame times during conversion; wrist and SLAM cameras stay separate + videos and add no rows. + +## 3. Who, what, where: episode info + +One MCAP metadata record named **`session-metadata`**. All values are strings. + +The first five keys are the required metadata of every EgoVerse episode, with the same +names and rules as a zarr episode ([CONTRIBUTING_DATA.md §6.3](../../../CONTRIBUTING_DATA.md)): +real values only (empty strings and placeholders such as `unknown`, `n/a`, `tbd`, `test` +are rejected). Keys are snake_case; a key not in this table fails validation, and +placeholders are rejected in every text key. + +| Key | Example | Meaning | +|---|---|---| +| `task_name` | `fold_clothes` | Task name; reuse an existing one if it fits. | +| `operator` | `op_07` | Stable ID of who performed it. | +| `lab` | `rl2` | Who the data came from: short, stable, lowercase (`a-z`, `0-9`, `_`, `-`), e.g. `rl2`, `mecka`. Never changes once chosen. | +| `scene` | `kitchen_a` | Where it was recorded. | +| `objects` | `baby_shirt_2t,basket` | Objects involved, comma-separated; every entry a real name. | +| `missing_metadata` | `scene` | *Optional.* Comma-separated required keys the source data truly does not have; leave those keys out. Only `scene` and `objects` can be listed: `task_name`, `operator` and `lab` are always required. **Only if absolutely necessary**: such episodes are much harder to filter and train on, and fail validation without `--allow-missing`. | +| `task_description` | `fold the 2T shirt on the blue table` | *Optional.* Full task text. | +| `source_recording_id` | `acme-rec-0001` | Your own ID for this recording. **Stable across every re-export or correction** (new labels, fixed calibration, re-run poses). Zarr episodes trimmed from it name it in their `source_recording_id` attr. | +| `embodiment` | `human_bimanual` | One of `human_{bimanual,left_arm,right_arm}`, `eva_{bimanual,left_arm,right_arm}`. Must match the topics in the file. | +| `rig_name` | `aria_gen1` | Capture setup. | +| `device_serial` | `1WM103600M1292` | Physical unit; ties the episode to its calibration. | +| `start_time_unix`, `end_time_unix` | `1736651249664` | Recording bounds, Unix **ms**. | +| `spec_version` | `0.1` | Version of this spec the file follows. | +| `license` | `CC-BY-4.0` | License the data is under. | +| `camera_timestamp_ref` | `mid` | *Human only.* Where in the exposure each frame is stamped: `start`, `mid` or `end`. | +| `shutter` | `rolling:5ms` | *Human only.* `global`, or `rolling:ms`. | +| `hand_joints` | see §6 | *Human only.* Names of the hand joints, in the order `HandKeypoints.positions` lists them. | + +**IDs.** Our recording ID is +`egomimic.rldb.lance.schema.recording_id(lab, source_recording_id)` (UUIDv5 of +`/`); the validator prints it. Task and annotation IDs are +computed downstream from their content, so vendors send none. + +**Tasks.** Without `/task` topics, `task_name` / `task_description` / `objects` describe one task +spanning the whole recording: [first steps frame, last steps frame + one frame), as +every span is half-open. With `/task`, the spans are authoritative and these keys +only summarize the session (e.g. `task_name=laundry` over iron / fold / pack spans); no extra +task is made from them. + +## 4. Topics + +Message types are Foxglove's (`foxglove-schemas-protobuf`) where one exists, otherwise +`egoverse.*` from `proto/egoverse.proto`. Topic names, message types and required fields +are checked exactly. + +### Human / egocentric + +| Topic | Type | | Rate | Notes | +|---|---|---|---|---| +| `/main-camera` | `foxglove.CompressedVideo` | MUST | ≥ 30 fps | Head-mounted RGB; the steps clock. | +| `/main-camera-info` | `foxglove.CameraCalibration` | MUST | once | See §7. | +| `/slam-left-camera`, `/slam-right-camera` | `foxglove.CompressedVideo` | optional | ≥ 30 fps | Each with its `-info`. | +| `/left-wrist-camera`, `/right-wrist-camera` | `foxglove.CompressedVideo` | optional | ≥ 30 fps | Each with its `-info`. | +| `/imu` | `egoverse.Imu` | MUST | ≥ 200 Hz | Raw: gravity included, no fusion. | +| `/imu-info` | `egoverse.ImuInfo` | MUST | once | Noise model; datasheet values are fine. | +| `/head-pose` | `foxglove.PoseInFrame` | MUST | ≥ 30 Hz | `frame_id = "world"`, pose = T_world_head. | +| `/head-pose-quality` | `egoverse.TrackingQuality` | MUST | one per head pose | `OK`, `LOST`, `RELOCALIZED` (first pose after a SLAM reset) or `UNKNOWN`. | +| `/{left,right}-hand-keypoints` | `egoverse.HandKeypoints` | MUST per arm in use | tracker rate | See §6. | +| `/eye-gaze` | `egoverse.EyeGaze` | optional | tracker rate | Yaw / pitch / depth in a frame between the eyes (+x left, +y up, +z forward). | +| `/tf_static` | `foxglove.FrameTransforms` | MUST | once | See §5. | + +No arm, gripper or action topics: actions are derived from future hand and head poses. + +### Robot (Eva with grippers) + +`{side}` is `left` and/or `right`, as the embodiment uses. + +| Topic | Type | | Notes | +|---|---|---|---| +| `/top-camera` | `foxglove.CompressedVideo` | MUST | ≥ 30 fps; the steps clock. With `-info`. | +| `/{side}-wrist-camera` | `foxglove.CompressedVideo` | MUST | ≥ 30 fps. With `-info`. | +| `/{side}-arm-state` | `egoverse.ArmState` | MUST | Measured joints (names, position, velocity, torque) and `ee_pose`. | +| `/{side}-arm-action` | `egoverse.ArmAction` | MUST | The command sent at that time: target joints and `ee_pose`. | +| `/{side}-ee-state` | `egoverse.GripperState` | MUST | Measured opening, 0 = closed, 1 = fully open. | +| `/{side}-ee-action` | `egoverse.GripperState` | MUST | Commanded opening, same range. | +| `/tf_static` | `foxglove.FrameTransforms` | MUST | See §5. | + +`ArmState` / `ArmAction`: `frame_id` is the arm-base frame (`left_arm_base`, +`right_arm_base`); every joint array has one entry per `joint_names` entry, in that +order; joints in rad (m for prismatic), velocities rad/s, torques N·m; `ee_pose` is +T_base_ee as 16 numbers, a **row-major** 4×4. ABC-130k's own `RobotState` / +`GripperState` are converted to these types before validation. + +### Both profiles: tasks and labels (optional) + +| Topic | Type | Notes | +|---|---|---| +| `/task` | `egoverse.Task` | One message per task performed: `name` (slug), `description`, `objects`; starts at its log time, ends at `end_time_unix_ns` (after the start). | +| `/subtask-annotation` | `egoverse.Annotation` | A language label `data` over [log time, `end_time_unix_ns`); `end_time_unix_ns = 0` is a point event (one frame wide). `annotation_type` is `language` (the default when empty); `annotation_source` defaults to the episode's `lab`. | + +Every label lies **inside one task**; without `/task`, inside the whole-recording task +made from the episode info. All spans lie inside the recorded data. Labels added or corrected after delivery go through +the Lance annotation review (`egomimic/rldb/lance/annotations.py`), not a re-export. + +## 5. Frames and conventions + +- Right-handed frames; metres, radians, seconds. +- Quaternions are Foxglove's named `x, y, z, w` fields and must be unit length. +- A transform or pose is **T_parent_child**: the child frame's pose in the parent + (Foxglove's `FrameTransform` / `PoseInFrame` convention). +- Every camera frame is a Foxglove optical frame: +x right, +y down, +z forward, in the + raw sensor orientation (see `upright_rotation_cw_deg`, §7). A camera's video, + `-info` and `-info-extra` all name that one frame. +- **Every frame any topic names must reach `world`** through `/tf_static` plus the + moving links the profile defines: + - **Human:** `world → head` moves via `/head-pose`. `/tf_static` links `head` to each + camera, the IMU, the gaze frame, and any frame hands are given in. The main camera + must be rigidly mounted on the head: a `/tf_static` path from `head` to its frame. + - **Robot:** `world → {side}_arm_base` and the top camera in `/tf_static`; + `{side}_arm_base → {side}_ee` moves via `/{side}-arm-state`; the wrist camera hangs + off `{side}_ee` in `/tf_static`. + +## 6. Hands + +`HandKeypoints`, one message per hand per tracker frame: + +| Field | | +|---|---| +| `frame_id` | Frame `positions` and `wrist_pose` are in (linked to `world`). | +| `positions` | `(x, y, z)` in m per `hand_joints` entry, flattened; or empty for devices without keypoints. | +| `confidence` | Whole-hand tracking confidence in [0, 1]. | +| `joint_confidence` | Optional: one value in [0, 1] per joint. | +| `wrist_pose` | Optional: T_frame_wrist. Required when `positions` is empty. | +| `valid` | `false` when the hand is not tracked. **Never write zeros instead.** | + +**`hand_joints`** names the joints the device tracks, in its own order, from this +vocabulary: the 21 MANO joints `wrist`, `{thumb}_{cmc,mcp,ip,tip}`, +`{index,middle,ring,pinky}_{mcp,pip,dip,tip}`, plus `palm_center`. Each name at most +once; `wrist` is required. Joints with MANO names are what our MANO fit matches; extras +are kept but not fitted. A device that already outputs MANO sends the canonical order: + +``` +wrist,thumb_cmc,thumb_mcp,thumb_ip,thumb_tip,index_mcp,index_pip,index_dip,index_tip, +middle_mcp,middle_pip,middle_dip,middle_tip,ring_mcp,ring_pip,ring_dip,ring_tip, +pinky_mcp,pinky_pip,pinky_dip,pinky_tip +``` + +and a device with its own layout lists it as is, e.g. Aria MPS's 21 landmarks (no thumb +CMC, a palm center instead): + +``` +thumb_tip,index_tip,middle_tip,ring_tip,pinky_tip,wrist,thumb_mcp,thumb_ip, +index_mcp,index_pip,index_dip,middle_mcp,middle_pip,middle_dip,ring_mcp,ring_pip, +ring_dip,pinky_mcp,pinky_pip,pinky_dip,palm_center +``` + +## 7. Cameras + +- **Video.** H.264 or H.265, Annex B byte stream, `CompressedVideo.format` `h264` / + `h265`. **Each message is exactly one frame**, stamped with that frame's capture time; + the first message is a keyframe and carries SPS/PPS (repeat them at every keyframe). + **No B-frames**, so messages decode one-to-one into frames in timestamp order (as in + `foxglove.CompressedVideo`); with x264 pass `-bf 0`, with x265 `bframes=0`. + Original resolution, frame rate and orientation: no resizing, cropping, undistorting + or rotating. Send the video as the device stores it, or lightly compressed (H.264 / + H.265 at a high-quality setting), encoded at most once. +- **Calibration** `-info`: exactly one `foxglove.CameraCalibration` message, + finite values, principal point inside the image: + `width` / `height` equal the decoded frames; `K` has fx, fy > 0. Name the lens model in + `distortion_model` and give exactly its number of coefficients in `D` (an empty model + with no coefficients is a pinhole camera): + + | `distortion_model` | `K` | `D` | + |---|---|---| + | `plumb_bob` | fx, fy, cx, cy | k1, k2, p1, p2, k3 | + | `rational_polynomial` | fx, fy, cx, cy | k1, k2, p1, p2, k3, k4, k5, k6 | + | `kannala_brandt` | fx, fy, cx, cy | k1, k2, k3, k4 | + | `fisheye624` (Aria) | f, f, cx, cy | the 12 coefficients after f, cx, cy, in projectaria order (k0–k5, p0, p1, s0–s3) | + +- **Optional** `-info-extra`, `egoverse.CameraInfoExtra`, published once: + `valid_radius` (px from the principal point where the lens model holds; 0 = whole + image) and `upright_rotation_cw_deg` (0, 90, 180 or 270: the clockwise turn that makes + a frame upright, e.g. 90 for Aria's sideways RGB sensor). + +## 8. Please don't + +- Resize, crop, rotate or undistort video, or re-encode it more than once. One light + compression pass is fine. *(Devices that store frames as images, e.g. Aria VRS, + encode them once to H.264 / H.265 at full resolution.)* +- Pack several frames into one video message. +- Resample or align streams to one rate; we match by timestamp. +- Trim streams to the camera's span; deliver every sample. +- Write zeros when tracking is lost; set `valid = false`. +- Remove gravity from, filter or fuse the IMU data. +- Ship calibration as a separate file. + +## 9. How a delivery is checked + +`python -m egomimic.rldb.mcap.validate episode_/ [...]` prints `PASS` or every +problem per episode (with the episode's `recording_id`), and exits 1 if any fails. Every +problem fails the episode. + +| Group | Checks | +|---|---| +| everything there | Chunk CRCs valid; required topics present with the exact type and encoding, one channel each; once-topics published once; required fields non-empty on every message; exactly one `session-metadata` record; every episode-info key known and present, values not placeholders, the required metadata real values (waivers in `missing_metadata` fail unless `--allow-missing`; only `scene` and `objects` can be waived), integers / enums valid; `embodiment` matches the topics; `annotation.mcap` holds only label topics. | +| video usable | Codec exactly `h264` / `h265`, the same on every message; every message holds a coded picture; first message a keyframe; no B-frames; one frame per message; frames decode without bitstream errors; size matches the calibration; ≥ 30 fps. | +| clocks | Unix ns (after 2015); never decreasing; no gap over 0.5 s; streams cover the recording to within 1 s; start / end times match the data to within 1 s; head pose and quality ≥ 30 Hz. | +| calibration | Every camera has a valid `-info` with a known lens model and its coefficient count; `-info-extra` values valid; a camera's video and calibration name the same frame; every named frame reaches `world`; a head-worn rig's main camera has a `/tf_static` path to `head`; `/tf_static` rotations unit and translations finite. | +| imu | ≥ 200 Hz; mean \|accel\| in the stillest 1 s within 0.5 m/s² of 9.81 (gravity not removed); noise values > 0. | +| poses | No NaN / Inf, unit rotations (head, wrist, `/tf_static`); hand positions match `hand_joints`; confidences in [0, 1]; no all-zero hands marked valid; gaze finite when valid; `ee_pose` a rigid 4×4; joint arrays match `joint_names`; grippers in [0, 1]. | +| tasks & labels | Spans end after they start (labels may be points); inside the recorded data; each label inside one task; `annotation_type` known. | + +The 30 fps, 200 Hz, 30 Hz and 9.81 m/s² thresholds are fixed. The others +(5 % rate tolerance, 0.5 s gap, 1 s span tolerance, 0.5 m/s² gravity tolerance, 1e-3 +unit tolerance) are **provisional, to be tuned on real deliveries**; all live in +`spec.py`. Structural video checks (non-empty, frame_id, format, holds a picture) run on +every message; full decoding runs on the first 60 messages of each camera. The converter +then decodes all of it and rejects an episode where any message is not exactly one frame. + +## 10. Where each Lance field comes from + +| Lance | MCAP | +|---|---| +| `recordings.recording_id` | `recording_id(lab, source_recording_id)` | +| `recordings.vendor_recording_id` | `source_recording_id` | +| `recordings.lab`, `operator`, `embodiment`, `rig_name`, `scene`, `license` | episode info | +| `recordings.captured_at` | `start_time_unix` | +| `recordings.num_frames`, `fps` | Decoded steps-clock frames and their timestamps | +| `recordings.video_path` | Assigned when each camera topic is extracted | +| `recordings.intrinsics` / `extrinsics` | Each camera's `-info` (+ `-info-extra`) / `/tf_static` and the moving links | +| `tasks.*` | `/task` messages, or the episode-info task keys over the whole recording | +| `annotations.*` | `/subtask-annotation` (type, text, source, span) | +| `steps.timestamp_ns` | Steps-clock frame times | +| `steps.head_pose`, `head_pose_valid` | `/head-pose` (× the camera's link), `/head-pose-quality`, timing gaps | +| `steps.{side}.keypoints`, `ee_pose` | `HandKeypoints.positions`, through the MANO fit when `hand_joints` is not canonical MANO | +| `steps.{side}.wrist_pose`, `confidence`, `{side}_valid` | `HandKeypoints.wrist_pose`, `confidence`, `valid` | +| `steps.eye_gaze` | `/eye-gaze` yaw, pitch, depth | +| Eva steps `obs_*` / `cmd_*` | `/{side}-arm-state` / `-arm-action` (`position`, `ee_pose`), `/{side}-ee-state` / `-ee-action` | + +## 11. Not in 0.1 + +Deliberately left out of this version, to revisit with real deliveries: + +- Graded SLAM quality (Aria MPS gives 0 / 0.5 / 1), beyond the `TrackingQuality` states. +- More than one IMU per device. +- Video from devices that store per-frame images: re-encode (as now) or accept them. +- Dexterous hands, tactile and auxiliary joints (Sharpa). diff --git a/egomimic/rldb/mcap/__init__.py b/egomimic/rldb/mcap/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/egomimic/rldb/mcap/proto/egoverse.proto b/egomimic/rldb/mcap/proto/egoverse.proto new file mode 100644 index 0000000000..70624059cd --- /dev/null +++ b/egomimic/rldb/mcap/proto/egoverse.proto @@ -0,0 +1,132 @@ +// Generated by `python -m egomimic.rldb.mcap.schemas`; do not edit by hand. +// EgoVerse raw data spec 0.1: message types not covered by foxglove schemas. + +syntax = "proto3"; + +package egoverse; + +// A 3D vector. +message Vector3 { + double x = 1; + double y = 2; + double z = 3; +} + +// A rotation; must be unit length. +message Quaternion { + double x = 1; + double y = 2; + double z = 3; + double w = 4; +} + +// A child frame's pose in its parent (T_parent_child): m and a unit quaternion. +message Pose { + Vector3 position = 1; + Quaternion orientation = 2; +} + +// Optional per-camera facts foxglove.CameraCalibration has no field for, on the +// camera's -info-extra, published once next to -info. +message CameraInfoExtra { + string frame_id = 1; // the camera's frame, as in its -info + double valid_radius = 2; // px from the principal point where the lens model holds (fisheye); 0 = the whole image + int32 upright_rotation_cw_deg = 3; // clockwise turn (0, 90, 180, 270) that makes a frame upright, for sensors mounted sideways +} + +// One raw IMU sample, straight off the device: no gravity removal, no fusion. +message Imu { + string frame_id = 1; // IMU frame; must be linked in /tf_static + Vector3 linear_acceleration = 2; // m/s^2, gravity included + Vector3 angular_velocity = 3; // rad/s +} + +// IMU noise model, as SLAM/VIO tools expect it. Published once. Datasheet values +// are fine when the device does not calibrate them. +message ImuInfo { + string frame_id = 1; + double accel_noise_density = 2; // m/s^2/sqrt(Hz) + double accel_random_walk = 3; // m/s^3/sqrt(Hz) + double gyro_noise_density = 4; // rad/s/sqrt(Hz) + double gyro_random_walk = 5; // rad/s^2/sqrt(Hz) +} + +// State of the vendor's SLAM at this time, one per head pose. +message TrackingQuality { + enum State { + UNKNOWN = 0; + OK = 1; + LOST = 2; + RELOCALIZED = 3; + } + State state = 1; +} + +// One hand at one time: its joints and/or the wrist pose. Joints are in the order +// the episode's hand-joints key lists them (names from spec.HAND_JOINTS). Set +// valid = false when the hand is not tracked; never write zeros in its place. +// A device without keypoints leaves positions empty and sets wrist_pose. +message HandKeypoints { + string frame_id = 1; // frame positions and wrist_pose are in + repeated double positions = 2; // (x, y, z) in m per hand-joints entry, flattened; or empty + float confidence = 3; // whole-hand tracking confidence in [0, 1] + bool valid = 4; // hand tracked and its values usable + repeated float joint_confidence = 5; // optional: per-joint confidences in [0, 1], one per joint + Pose wrist_pose = 6; // optional: T_frame_wrist +} + +// Where the wearer looks, as in Aria MPS eye gaze. frame_id is centered between +// the eyes with +x to the wearer's left, +y up, +z forward; the gaze point is +// depth * normalize(tan(yaw), tan(pitch), 1). Set valid = false when gaze is not +// tracked; never write zeros in its place. +message EyeGaze { + string frame_id = 1; // gaze frame; linked in /tf_static + double yaw = 2; // rad, positive = looking left + double pitch = 3; // rad, positive = looking up + double depth = 4; // m, distance to the gaze point + bool valid = 5; // gaze tracked and values usable +} + +// Measured state of one arm. Joint arrays follow joint_names. ee_pose is +// T_base_ee: the end effector in the arm-base frame (frame_id), row-major 4x4. +message ArmState { + string frame_id = 1; // arm-base frame, e.g. "left_arm_base" + repeated string joint_names = 2; + repeated double position = 3; // rad (or m, prismatic) + repeated double velocity = 4; // rad/s (or m/s) + repeated double torque = 5; // N*m (or N) + repeated double ee_pose = 6; // 16 values, row-major +} + +// Command sent to one arm at this time (the target it was told to reach), +// same conventions as ArmState. +message ArmAction { + string frame_id = 1; // arm-base frame + repeated string joint_names = 2; + repeated double position = 3; // rad (or m, prismatic) + repeated double ee_pose = 4; // 16 values, row-major +} + +// Gripper opening, measured (ee-state) or commanded (ee-action). +message GripperState { + double position = 1; // 0 = closed, 1 = fully open +} + +// One task performed in the recording; a recording may hold several +// (e.g. iron, then fold, then pack). Starts at the message's log time. +message Task { + string name = 1; // task slug, e.g. "fold_clothes" + string description = 2; // free text for this instance + repeated string objects = 3; // object slugs involved + int64 end_time_unix_ns = 4; // task end, absolute Unix ns +} + +// A label over [start, end), start being the message's log time; it must lie +// inside one /task (or the recording, without /task). Field `data` matches +// ABC-130k's /subtask-annotation. +message Annotation { + string data = 1; // English text + int64 end_time_unix_ns = 2; // end, absolute Unix ns; 0 = point event (one frame wide) + string annotation_type = 3; // default "language" + string annotation_source = 4; // who wrote it; default the episode's lab +} diff --git a/egomimic/rldb/mcap/schemas.py b/egomimic/rldb/mcap/schemas.py new file mode 100644 index 0000000000..6247ac24de --- /dev/null +++ b/egomimic/rldb/mcap/schemas.py @@ -0,0 +1,324 @@ +"""EgoVerse's own MCAP message types (raw data spec 0.1). + +Only what Foxglove's schemas do not cover. Video, camera calibration, transforms and head +pose use `foxglove.CompressedVideo`, `foxglove.CameraCalibration`, `foxglove.FrameTransforms` +and `foxglove.PoseInFrame` (package `foxglove-schemas-protobuf`). Robot vendors use the +ArmState / ArmAction / GripperState below; ABC-130k's own robot types are normalized to +them before validation. + +The messages are declared once, here, and built into a descriptor pool at import time, so +no `protoc` step is needed. `render_proto()` prints the same definitions as the +`proto/egoverse.proto` file handed to vendors; a unit test keeps the two in sync. + +A message's time is its MCAP log time (absolute Unix ns), so none of them carry a timestamp; +spans (Task, Annotation) add only their end time. + + python -m egomimic.rldb.mcap.schemas > egomimic/rldb/mcap/proto/egoverse.proto +""" + +from dataclasses import dataclass, field + +from google.protobuf import descriptor_pb2, descriptor_pool, message_factory + +PACKAGE = "egoverse" +PROTO_FILE = "egoverse.proto" + +_F = descriptor_pb2.FieldDescriptorProto +_SCALARS = { + "double": _F.TYPE_DOUBLE, + "float": _F.TYPE_FLOAT, + "int64": _F.TYPE_INT64, + "string": _F.TYPE_STRING, + "int32": _F.TYPE_INT32, + "bool": _F.TYPE_BOOL, +} + + +@dataclass(frozen=True) +class Field: + name: str + kind: str # a scalar above, an enum of the same message, or another message here + number: int # wire tag: never changed or reused; a new field takes a new number + repeated: bool = False + doc: str = "" + + +@dataclass(frozen=True) +class Enum: + name: str + values: tuple[ + str, ... + ] # numbered 0, 1, ...; proto3 needs the 0 value to mean "unset" + + +@dataclass(frozen=True) +class Message: + name: str + doc: str + fields: tuple[Field, ...] + enums: tuple[Enum, ...] = field(default=()) + + +MESSAGES = ( + Message( + "Vector3", + "A 3D vector.", + (Field("x", "double", 1), Field("y", "double", 2), Field("z", "double", 3)), + ), + Message( + "Quaternion", + "A rotation; must be unit length.", + ( + Field("x", "double", 1), + Field("y", "double", 2), + Field("z", "double", 3), + Field("w", "double", 4), + ), + ), + Message( + "Pose", + "A child frame's pose in its parent (T_parent_child): m and a unit quaternion.", + (Field("position", "Vector3", 1), Field("orientation", "Quaternion", 2)), + ), + Message( + "CameraInfoExtra", + "Optional per-camera facts foxglove.CameraCalibration has no field for, on the\n" + "camera's -info-extra, published once next to -info.", + ( + Field("frame_id", "string", 1, doc="the camera's frame, as in its -info"), + Field( + "valid_radius", + "double", + 2, + doc="px from the principal point where the lens model holds (fisheye); " + "0 = the whole image", + ), + Field( + "upright_rotation_cw_deg", + "int32", + 3, + doc="clockwise turn (0, 90, 180, 270) that makes a frame upright, for " + "sensors mounted sideways", + ), + ), + ), + Message( + "Imu", + "One raw IMU sample, straight off the device: no gravity removal, no fusion.", + ( + Field( + "frame_id", "string", 1, doc="IMU frame; must be linked in /tf_static" + ), + Field("linear_acceleration", "Vector3", 2, doc="m/s^2, gravity included"), + Field("angular_velocity", "Vector3", 3, doc="rad/s"), + ), + ), + Message( + "ImuInfo", + "IMU noise model, as SLAM/VIO tools expect it. Published once. Datasheet values\n" + "are fine when the device does not calibrate them.", + ( + Field("frame_id", "string", 1), + Field("accel_noise_density", "double", 2, doc="m/s^2/sqrt(Hz)"), + Field("accel_random_walk", "double", 3, doc="m/s^3/sqrt(Hz)"), + Field("gyro_noise_density", "double", 4, doc="rad/s/sqrt(Hz)"), + Field("gyro_random_walk", "double", 5, doc="rad/s^2/sqrt(Hz)"), + ), + ), + Message( + "TrackingQuality", + "State of the vendor's SLAM at this time, one per head pose.", + (Field("state", "State", 1),), + enums=(Enum("State", ("UNKNOWN", "OK", "LOST", "RELOCALIZED")),), + ), + Message( + "HandKeypoints", + "One hand at one time: its joints and/or the wrist pose. Joints are in the order\n" + "the episode's hand-joints key lists them (names from spec.HAND_JOINTS). Set\n" + "valid = false when the hand is not tracked; never write zeros in its place.\n" + "A device without keypoints leaves positions empty and sets wrist_pose.", + ( + Field("frame_id", "string", 1, doc="frame positions and wrist_pose are in"), + Field( + "positions", + "double", + 2, + repeated=True, + doc="(x, y, z) in m per hand-joints entry, flattened; or empty", + ), + Field( + "confidence", + "float", + 3, + doc="whole-hand tracking confidence in [0, 1]", + ), + Field("valid", "bool", 4, doc="hand tracked and its values usable"), + Field( + "joint_confidence", + "float", + 5, + repeated=True, + doc="optional: per-joint confidences in [0, 1], one per joint", + ), + Field("wrist_pose", "Pose", 6, doc="optional: T_frame_wrist"), + ), + ), + Message( + "EyeGaze", + "Where the wearer looks, as in Aria MPS eye gaze. frame_id is centered between\n" + "the eyes with +x to the wearer's left, +y up, +z forward; the gaze point is\n" + "depth * normalize(tan(yaw), tan(pitch), 1). Set valid = false when gaze is not\n" + "tracked; never write zeros in its place.", + ( + Field("frame_id", "string", 1, doc="gaze frame; linked in /tf_static"), + Field("yaw", "double", 2, doc="rad, positive = looking left"), + Field("pitch", "double", 3, doc="rad, positive = looking up"), + Field("depth", "double", 4, doc="m, distance to the gaze point"), + Field("valid", "bool", 5, doc="gaze tracked and values usable"), + ), + ), + Message( + "ArmState", + "Measured state of one arm. Joint arrays follow joint_names. ee_pose is\n" + "T_base_ee: the end effector in the arm-base frame (frame_id), row-major 4x4.", + ( + Field("frame_id", "string", 1, doc='arm-base frame, e.g. "left_arm_base"'), + Field("joint_names", "string", 2, repeated=True), + Field("position", "double", 3, repeated=True, doc="rad (or m, prismatic)"), + Field("velocity", "double", 4, repeated=True, doc="rad/s (or m/s)"), + Field("torque", "double", 5, repeated=True, doc="N*m (or N)"), + Field("ee_pose", "double", 6, repeated=True, doc="16 values, row-major"), + ), + ), + Message( + "ArmAction", + "Command sent to one arm at this time (the target it was told to reach),\n" + "same conventions as ArmState.", + ( + Field("frame_id", "string", 1, doc="arm-base frame"), + Field("joint_names", "string", 2, repeated=True), + Field("position", "double", 3, repeated=True, doc="rad (or m, prismatic)"), + Field("ee_pose", "double", 4, repeated=True, doc="16 values, row-major"), + ), + ), + Message( + "GripperState", + "Gripper opening, measured (ee-state) or commanded (ee-action).", + (Field("position", "double", 1, doc="0 = closed, 1 = fully open"),), + ), + Message( + "Task", + "One task performed in the recording; a recording may hold several\n" + "(e.g. iron, then fold, then pack). Starts at the message's log time.", + ( + Field("name", "string", 1, doc='task slug, e.g. "fold_clothes"'), + Field("description", "string", 2, doc="free text for this instance"), + Field("objects", "string", 3, repeated=True, doc="object slugs involved"), + Field("end_time_unix_ns", "int64", 4, doc="task end, absolute Unix ns"), + ), + ), + Message( + "Annotation", + "A label over [start, end), start being the message's log time; it must lie\n" + "inside one /task (or the recording, without /task). Field `data` matches\n" + "ABC-130k's /subtask-annotation.", + ( + Field("data", "string", 1, doc="English text"), + Field( + "end_time_unix_ns", + "int64", + 2, + doc="end, absolute Unix ns; 0 = point event (one frame wide)", + ), + Field("annotation_type", "string", 3, doc='default "language"'), + Field( + "annotation_source", + "string", + 4, + doc="who wrote it; default the episode's lab", + ), + ), + ), +) + + +def file_descriptor_proto() -> descriptor_pb2.FileDescriptorProto: + fp = descriptor_pb2.FileDescriptorProto( + name=PROTO_FILE, package=PACKAGE, syntax="proto3" + ) + for m in MESSAGES: + mp = fp.message_type.add(name=m.name) + for e in m.enums: + ep = mp.enum_type.add(name=e.name) + for number, value in enumerate(e.values): + ep.value.add(name=value, number=number) + enum_names = {e.name for e in m.enums} + for f in m.fields: + label = _F.LABEL_REPEATED if f.repeated else _F.LABEL_OPTIONAL + fd = mp.field.add(name=f.name, number=f.number, label=label) + if f.kind in _SCALARS: + fd.type = _SCALARS[f.kind] + elif f.kind in enum_names: + fd.type = _F.TYPE_ENUM + fd.type_name = f".{PACKAGE}.{m.name}.{f.kind}" + else: + fd.type = _F.TYPE_MESSAGE + fd.type_name = f".{PACKAGE}.{f.kind}" + return fp + + +# Own pool, so importing this module never collides with another egoverse.proto. +_pool = descriptor_pool.DescriptorPool() +_pool.Add(file_descriptor_proto()) + + +def _message_class(name: str): + return message_factory.GetMessageClass( + _pool.FindMessageTypeByName(f"{PACKAGE}.{name}") + ) + + +Vector3 = _message_class("Vector3") +Quaternion = _message_class("Quaternion") +Pose = _message_class("Pose") +CameraInfoExtra = _message_class("CameraInfoExtra") +Imu = _message_class("Imu") +ImuInfo = _message_class("ImuInfo") +TrackingQuality = _message_class("TrackingQuality") +HandKeypoints = _message_class("HandKeypoints") +EyeGaze = _message_class("EyeGaze") +ArmState = _message_class("ArmState") +ArmAction = _message_class("ArmAction") +GripperState = _message_class("GripperState") +Task = _message_class("Task") +Annotation = _message_class("Annotation") + + +def render_proto() -> str: + """The `.proto` text for MESSAGES, as handed to vendors.""" + lines = [ + "// Generated by `python -m egomimic.rldb.mcap.schemas`; do not edit by hand.", + "// EgoVerse raw data spec 0.1: message types not covered by foxglove schemas.", + "", + 'syntax = "proto3";', + "", + f"package {PACKAGE};", + ] + for m in MESSAGES: + lines.append("") + lines += [f"// {line}" for line in m.doc.split("\n")] + lines.append(f"message {m.name} {{") + for e in m.enums: + lines.append(f" enum {e.name} {{") + lines += [f" {v} = {n};" for n, v in enumerate(e.values)] + lines.append(" }") + for f in m.fields: + prefix = "repeated " if f.repeated else "" + comment = f" // {f.doc}" if f.doc else "" + lines.append(f" {prefix}{f.kind} {f.name} = {f.number};{comment}") + lines.append("}") + return "\n".join(lines) + "\n" + + +if __name__ == "__main__": + print(render_proto(), end="") diff --git a/egomimic/rldb/mcap/spec.py b/egomimic/rldb/mcap/spec.py new file mode 100644 index 0000000000..9ed4d4a430 --- /dev/null +++ b/egomimic/rldb/mcap/spec.py @@ -0,0 +1,367 @@ +"""Raw-data MCAP spec 0.1 in machine-readable form: what `validate.py` checks against. + +SPEC.md is the human-readable version of this file; change both together. Numbers the +plan states are used as-is; the ones it does not state are marked PROVISIONAL, to be tuned +on real deliveries. + +An episode is one whole recording session, never a pre-cut clip: it becomes one row of +the Lance `recordings` table, and the tasks inside it become `tasks` rows. + + episode_/ + ├── episode.mcap # everything: video, sensors, poses, calibration, episode info + └── annotation.mcap # optional: /task spans and /subtask-annotation labels + +Every message is stamped with absolute Unix time in nanoseconds (its MCAP log time), in +both files. Every camera is a topic ending in `-camera`; its calibration is on the sibling +`-info`. Each video message holds exactly one frame, stamped with its capture time. + +The profile's `step_clock` camera defines the Lance steps rows: one row per decoded frame +of it, every other stream aligned to those times during conversion. Other cameras stay +separate videos and add no rows. + +Who, what and where: the episode info carries the same required fields as a zarr +episode (task_name, operator, lab, scene, objects; CONTRIBUTING_DATA.md §6.3), named and +checked exactly as in egomimic.rldb.zarr.required_metadata. `lab` is who the data came +from (rl2, mecka, microagi). Hardware goes in rig_name / device_serial. Task text is +task_name, task_description and the annotations; there is no separate instruction. + +IDs: Lance recording_id = recording_id(lab, source_recording_id) below, so +source_recording_id must stay the same across every re-export or correction of the +recording. A zarr episode trimmed from it names it in its source_recording_id attr. Task +and annotation IDs are computed downstream from their content. +""" + +import re +import uuid +from dataclasses import dataclass + +from egomimic.rldb.zarr.episode_rules import EMBODIMENTS, QUAT_UNIT_TOL, arm_sides +from egomimic.rldb.zarr.required_metadata import ( + REQUIRED_METADATA_FIELDS, + WAIVABLE_FIELDS, +) + +SPEC_VERSION = "0.1" +SUPPORTED_SPEC_VERSIONS = ("0.1",) + +EPISODE_FILE = "episode.mcap" +# Its topics may instead be inline in episode.mcap, as ABC-130k does. +ANNOTATION_FILE = "annotation.mcap" +# ABC's delivery format names its episode-info record this; keys are snake_case. +METADATA_RECORD = "session-metadata" +MISSING_METADATA_KEY = "missing_metadata" + +CAMERA_SUFFIX = "-camera" +CALIBRATION_SUFFIX = "-info" +# Optional egoverse.CameraInfoExtra (valid radius, upright rotation) next to the -info. +CAMERA_EXTRA_SUFFIX = "-info-extra" +UPRIGHT_ROTATIONS = (0, 90, 180, 270) +# CameraCalibration.distortion_model -> number of D coefficients. An empty model with no +# coefficients is a pinhole camera. +DISTORTION_MODELS = { + "plumb_bob": 5, + "rational_polynomial": 8, + "kannala_brandt": 4, + "fisheye624": 12, +} + +# Hand joint vocabulary. Vendors list the joints their device tracks, in the order they +# write them, in the "hand_joints" key. Names that are MANO joints are what the MANO fit +# (egomimic/scripts/aria_process/aria_utils.py) matches; extras are kept but not fitted. +# Canonical MANO order, as Human.FINGER_EDGES uses it: a vendor sending this exactly +# needs no fit. +MANO_JOINTS = ( + "wrist", + *( + f"{finger}_{joint}" + for finger, joints in ( + ("thumb", ("cmc", "mcp", "ip", "tip")), + ("index", ("mcp", "pip", "dip", "tip")), + ("middle", ("mcp", "pip", "dip", "tip")), + ("ring", ("mcp", "pip", "dip", "tip")), + ("pinky", ("mcp", "pip", "dip", "tip")), + ) + for joint in joints + ), +) +EXTRA_HAND_JOINTS = ("palm_center",) +HAND_JOINTS = MANO_JOINTS + EXTRA_HAND_JOINTS +# Shutter: global, or rolling with its readout time. +SHUTTER = re.compile(r"global|rolling:[0-9]+(\.[0-9]+)?ms") + +# Copied from egomimic/rldb/lance/schema.py (PR #716) so the validator needs no Lance +# install; a unit test checks the two agree once that module is on main. +RECORDING_NAMESPACE = uuid.UUID("6f1d4c3e-2b1a-5e7f-9c8d-0a1b2c3d4e5f") + + +def recording_id(lab: str, source_recording_id: str) -> str: + """The Lance recordings.recording_id this episode will get.""" + return str(uuid.uuid5(RECORDING_NAMESPACE, f"{lab}/{source_recording_id}")) + + +# Frames every vendor uses by these names in /tf_static and in poses. +WORLD_FRAME = "world" +HEAD_FRAME = "head" + +# ---- Thresholds stated in the plan ---- +MIN_VIDEO_FPS = 30.0 +MIN_IMU_HZ = 200.0 +MIN_HEAD_POSE_HZ = 30.0 +GRAVITY = 9.81 # m/s^2: mean |accel| of a raw IMU at rest +VIDEO_CODECS = ("h264", "h265") +GRIPPER_RANGE = (0.0, 1.0) # 0 = closed, 1 = open + +# ---- PROVISIONAL thresholds (not in the plan) ---- +# A measured rate may fall this fraction below its minimum (29.97 fps passes). +RATE_TOL = 0.05 +GRAVITY_TOL = 0.5 # m/s^2 allowed around GRAVITY in the stillest window +STILL_WINDOW_S = 1.0 # window used to find the IMU at rest +MAX_GAP_S = 0.5 # largest allowed gap between consecutive messages of one stream +SPAN_TOL_S = 1.0 # how far a stream may start late / end early vs. the episode +METADATA_TIME_TOL_S = 1.0 # start/end_time_unix vs. the data's actual span +UNIT_TOL = QUAT_UNIT_TOL # |quaternion| - 1, and R^T R - I for rotation matrices +# Frames fully decoded per camera; structural checks run on every message. +VIDEO_SAMPLE_FRAMES = 60 +MIN_UNIX_NS = 1_420_070_400 * 10**9 # 2015-01-01: anything earlier is not Unix time + + +@dataclass(frozen=True) +class Topic: + """One channel an episode may carry. + + `kind` picks the checks the validator runs on it. `schema` is the exact schema name + required, or None for any. `fields` must be non-empty on every message. `side` marks a + per-arm topic, dropped for single-arm embodiments. + """ + + name: str + kind: str + required: bool = True + schema: str | None = None + fields: tuple[str, ...] = () + min_hz: float | None = None + side: str | None = None + + +@dataclass(frozen=True) +class MovingLink: + """A parent -> child transform that changes over time, published on `topic` + instead of /tf_static (e.g. the head moving in the world).""" + + parent: str + child: str + topic: str + + +@dataclass(frozen=True) +class MetaKey: + """One episode-info key. MCAP metadata values are strings; `kind` is what the + string must parse as: "str", "int", "enum" (one of `values`), "shutter" + (SHUTTER), "hand_joints" (comma-separated HAND_JOINTS names, each once, + including wrist), "required_metadata" (one of REQUIRED_METADATA_FIELDS, + checked with the zarr rules; `objects` is comma-separated) or "waiver" + (comma-separated REQUIRED_METADATA_FIELDS names the source does not have).""" + + name: str + kind: str = "str" + values: tuple[str, ...] = () + human_only: bool = False + optional: bool = False + + +@dataclass(frozen=True) +class Profile: + name: str + step_clock: str # camera whose frames define the Lance steps rows + topics: tuple[Topic, ...] + moving_links: tuple[MovingLink, ...] + + +_VIDEO = dict( + kind="video", + schema="foxglove.CompressedVideo", + min_hz=MIN_VIDEO_FPS, + fields=("data", "format", "frame_id"), +) +_TF = Topic( + "/tf_static", "tf_static", schema="foxglove.FrameTransforms", fields=("transforms",) +) +# Optional in either profile. Without /task, the episode-info task_name / task_description / objects +# keys describe one task spanning the whole recording; with /task, the spans are +# authoritative and those keys only summarize the session (no task row is made from them). +_LABELS = ( + Topic("/task", "task", required=False, schema="egoverse.Task", fields=("name",)), + Topic( + "/subtask-annotation", + "annotation", + required=False, + schema="egoverse.Annotation", + fields=("data",), + ), +) +# Must match egomimic.rldb.lance.schema.ANNOTATION_TYPES (PR #717); empty means "language". +ANNOTATION_TYPES = ("language",) + + +def _per_side(make) -> tuple[Topic, ...]: + return tuple(make(side) for side in ("left", "right")) + + +HUMAN = Profile( + name="human", + step_clock="/main-camera", + topics=( + Topic("/main-camera", **_VIDEO), + Topic("/slam-left-camera", **_VIDEO, required=False), + Topic("/slam-right-camera", **_VIDEO, required=False), + *_per_side(lambda s: Topic(f"/{s}-wrist-camera", **_VIDEO, required=False)), + Topic( + "/imu", + "imu", + schema="egoverse.Imu", + min_hz=MIN_IMU_HZ, + fields=("frame_id", "linear_acceleration", "angular_velocity"), + ), + Topic("/imu-info", "imu_info", schema="egoverse.ImuInfo", fields=("frame_id",)), + Topic( + "/head-pose", + "pose", + schema="foxglove.PoseInFrame", + min_hz=MIN_HEAD_POSE_HZ, + fields=("frame_id", "pose"), + ), + Topic( + "/head-pose-quality", + "tracking_quality", + schema="egoverse.TrackingQuality", + min_hz=MIN_HEAD_POSE_HZ, + ), + *_per_side( + lambda s: Topic( + f"/{s}-hand-keypoints", + "hand_keypoints", + schema="egoverse.HandKeypoints", + fields=("frame_id",), + side=s, + ) + ), + Topic( + "/eye-gaze", + "eye_gaze", + required=False, + schema="egoverse.EyeGaze", + fields=("frame_id",), + ), + _TF, + *_LABELS, + ), + moving_links=(MovingLink(WORLD_FRAME, HEAD_FRAME, "/head-pose"),), +) + +ROBOT = Profile( + name="robot", + step_clock="/top-camera", + topics=( + Topic("/top-camera", **_VIDEO), + *_per_side(lambda s: Topic(f"/{s}-wrist-camera", **_VIDEO, side=s)), + *_per_side( + lambda s: Topic( + f"/{s}-arm-state", + "arm", + schema="egoverse.ArmState", + side=s, + fields=( + "frame_id", + "joint_names", + "position", + "velocity", + "torque", + "ee_pose", + ), + ) + ), + *_per_side( + lambda s: Topic( + f"/{s}-arm-action", + "arm", + schema="egoverse.ArmAction", + side=s, + fields=("frame_id", "joint_names", "position", "ee_pose"), + ) + ), + *_per_side( + lambda s: Topic( + f"/{s}-ee-state", "gripper", schema="egoverse.GripperState", side=s + ) + ), + *_per_side( + lambda s: Topic( + f"/{s}-ee-action", "gripper", schema="egoverse.GripperState", side=s + ) + ), + _TF, + *_LABELS, + ), + # The end effector moves relative to its arm base; the wrist camera hangs off it. + moving_links=tuple( + MovingLink(f"{s}_arm_base", f"{s}_ee", f"/{s}-arm-state") + for s in ("left", "right") + ), +) + +METADATA_KEYS = ( + # The zarr required metadata (CONTRIBUTING_DATA.md §6.3), same names and rules. + *(MetaKey(f, "required_metadata") for f in REQUIRED_METADATA_FIELDS), + MetaKey("task_description", optional=True), # full task text + # Comma-separated required fields the source truly does not have (MissingMetadata). + # Only if absolutely necessary; the episode then fails without --allow-missing. + MetaKey(MISSING_METADATA_KEY, "waiver", optional=True), + MetaKey("embodiment", "enum", EMBODIMENTS), + MetaKey("rig_name"), + MetaKey("device_serial"), + MetaKey("start_time_unix", "int"), # ms + MetaKey("end_time_unix", "int"), # ms + # The source's own ID for this recording; stable across re-exports and corrections. + MetaKey("source_recording_id"), + MetaKey("spec_version", "enum", SUPPORTED_SPEC_VERSIONS), + MetaKey("license"), # e.g. "CC-BY-4.0"; the Lance recordings.license column + # Where in the exposure frames are stamped. + MetaKey("camera_timestamp_ref", "enum", ("start", "mid", "end"), human_only=True), + MetaKey("shutter", "shutter", human_only=True), # "global" or "rolling:16ms" + # HandKeypoints.positions order, chosen by the vendor, e.g. ",".join(MANO_JOINTS) + MetaKey("hand_joints", "hand_joints", human_only=True), +) +# Every converter must give these (task_name, operator, lab); only scene and objects +# may be listed in missing_metadata. The shared waiver_errors rejects the rest. +UNWAIVABLE_FIELDS = tuple( + f for f in REQUIRED_METADATA_FIELDS if f not in WAIVABLE_FIELDS +) +# Topics published exactly once per episode. +ONCE_TOPICS = ("/tf_static", "/imu-info") +# Topic kinds whose messages form a time series; the rest are published once or rarely. +STREAM_KINDS = frozenset( + { + "video", + "imu", + "pose", + "tracking_quality", + "hand_keypoints", + "arm", + "gripper", + "eye_gaze", + } +) +# Topics allowed in annotation.mcap. +LABEL_TOPICS = frozenset({"/task", "/subtask-annotation"}) + + +def profile_for(embodiment: str) -> Profile: + return HUMAN if embodiment.startswith("human_") else ROBOT + + +def topics_for(embodiment: str) -> tuple[Topic, ...]: + """The profile's topics, minus per-side topics of an arm the embodiment lacks.""" + sides = arm_sides(embodiment) + return tuple( + t for t in profile_for(embodiment).topics if t.side is None or t.side in sides + ) diff --git a/egomimic/rldb/mcap/validate.py b/egomimic/rldb/mcap/validate.py new file mode 100644 index 0000000000..74cddb7cee --- /dev/null +++ b/egomimic/rldb/mcap/validate.py @@ -0,0 +1,1199 @@ +"""Check a raw MCAP episode against the raw-data spec (spec.py, SPEC.md). + + python -m egomimic.rldb.mcap.validate episode_/ [episode_/ ...] + +Prints PASS, or every problem found, per episode; exits 1 if any episode fails. The +checks fall in seven groups: + + 1. everything there required topics, types, non-empty fields, episode info + 2. video usable codec, starts on a keyframe, no B-frames, decodes, size matches + calibration, fps + 3. clocks Unix ns, in order, no big gaps, streams cover the episode, rates + 4. calibration every camera has intrinsics and a path to the world frame + 5. imu raw (gravity still in it) and fast enough, noise specs present + 6. poses no NaNs, unit rotations, gripper in [0, 1], lost hands flagged not zeroed + 7. tasks & labels /task and /subtask-annotation spans are ordered, inside the data, + each label inside one task (the rules Lance's annotation review + applies, egomimic/rldb/lance/annotations.py in PR #717) + +The file is read once, in file order, with chunk and data-section CRCs checked; topics +the spec does not define are counted but never decoded. Every video message is +checked structurally (non-empty, one frame_id and format per stream, holds an H.264 / +H.265 picture); only the first VIDEO_SAMPLE_FRAMES are fully decoded, each of which +must decode to exactly one frame, so that every frame has its own capture time (Lance +needs one steps row per frame). annotation.mcap, when present, is read too and may only +hold the label topics. + +MCAP reading and packet-by-packet video decoding follow PR #606's +egomimic/scripts/abc_process/abc_to_zarr.py. +""" + +import argparse +import math +import re +import sys +from array import array +from collections import Counter, defaultdict +from collections.abc import Callable +from dataclasses import dataclass, field +from pathlib import Path + +import numpy as np + +from egomimic.rldb.mcap import spec +from egomimic.rldb.zarr.required_metadata import ( + REQUIRED_METADATA_FIELDS, + check_stored_metadata, + is_placeholder, + waiver_errors, +) + +GROUPS = { + "present": "everything there", + "video": "video usable", + "clocks": "clocks", + "calibration": "calibration", + "imu": "imu", + "poses": "poses", + "labels": "tasks & labels", +} +# Annex B NAL unit types that start a decodable picture (IDR / IRAP). +KEYFRAME_NALS = {"h264": {5}, "h265": set(range(16, 22))} +# NAL unit types that carry picture data (coded slices, VCL). +PICTURE_NALS = {"h264": set(range(1, 6)), "h265": set(range(0, 22))} + + +@dataclass(frozen=True) +class Problem: + group: str + topic: str | None + message: str + + def __str__(self) -> str: + where = f"{self.topic}: " if self.topic else "" + return f"[{GROUPS[self.group]}] {where}{self.message}" + + +Add = Callable[[str | None, str], None] + + +def _adder(out: list[Problem], group: str) -> Add: + """add(topic, message) appends a Problem of ``group`` to ``out``.""" + + def add(topic: str | None, message: str) -> None: + out.append(Problem(group, topic, message)) + + return add + + +@dataclass +class _Stream: + """Everything one pass keeps about one topic.""" + + schema: str + encoding: str + times: array = field(default_factory=lambda: array("q")) + # Decoded messages of the rare topics, and the video sample; time series keep only + # `values` instead. + msgs: list = field(default_factory=list) + # Time series: _ROWS[kind] of every message, flattened, `width` per message. + values: array = field(default_factory=lambda: array("d")) + width: int = 0 + row_error: Exception | None = None # a message _ROWS could not read (wrong type) + # field -> messages where it is empty + empty: Counter = field(default_factory=Counter) + missing_fields: set = field(default_factory=set) # fields the message type lacks + undecodable: bool = False + channels: set = field(default_factory=set) + frame_ids: Counter = field(default_factory=Counter) # of messages that have one + # Video only, over every message (msgs holds just the sample). + formats: Counter = field(default_factory=Counter) + no_picture: int = 0 + + @property + def n(self) -> int: + return len(self.times) + + def span_s(self) -> float: + return (max(self.times) - min(self.times)) / 1e9 if self.times else 0.0 + + def rate_hz(self) -> float: + span = self.span_s() + return (self.n - 1) / span if span > 0 else 0.0 + + +@dataclass +class _Episode: + path: Path + metadata: dict[str, dict[str, str]] # record name -> its key/values + streams: dict[str, _Stream] + record_counts: Counter = field(default_factory=Counter) # record name -> count + + @property + def info(self) -> dict[str, str]: + return self.metadata.get(spec.METADATA_RECORD, {}) + + +# ---------------------------------------------------------------- reading + + +def frames_in_message(data: bytes, codec: str) -> int: + """Pictures (access units) in one video message; the spec wants exactly 1, so every + frame has its own capture time. Counted with the parser: the decoder silently drops + the extra frames of a packed message. Shared with the MCAP -> Lance converter.""" + import av + + parser = av.CodecContext.create("hevc" if codec == "h265" else "h264", "r") + with av.logging.Capture(): # lone P-frames log "non-existing PPS" + return len(parser.parse(data)) + len(parser.parse(None)) + + +def _nal_types(data: bytes, codec: str) -> set[int]: + types, i = set(), 0 + while (j := data.find(b"\x00\x00\x01", i)) >= 0 and j + 3 < len(data): + b = data[j + 3] + types.add(b & 0x1F if codec == "h264" else (b >> 1) & 0x3F) + i = j + 3 + return types + + +def _is_empty(msg, name: str) -> bool: + value = getattr(msg, name) + if isinstance(value, (str, bytes)) or hasattr(value, "__len__"): + return len(value) == 0 + try: + return not msg.HasField(name) # sub-message + except ValueError: + return False # proto3 scalar: zero is a legitimate value + + +def _note_video(s: _Stream, msg) -> None: + fmt = str(getattr(msg, "format", "")) + s.formats[fmt] += 1 + data = bytes(getattr(msg, "data", b"")) + if data and fmt in PICTURE_NALS and not _nal_types(data, fmt) & PICTURE_NALS[fmt]: + s.no_picture += 1 + + +def _imu_row(m) -> tuple: + a, g = m.linear_acceleration, m.angular_velocity + return (a.x, a.y, a.z, g.x, g.y, g.z) + + +def _hand_row(m) -> tuple: + pos = np.asarray(m.positions, dtype=np.float64) + finite = bool(np.isfinite(pos).all()) + wrist = m.HasField("wrist_pose") + conf_ok = 0 <= m.confidence <= 1 and all(0 <= c <= 1 for c in m.joint_confidence) + return ( + m.valid, + len(pos), + wrist, + wrist and _pose_ok(m.wrist_pose), + conf_ok, + len(m.joint_confidence), + finite, + finite and not pos.any(), + ) + + +def _arm_row(m) -> tuple: + names = len(m.joint_names) != len(m.position) or any( + hasattr(m, a) and len(getattr(m, a)) != len(m.position) + for a in ("velocity", "torque") + ) + return (names, _rigid_4x4(m)) + + +# Topic.kind -> what its checks need of one message, as a fixed-width row of floats. +_ROWS = { + "imu": (6, _imu_row), + "pose": (1, lambda m: (_pose_ok(m.pose),)), + "hand_keypoints": (8, _hand_row), + "eye_gaze": ( + 1, + lambda m: (m.valid and not all(map(math.isfinite, (m.yaw, m.pitch, m.depth))),), + ), + "arm": (2, _arm_row), + "gripper": (1, lambda m: (m.position,)), + "tracking_quality": (0, lambda m: ()), # only its fields are checked +} + + +def _rows(s: _Stream) -> np.ndarray: + """The stream's rows as an (n, width) array; raises if a message was unreadable.""" + if s.row_error is not None: + raise s.row_error + return np.array(s.values, dtype=np.float64).reshape(-1, s.width) + + +def _note(s: _Stream, topic: str, kind: str | None, fields: tuple, msg) -> None: + for name in fields: + if not hasattr(msg, name): + s.missing_fields.add(name) + elif _is_empty(msg, name): + s.empty[name] += 1 + if hasattr(msg, "frame_id"): # a type without it is reported in group 1 + s.frame_ids[str(msg.frame_id)] += 1 + if topic.endswith(spec.CAMERA_SUFFIX): + _note_video(s, msg) + if len(s.msgs) < spec.VIDEO_SAMPLE_FRAMES: + s.msgs.append(msg) # only the sample is kept and fully decoded + elif kind in _ROWS: + s.width, row = _ROWS[kind] + if s.row_error is None: + try: + s.values.extend(map(float, row(msg))) + except Exception as e: + s.row_error = e.with_traceback(None) + else: + s.msgs.append(msg) + + +def _read(path: Path) -> _Episode: + from mcap.records import Channel, Message, Metadata, Schema + from mcap.stream_reader import StreamReader + from mcap_protobuf.decoder import DecoderFactory + + topics = {t.name: t for p in (spec.HUMAN, spec.ROBOT) for t in p.topics} + camera_topics = tuple( + spec.CAMERA_SUFFIX + x + for x in ("", spec.CALIBRATION_SUFFIX, spec.CAMERA_EXTRA_SUFFIX) + ) + factory = DecoderFactory() + decoders: dict[int, object] = {} + schemas: dict[int, Schema] = {} + channels: dict[int, Channel] = {} + streams: dict[str, _Stream] = {} + metadata: dict[str, dict[str, str]] = {} + record_counts: Counter = Counter() + + # One linear pass: it checks the data-section CRC of unchunked files as well as + # chunk CRCs (SeekingReader skips both without chunk indexes), and finds metadata + # records whether or not the summary indexes them. + with open(path, "rb") as f: + for record in StreamReader(f, validate_crcs=True).records: + if isinstance(record, Schema): + schemas[record.id] = record + elif isinstance(record, Channel): + channels[record.id] = record + elif isinstance(record, Metadata): + metadata[record.name] = dict(record.metadata) + record_counts[record.name] += 1 + elif isinstance(record, Message): + channel = channels[record.channel_id] + topic = channel.topic + s = streams.get(topic) + if s is None: + schema = schemas.get(channel.schema_id) + s = streams[topic] = _Stream( + schema.name if schema else "", channel.message_encoding + ) + s.times.append(record.log_time) + s.channels.add(channel.id) + t = topics.get(topic) + if t is None and not topic.endswith(camera_topics): + continue # not in the spec: counted, never decoded + if channel.id not in decoders: + decoders[channel.id] = factory.decoder_for( + channel.message_encoding, schemas.get(channel.schema_id) + ) + decode = decoders[channel.id] + if decode is None: + s.undecodable = True + continue + msg = decode(record.data) + _note(s, topic, t and t.kind, t.fields if t else (), msg) + return _Episode(path, metadata, streams, record_counts) + + +def _merge_annotation_file(ep: _Episode, path: Path) -> list[Problem]: + """Add annotation.mcap's label topics to ``ep``; problems with the file.""" + out = [] + add = _adder(out, "present") + annotation = path.with_name(spec.ANNOTATION_FILE) + if path.name != spec.EPISODE_FILE or not annotation.exists(): + return out + try: + extra = _read(annotation) + except Exception as e: + add(None, f"cannot read {annotation.name}: {type(e).__name__}: {e}") + return out + if extra.metadata: + add( + None, + f"{annotation.name} carries metadata records {sorted(extra.metadata)}; " + f"episode info belongs in {path.name} only", + ) + for topic, s in sorted(extra.streams.items()): + if topic not in spec.LABEL_TOPICS: + add(topic, f"only {sorted(spec.LABEL_TOPICS)} belong in {annotation.name}") + elif topic in ep.streams: + add(topic, f"in both {path.name} and {annotation.name}") + else: + ep.streams[topic] = s + return out + + +# ---------------------------------------------------------------- 1. everything there + + +def _embodiment(ep: _Episode) -> str: + """Declared embodiment, or a guess from the topics when it is missing/invalid.""" + declared = ep.info.get("embodiment", "") + if declared in spec.EMBODIMENTS: + return declared + return "human_bimanual" if "/main-camera" in ep.streams else "eva_bimanual" + + +def _int(value: str) -> int | None: + """The integer an episode-info value spells, or None (int() alone takes "²", " 5").""" + return int(value) if re.fullmatch(r"-?[0-9]+", value) else None + + +def _hand_joints_problem(value: str) -> str: + names = value.split(",") + unknown = [n for n in names if n not in spec.HAND_JOINTS] + if unknown: + return f"unknown joints {unknown}; use names from {list(spec.HAND_JOINTS)}" + if len(set(names)) != len(names): + return "lists a joint twice" + if "wrist" not in names: + return "must include wrist" + return "" + + +def _hand_joint_count(ep: _Episode) -> int: + """Joints per hand the episode declares; MANO's 21 when the key is unusable.""" + value = ep.info.get("hand_joints", "") + if value and not _hand_joints_problem(value): + return len(value.split(",")) + return len(spec.MANO_JOINTS) + + +def _required_metadata_problems( + info: dict[str, str], allow_missing: bool +) -> tuple[list[str], list[str]]: + """(problems, waiver warnings) from the zarr required-metadata rules on the episode + info (CONTRIBUTING_DATA.md §6.3). Fields named in `missing_metadata` are waived + (MissingMetadata), except spec.UNWAIVABLE_FIELDS.""" + raw = info.get(spec.MISSING_METADATA_KEY, "") + waived = {f.strip() for f in raw.split(",") if f.strip()} + out = waiver_errors(info, waived) + meta = {} + for name in REQUIRED_METADATA_FIELDS: + value = info.get(name) + if name in waived and not value: + meta[name] = None + elif value is not None: + meta[name] = value + errors, warnings, _passed = check_stored_metadata(meta, allow_missing=allow_missing) + return out + errors, warnings + + +def _str_value_problem(key: spec.MetaKey, value: str) -> str | None: + return ( + f" = {value!r} is a placeholder, not a value" if is_placeholder(value) else None + ) + + +def _int_value_problem(key: spec.MetaKey, value: str) -> str | None: + return f" = {value!r} is not an integer" if _int(value) is None else None + + +def _enum_value_problem(key: spec.MetaKey, value: str) -> str | None: + if value in key.values: + return None + return f" = {value!r}, expected one of {list(key.values)}" + + +def _shutter_value_problem(key: spec.MetaKey, value: str) -> str | None: + if spec.SHUTTER.fullmatch(value): + return None + return f" = {value!r}, expected 'global' or 'rolling:ms'" + + +def _hand_joints_value_problem(key: spec.MetaKey, value: str) -> str | None: + bad = _hand_joints_problem(value) + return f": {bad}" if bad else None + + +# MetaKey.kind -> what follows "episode info 'key'" in its problem, or None if fine. +# required_metadata and waiver keys are checked by _required_metadata_problems. +_META_VALUE_CHECKS = { + "str": _str_value_problem, + "int": _int_value_problem, + "enum": _enum_value_problem, + "shutter": _shutter_value_problem, + "hand_joints": _hand_joints_value_problem, +} + + +def check_present( + ep: _Episode, embodiment: str, allow_missing: bool = False +) -> tuple[list[Problem], list[str]]: + """(problems, waiver warnings) for group 1.""" + out = [] + add = _adder(out, "present") + expected = spec.topics_for(embodiment) + for t in expected: + s = ep.streams.get(t.name) + if s is None: + if t.required: + add(t.name, "missing required topic") + continue + if t.schema and s.schema != t.schema: + add(t.name, f"type is {s.schema!r}, expected {t.schema!r}") + if s.encoding != "protobuf" or s.undecodable: + add( + t.name, f"cannot decode {s.encoding!r} messages; the spec uses protobuf" + ) + for name in sorted(s.missing_fields): + add(t.name, f"message type has no field {name!r}") + for name, count in sorted(s.empty.items()): + add(t.name, f"{name!r} empty on {count}/{s.n} messages") + for topic, s in sorted(ep.streams.items()): + if len(s.channels) > 1: + add(topic, f"published on {len(s.channels)} channels; use one per topic") + once = topic in spec.ONCE_TOPICS or topic.endswith(spec.CAMERA_EXTRA_SUFFIX) + if once and s.n > 1: + add(topic, f"published {s.n} times, expected once") + + # The embodiment must match the topics: no topics of the other profile or an unused arm. + allowed = {t.name for t in expected} + for p in (spec.HUMAN, spec.ROBOT): + for t in p.topics: + if t.name in ep.streams and t.name not in allowed: + add(t.name, f"topic does not fit embodiment {embodiment!r}") + + if spec.METADATA_RECORD not in ep.metadata: + found = ", ".join(sorted(ep.metadata)) or "none" + add(None, f"no {spec.METADATA_RECORD!r} metadata record (found: {found})") + return out, [] + if ep.record_counts[spec.METADATA_RECORD] > 1: + add( + None, + f"{ep.record_counts[spec.METADATA_RECORD]} {spec.METADATA_RECORD!r} " + "records; write exactly one", + ) + known = {k.name for k in spec.METADATA_KEYS} + for name in sorted(set(ep.info) - known): + hint = name.replace("-", "_") + hint = f"; did you mean {hint!r}?" if hint in known else "" + add(None, f"unknown episode info key {name!r}{hint}") + messages, warnings = _required_metadata_problems(ep.info, allow_missing) + for message in messages: + add(None, message) + human = spec.profile_for(embodiment) is spec.HUMAN + for key in spec.METADATA_KEYS: + check = _META_VALUE_CHECKS.get(key.kind) + if check is None or (key.human_only and not human): + continue + value = ep.info.get(key.name) + if value is None or value == "": + if not key.optional: + add(None, f"episode info {key.name!r} missing") + elif (bad := check(key, value)) is not None: + add(None, f"episode info {key.name!r}{bad}") + return out, warnings + + +# ---------------------------------------------------------------- 2. video usable + + +def _codec(fmt: str) -> str | None: + return fmt if fmt in spec.VIDEO_CODECS else None + + +def _calibration(ep: _Episode, camera: str): + s = ep.streams.get(camera + spec.CALIBRATION_SUFFIX) + if not s or not s.msgs or s.schema != "foxglove.CameraCalibration": + return None + return s.msgs[0] + + +def check_video(ep: _Episode) -> list[Problem]: + import av + + out = [] + add = _adder(out, "video") + for topic, s in sorted(ep.streams.items()): + if not topic.endswith(spec.CAMERA_SUFFIX) or not s.msgs: + continue + fmt = str(getattr(s.msgs[0], "format", "")) + codec = _codec(fmt) + if codec is None: + add(topic, f"format {fmt!r}, expected one of {list(spec.VIDEO_CODECS)}") + continue + if len(s.formats) > 1: + add( + topic, + f"format changes within the stream: {dict(sorted(s.formats.items()))}", + ) + if s.no_picture: + add( + topic, + f"{s.no_picture}/{s.n} messages hold no {codec} picture (coded slice)", + ) + keyframes = [ + i + for i, msg in enumerate(s.msgs) + if _nal_types(bytes(msg.data), codec) & KEYFRAME_NALS[codec] + ] + if not keyframes: + add( + topic, + f"no keyframe (Annex B IDR/IRAP) in the first {len(s.msgs)} messages", + ) + continue + if keyframes[0] != 0: + add(topic, "first message is not a keyframe (Annex B IDR/IRAP)") + + # Decode from the first keyframe, so a bad start is reported once, above. + sample = s.msgs[keyframes[0] :] + + packed = sum(frames_in_message(bytes(m.data), codec) != 1 for m in sample) + if packed: + add( + topic, + f"{packed}/{len(sample)} sampled messages do not hold exactly one " + "frame; each message must be one frame at its capture time", + ) + ctx = av.CodecContext.create("hevc" if codec == "h265" else "h264", "r") + # Fail on bitstream errors instead of concealing them. + ctx.options = {"err_detect": "explode"} + # pts = message index: a decoder that hands frames back out of that order is + # reordering B-frames (has_b_frames alone is set by streams that have none). + out_order, b_frames, size = [], False, None + try: + for i, msg in enumerate(sample): + packet = av.Packet(bytes(msg.data)) + packet.pts = i + for frame in ctx.decode(packet): + out_order.append(frame.pts) + b_frames |= frame.pict_type == av.video.frame.PictureType.B + size = size or (frame.width, frame.height) + for frame in ctx.decode(None): + out_order.append(frame.pts) + b_frames |= frame.pict_type == av.video.frame.PictureType.B + size = size or (frame.width, frame.height) + except av.error.FFmpegError as e: + add(topic, f"decode error in the first {len(sample)} messages: {e}") + continue + frames = len(out_order) + if frames == 0: + add(topic, f"no frames decoded from the first {len(sample)} messages") + continue + if b_frames or (not packed and out_order != sorted(out_order)): + add( + topic, + "uses B-frames; re-encode without them (x264: -bf 0), so each " + "message decodes to its own frame in timestamp order", + ) + if not packed and frames != len(sample): + add( + topic, + f"{frames} frames decoded from {len(sample)} sampled messages; " + "each message must decode to one frame", + ) + + calib = _calibration(ep, topic) + if calib is not None and size != (int(calib.width), int(calib.height)): + add( + topic, + f"frames are {size[0]}x{size[1]}, calibration says " + f"{calib.width}x{calib.height}", + ) + + fps = s.rate_hz() # meaningless for packed messages, reported above + if not packed and fps < spec.MIN_VIDEO_FPS * (1 - spec.RATE_TOL): + add(topic, f"{fps:.1f} fps, need >= {spec.MIN_VIDEO_FPS:g}") + return out + + +# ---------------------------------------------------------------- 3. clocks + + +def _time_series(ep: _Episode, embodiment: str) -> dict[str, _Stream]: + """Cameras and the spec's time-series topics: what defines the recording's span.""" + expected = {t.name: t for t in spec.topics_for(embodiment)} + return { + topic: s + for topic, s in ep.streams.items() + if s.times + and ( + topic.endswith(spec.CAMERA_SUFFIX) + or (topic in expected and expected[topic].kind in spec.STREAM_KINDS) + ) + } + + +def _span(streams: dict[str, _Stream]) -> tuple[int, int]: + return ( + min(min(s.times) for s in streams.values()), + max(max(s.times) for s in streams.values()), + ) + + +def check_clocks(ep: _Episode, embodiment: str) -> list[Problem]: + out = [] + add = _adder(out, "clocks") + expected = {t.name: t for t in spec.topics_for(embodiment)} + streams = _time_series(ep, embodiment) + for topic, s in sorted(ep.streams.items()): + if s.times and min(s.times) < spec.MIN_UNIX_NS: + add(topic, "timestamps are before 2015: not absolute Unix ns") + for topic, s in sorted(streams.items()): + t = np.asarray(s.times, dtype=np.int64) + back = int((np.diff(t) < 0).sum()) + if back: + add(topic, f"timestamps go backwards {back} times") + same = int((np.diff(t) == 0).sum()) + if same and topic.endswith(spec.CAMERA_SUFFIX): + # Every frame needs its own time: Lance steps timestamps strictly increase. + add(topic, f"{same} frames repeat a timestamp") + if len(t) > 1: + gap = float(np.diff(np.sort(t)).max()) / 1e9 + if gap > spec.MAX_GAP_S: + add(topic, f"{gap:.2f} s gap, max {spec.MAX_GAP_S:g} s") + # Video rates are checked from decoded frames; IMU rate in its own group. + t_spec = expected.get(topic) + if t_spec and t_spec.min_hz and t_spec.kind not in ("video", "imu"): + if s.rate_hz() < t_spec.min_hz * (1 - spec.RATE_TOL): + add(topic, f"{s.rate_hz():.1f} Hz, need >= {t_spec.min_hz:g}") + + if not streams: + return out + start, end = _span(streams) + for topic, s in sorted(streams.items()): + late, early = (min(s.times) - start) / 1e9, (end - max(s.times)) / 1e9 + if late > spec.SPAN_TOL_S or early > spec.SPAN_TOL_S: + add( + topic, + f"covers only part of the episode (starts {late:.1f} s late, " + f"ends {early:.1f} s early)", + ) + + for key, data_ns in (("start_time_unix", start), ("end_time_unix", end)): + value = ep.info.get(key, "") + if _int(value) is not None: + off = abs(_int(value) * 1e6 - data_ns) / 1e9 + if off > spec.METADATA_TIME_TOL_S: + add(None, f"episode info {key!r} is {off:.1f} s from the data") + return out + + +# ---------------------------------------------------------------- 4. calibration + + +def _quat_ok(q) -> bool: + v = [q.x, q.y, q.z, q.w] + return ( + all(math.isfinite(c) for c in v) + and abs(math.sqrt(sum(c * c for c in v)) - 1) <= spec.UNIT_TOL + ) + + +def _reach(edges: dict, start: str) -> set[str]: + reached, todo = {start}, [start] + while todo: + for nxt in edges[todo.pop()] - reached: + reached.add(nxt) + todo.append(nxt) + return reached + + +def check_calibration(ep: _Episode, embodiment: str) -> list[Problem]: + out = [] + cameras = sorted(t for t in ep.streams if t.endswith(spec.CAMERA_SUFFIX)) + for cam in cameras: + try: + out += _check_camera_calibration(ep, cam) + except Exception as e: # a wrong-typed -info / -info-extra + out.append( + Problem("calibration", cam, f"could not check: {type(e).__name__}: {e}") + ) + return out + _check_frames(ep, embodiment, cameras) + + +def _check_camera_calibration(ep: _Episode, cam: str) -> list[Problem]: + out = [] + add = _adder(out, "calibration") + info = ep.streams.get(cam + spec.CALIBRATION_SUFFIX) + calib = _calibration(ep, cam) + if calib is None: + name = cam + spec.CALIBRATION_SUFFIX + if not info: + add(cam, f"no {name} calibration") + elif info.schema != "foxglove.CameraCalibration": + add( + cam, + f"{name} is {info.schema or 'untyped'}, not foxglove.CameraCalibration", + ) + else: + add(cam, f"{name} has no decodable message") + return out + video = ep.streams.get(cam) + if video and video.frame_ids: + named = set(video.frame_ids) + if named != {str(calib.frame_id)}: + add( + cam, + f"video frames name {sorted(named)} but its calibration names " + f"{calib.frame_id!r}: they must be the camera's one frame", + ) + if info.schema != "foxglove.CameraCalibration" or info.n != 1: + add( + cam + spec.CALIBRATION_SUFFIX, + f"expected one foxglove.CameraCalibration, got {info.n} " + f"{info.schema or 'untyped'} message(s)", + ) + K = list(getattr(calib, "K", [])) + D = list(getattr(calib, "D", [])) + if not all(math.isfinite(v) for v in K + D): + add(cam, "K or D has NaN / Inf") + elif len(K) == 9 and not (0 <= K[2] <= calib.width and 0 <= K[5] <= calib.height): + add( + cam, + f"principal point ({K[2]:g}, {K[5]:g}) is outside the " + f"{calib.width}x{calib.height} image", + ) + if len(K) != 9 or K[0] <= 0 or K[4] <= 0 or not calib.width or not calib.height: + add(cam, "calibration has no valid K (fx, fy > 0) or size") + model, n_d = str(calib.distortion_model), len(calib.D) + if model == "" and n_d == 0: + pass # pinhole + elif model not in spec.DISTORTION_MODELS: + add( + cam, + f"distortion_model {model!r}, expected one of " + f"{list(spec.DISTORTION_MODELS)}", + ) + elif n_d != spec.DISTORTION_MODELS[model]: + add( + cam, + f"{model} needs {spec.DISTORTION_MODELS[model]} D coefficients, got {n_d}", + ) + + extra = ep.streams.get(cam + spec.CAMERA_EXTRA_SUFFIX) + if extra and extra.msgs: + m = extra.msgs[0] + if m.upright_rotation_cw_deg not in spec.UPRIGHT_ROTATIONS: + add( + cam, + f"upright_rotation_cw_deg {m.upright_rotation_cw_deg}, expected " + f"one of {list(spec.UPRIGHT_ROTATIONS)}", + ) + if not (math.isfinite(m.valid_radius) and m.valid_radius >= 0): + add(cam, f"valid_radius {m.valid_radius} is not >= 0") + if m.frame_id != str(getattr(calib, "frame_id", m.frame_id)): + add( + cam + spec.CAMERA_EXTRA_SUFFIX, + f"frame_id {m.frame_id!r} differs from the camera's -info", + ) + return out + + +def _check_frames(ep: _Episode, embodiment: str, cameras: list[str]) -> list[Problem]: + out = [] + add = _adder(out, "calibration") + # Frame graph: /tf_static edges plus the moving links whose topic is present. + edges = defaultdict(set) + static = defaultdict(set) # /tf_static alone: what is rigidly mounted to what + tf = ep.streams.get("/tf_static") + bad_rot = bad_tr = 0 + for msg in tf.msgs if tf else (): + for x in msg.transforms: + for g in (edges, static): + g[x.parent_frame_id].add(x.child_frame_id) + g[x.child_frame_id].add(x.parent_frame_id) + bad_rot += not _quat_ok(x.rotation) + t = x.translation + bad_tr += not all(math.isfinite(v) for v in (t.x, t.y, t.z)) + if bad_rot: + add("/tf_static", f"{bad_rot} transforms have a non-unit rotation") + if bad_tr: + add("/tf_static", f"{bad_tr} transforms have a NaN / Inf translation") + sides = spec.arm_sides(embodiment) + profile = spec.profile_for(embodiment) + for link in profile.moving_links: + if link.topic in ep.streams and ( + not link.parent.endswith("_arm_base") or link.parent.split("_")[0] in sides + ): + edges[link.parent].add(link.child) + edges[link.child].add(link.parent) + reached = _reach(edges, spec.WORLD_FRAME) + + # A head-worn rig's steps camera rides on the head: its frame must be rigidly + # mounted to the head (a /tf_static path), which is how the steps are placed. + if any(link.child == spec.HEAD_FRAME for link in profile.moving_links): + clock = _calibration(ep, profile.step_clock) + fid = str(clock.frame_id) if clock is not None else "" + # (An unplaced frame is already reported below as having no path to world.) + if fid in reached and fid not in _reach(static, spec.HEAD_FRAME): + add( + profile.step_clock, + f"frame {clock.frame_id!r} has no fixed path to " + f"{spec.HEAD_FRAME!r} via /tf_static: the steps camera must be " + "mounted on the head", + ) + + # Cameras and the IMU must name their frame; every frame any topic names (hands, + # wrist poses, gaze, ...) must be placed in the world. + must_name = set(cameras) | {"/imu"} + for topic, s in sorted(ep.streams.items()): + skip = topic == "/tf_static" or topic.endswith( + (spec.CALIBRATION_SUFFIX, spec.CAMERA_EXTRA_SUFFIX) + ) + if skip or not s.frame_ids: + continue # a camera's -info / -info-extra share its frame, checked above + named = set(s.frame_ids) + if "" in named and topic in must_name: + add(topic, "no frame_id, so it cannot be placed") + for frame in sorted(named - reached - {""}): + add( + topic, + f"frame {frame!r} has no path to {spec.WORLD_FRAME!r} via /tf_static", + ) + head = ep.streams.get("/head-pose") + if head and head.frame_ids and set(head.frame_ids) != {spec.WORLD_FRAME}: + add("/head-pose", f"poses must be in frame {spec.WORLD_FRAME!r} (T_world_head)") + return out + + +# ---------------------------------------------------------------- 5. imu + + +def check_imu(ep: _Episode) -> list[Problem]: + out = [] + add = _adder(out, "imu") + s = ep.streams.get("/imu") + rows = _rows(s) if s and s.width else () + if len(rows): + if s.rate_hz() < spec.MIN_IMU_HZ * (1 - spec.RATE_TOL): + add("/imu", f"{s.rate_hz():.0f} Hz, need >= {spec.MIN_IMU_HZ:g}") + acc, gyr = rows[:, :3], rows[:, 3:] + if not (np.isfinite(acc).all() and np.isfinite(gyr).all()): + add("/imu", "NaN/Inf samples") + else: + # Stillest window = lowest mean rotation rate; a raw IMU there reads ~9.81 m/s^2. + w = max(1, min(len(acc), round(s.rate_hz() * spec.STILL_WINDOW_S))) + gyro_mag = np.convolve( + np.linalg.norm(gyr, axis=1), np.ones(w) / w, mode="valid" + ) + i = int(np.argmin(gyro_mag)) + still = float(np.linalg.norm(acc[i : i + w], axis=1).mean()) + if abs(still - spec.GRAVITY) > spec.GRAVITY_TOL: + add( + "/imu", + f"mean |accel| at rest = {still:.2f} m/s^2, expected " + f"~{spec.GRAVITY} (gravity removed, or wrong units?)", + ) + info = ep.streams.get("/imu-info") + if info and info.msgs: + names = ( + "accel_noise_density", + "accel_random_walk", + "gyro_noise_density", + "gyro_random_walk", + ) + bad = [n for n in names if not getattr(info.msgs[0], n, 0) > 0] + if bad: + add("/imu-info", f"noise values must be > 0: {', '.join(bad)}") + return out + + +# ---------------------------------------------------------------- 6. poses + + +def _pose_ok(pose) -> bool: + p = pose.position + return _quat_ok(pose.orientation) and all(math.isfinite(v) for v in (p.x, p.y, p.z)) + + +def _check_pose(ep: _Episode, name: str, rows: np.ndarray, add: Add) -> None: + bad = int((rows[:, 0] == 0).sum()) + if bad: + add(name, f"{bad}/{len(rows)} poses have NaNs or a non-unit rotation") + + +def _check_hand_keypoints(ep: _Episode, name: str, rows: np.ndarray, add: Add) -> None: + joints = _hand_joint_count(ep) + valid, n_pos, wrist, wrist_ok, conf_ok, n_conf, finite, zeros = rows.T.astype(bool) + n_pos, n_conf = rows[:, 1], rows[:, 5] + n = len(rows) + shape = int((~np.isin(n_pos, (0, 3 * joints))).sum()) + if shape: + add( + name, + f"{shape}/{n} messages have neither {3 * joints} positions " + f"({joints} hand-joints x xyz) nor none", + ) + empty = int((valid & (n_pos == 0) & ~wrist).sum()) + if empty: + add(name, f"{empty} valid messages have neither positions nor wrist_pose") + conf = int((~conf_ok | ~np.isin(n_conf, (0, joints))).sum()) + if conf: + add( + name, + f"{conf}/{n} messages have a confidence outside [0, 1] or " + f"joint_confidence not {joints} values", + ) + bad_wrist = int((valid & wrist & ~wrist_ok).sum()) + if bad_wrist: + add(name, f"{bad_wrist} valid wrist poses have NaNs or a non-unit rotation") + full = valid & (n_pos == 3 * joints) + nan, zero = int((full & ~finite).sum()), int((full & zeros).sum()) + if nan: + add(name, f"{nan} valid messages have NaN/Inf positions") + if zero: + add( + name, + f"{zero} messages are all-zero with valid=true; lost tracking must be " + "valid=false", + ) + + +def _check_eye_gaze(ep: _Episode, name: str, rows: np.ndarray, add: Add) -> None: + bad = int(rows[:, 0].sum()) + if bad: + add(name, f"{bad} valid messages have NaN/Inf") + + +def _rigid_4x4(m) -> bool: + pose = np.asarray(m.ee_pose, dtype=np.float64) + if ( + len(pose) != 16 + or not np.isfinite(pose).all() + or not np.isfinite(np.asarray(m.position)).all() + ): + return False + T = pose.reshape(4, 4) + R = T[:3, :3] + return ( + np.abs(R.T @ R - np.eye(3)).max() <= spec.UNIT_TOL + and abs(np.linalg.det(R) - 1) <= spec.UNIT_TOL + and np.abs(T[3] - [0, 0, 0, 1]).max() <= spec.UNIT_TOL + ) + + +def _check_arm(ep: _Episode, name: str, rows: np.ndarray, add: Add) -> None: + names, bad = int(rows[:, 0].sum()), int((rows[:, 1] == 0).sum()) + if names: + add( + name, f"{names}/{len(rows)} messages' joint arrays do not match joint_names" + ) + if bad: + add(name, f"{bad}/{len(rows)} poses are not a finite rigid 4x4 transform") + + +def _check_gripper(ep: _Episode, name: str, rows: np.ndarray, add: Add) -> None: + lo, hi = spec.GRIPPER_RANGE + vals = rows[:, 0] + bad = int((~np.isfinite(vals) | (vals < lo) | (vals > hi)).sum()) + if bad: + add(name, f"{bad}/{len(rows)} gripper values outside [{lo:g}, {hi:g}]") + + +# Topic.kind -> its group-6 check. +_POSE_CHECKS = { + "pose": _check_pose, + "hand_keypoints": _check_hand_keypoints, + "eye_gaze": _check_eye_gaze, + "arm": _check_arm, + "gripper": _check_gripper, +} + + +def check_poses(ep: _Episode, embodiment: str) -> list[Problem]: + out = [] + add = _adder(out, "poses") + for t in spec.topics_for(embodiment): + s = ep.streams.get(t.name) + check = _POSE_CHECKS.get(t.kind) + rows = _rows(s) if check and s and s.width else () + if len(rows): + check(ep, t.name, rows, add) + return out + + +# ---------------------------------------------------------------- 7. tasks & labels + + +def _spans(s: _Stream | None) -> list[tuple[int, int]]: + """(start, end) per message; end 0 (unset) gives end == start: a point event for a + label (one frame wide in Lance), an error for a task.""" + if not s or not s.msgs: + return [] + return [ + (t, int(getattr(m, "end_time_unix_ns", 0)) or t) + for t, m in zip(s.times, s.msgs) + ] + + +def check_labels(ep: _Episode, embodiment: str) -> list[Problem]: + out = [] + add = _adder(out, "labels") + streams = _time_series(ep, embodiment) + if not streams: + return out + start, end = _span(streams) + tol = spec.SPAN_TOL_S * 1e9 + tasks = _spans(ep.streams.get("/task")) + labels = _spans(ep.streams.get("/subtask-annotation")) + # A point label becomes one steps-camera frame wide in Lance (as the converter writes + # it), and Lance's annotation review wants the whole span inside a task. + step_clock = spec.profile_for(embodiment).step_clock + cam = ep.streams.get(step_clock) + cam_t = np.sort(np.asarray(cam.times if cam else [], dtype=np.int64)) + frame = float(np.median(np.diff(cam_t))) if len(cam_t) > 1 else 0.0 + wide = [(b, e if e > b else b + frame) for b, e in labels] + for topic, spans, point_ok in ( + ("/task", tasks, False), + ("/subtask-annotation", labels, True), + ): + backwards = sum(e < b or (e == b and not point_ok) for b, e in spans) + outside = sum(b < start - tol or e > end + tol for b, e in spans) + if backwards: + add(topic, f"{backwards} spans do not end after they start") + if outside: + add(topic, f"{outside}/{len(spans)} spans fall outside the recorded data") + if tasks: + orphans = sum(not any(tb <= b and e <= te for tb, te in tasks) for b, e in wide) + if orphans: + add( + "/subtask-annotation", + f"{orphans}/{len(labels)} labels do not lie inside one /task span", + ) + elif len(cam_t) and labels: + # The task made from the episode info: [first frame, last frame + one frame). + first, last = int(cam_t[0]), int(cam_t[-1]) + frame + beyond = sum(b < first or e > last for b, e in wide) + if beyond: + add( + "/subtask-annotation", + f"{beyond}/{len(labels)} labels fall outside {step_clock}'s frames " + "(no /task)", + ) + s = ep.streams.get("/subtask-annotation") + types = ( + {m.annotation_type for m in s.msgs if hasattr(m, "annotation_type")} + if s + else set() + ) + bad = sorted(types - {"", *spec.ANNOTATION_TYPES}) + if bad: + add( + "/subtask-annotation", + f"annotation_type {bad}, expected one of {list(spec.ANNOTATION_TYPES)}", + ) + return out + + +# ---------------------------------------------------------------- entry points + + +@dataclass +class Result: + embodiment: str # declared, or guessed from the topics; "" if unreadable + recording_id: str # the Lance recording_id it will get; "" without lab / source id + problems: list[Problem] + # Required metadata waived via missing_metadata, accepted with allow_missing. + warnings: list[str] = field(default_factory=list) + + @property + def ok(self) -> bool: + return not self.problems + + +def validate_episode(episode: Path, *, allow_missing: bool = False) -> Result: + """Validate one episode directory (or its episode.mcap). + + Required metadata waived in `missing_metadata` is a problem unless + `allow_missing`, then a warning. + """ + episode = Path(episode) + path = episode / spec.EPISODE_FILE if episode.is_dir() else episode + if not path.exists(): + return Result( + "", "", [Problem("present", None, f"no {spec.EPISODE_FILE} in {episode}")] + ) + try: + ep = _read(path) + except Exception as e: # truncated or not an MCAP at all + message = f"cannot read {path.name}: {type(e).__name__}: {e}" + return Result("", "", [Problem("present", None, message)]) + problems = _merge_annotation_file(ep, path) + embodiment = _embodiment(ep) + warnings: list[str] = [] + + def present() -> list[Problem]: + found, warns = check_present(ep, embodiment, allow_missing) + warnings.extend(warns) + return found + + for group, check in ( + ("present", present), + ("video", lambda: check_video(ep)), + ("clocks", lambda: check_clocks(ep, embodiment)), + ("calibration", lambda: check_calibration(ep, embodiment)), + ("imu", lambda: check_imu(ep)), + ("poses", lambda: check_poses(ep, embodiment)), + ("labels", lambda: check_labels(ep, embodiment)), + ): + try: + problems += check() + except Exception as e: # a topic of the wrong type the group's checks assume + problems.append( + Problem( + group, + None, + f"could not check: {type(e).__name__}: {e} (see 'type is' problems)", + ) + ) + lab = ep.info.get("lab", "") + source_id = ep.info.get("source_recording_id", "") + rid = spec.recording_id(lab, source_id) if lab and source_id else "" + return Result(embodiment, rid, problems, warnings) + + +def main(argv: list[str] | None = None) -> int: + p = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + p.add_argument( + "episodes", + nargs="+", + type=Path, + help="episode_/ dirs or episode.mcap files", + ) + p.add_argument( + "--allow-missing", + action="store_true", + help="Accept required metadata listed in missing_metadata (warn instead of fail)", + ) + args = p.parse_args(argv) + import av + + av.logging.set_level(av.logging.PANIC) # lone P-frames log "non-existing PPS" + failed = 0 + for episode in args.episodes: + r = validate_episode(episode, allow_missing=args.allow_missing) + label = ( + f"{episode} (spec {spec.SPEC_VERSION}, embodiment {r.embodiment or '?'}, " + f"recording_id {r.recording_id or '?'})" + ) + if r.problems: + failed += 1 + print(f"FAIL {label}: {len(r.problems)} problem(s)") + for problem in r.problems: + print(f" {problem}") + else: + print(f"PASS {label}") + for warning in r.warnings: + print(f" warning: {warning}") + if len(args.episodes) > 1: + print(f"\n{len(args.episodes) - failed}/{len(args.episodes)} passed") + return 1 if failed else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/egomimic/rldb/zarr/episode_rules.py b/egomimic/rldb/zarr/episode_rules.py new file mode 100644 index 0000000000..55014d4784 --- /dev/null +++ b/egomimic/rldb/zarr/episode_rules.py @@ -0,0 +1,147 @@ +""" +Episode-level rules shared by ZarrWriter.write and the episode validator, so the +two apply the same attr rules (CONTRIBUTING_DATA.md §6). The validator also +checks arrays (required keys, poses), which the writer does not. +""" + +import time +from datetime import datetime, timezone +from typing import Any + +import numpy as np + +SUPPORTED_FPS = (30, 60) + +# Earliest plausible recording start; earlier values are usually seconds or ns +# passed where epoch milliseconds were expected (§4). +MIN_START_TIME_MS = 1_420_070_400_000 # 2015-01-01 UTC +_EPISODE_HASH_FORMAT = "%Y-%m-%d-%H-%M-%S-%f" + +# Max |‖q‖ - 1| for pose quaternions; the MCAP validator uses the same value. +QUAT_UNIT_TOL = 1e-3 + +# Stamped in zarr.attrs by ZarrWriter. Episodes without it predate the required +# metadata rules; bump it when a rule change should not apply to older episodes. +SCHEMA_VERSION = 1 + +# Must match egomimic.rldb.embodiment.embodiment.EMBODIMENT (lowercased); a unit +# test checks. Listed here so the validators don't import torch. +EMBODIMENTS = ( + "human_right_arm", + "human_left_arm", + "human_bimanual", + "eva_right_arm", + "eva_left_arm", + "eva_bimanual", +) + + +def embodiment_error(embodiment: Any) -> str | None: + """Why ``embodiment`` is not a valid embodiment name (§9), or None.""" + if isinstance(embodiment, str) and embodiment.lower() in EMBODIMENTS: + return None + return ( + f"embodiment must be one of {list(EMBODIMENTS)}, got {embodiment!r}. " + "See CONTRIBUTING_DATA.md §9." + ) + + +def arm_sides(embodiment: str) -> tuple[str, ...]: + """Sides the embodiment uses: ("left", "right") for *_bimanual.""" + embodiment = embodiment.lower() + if embodiment.endswith("_left_arm"): + return ("left",) + if embodiment.endswith("_right_arm"): + return ("right",) + return ("left", "right") + + +def fps_error(fps: Any) -> str | None: + """Why ``fps`` is not a supported frame rate, or None.""" + # type() rather than isinstance: bool and np.int64 would pass the `in` test + # and then fail (or change type) when written to zarr.json. + if type(fps) is int and fps in SUPPORTED_FPS: + return None + return f"fps must be an int in {SUPPORTED_FPS}, got {fps!r}. See CONTRIBUTING_DATA.md §6.3." + + +def start_time_error(start_time_ms: Any) -> str | None: + """Why ``start_time_ms`` is not a plausible UTC epoch-ms recording start, or None.""" + now_ms = int(time.time() * 1000) + if type(start_time_ms) is int and MIN_START_TIME_MS <= start_time_ms <= now_ms: + return None + return ( + f"recording start time {start_time_ms!r} is not UTC epoch milliseconds " + "between 2015-01-01 and now (seconds or nanoseconds passed by mistake?). " + "See CONTRIBUTING_DATA.md §4." + ) + + +def episode_hash_start_ms(name: str) -> int | None: + """Epoch ms encoded in an episode hash like 2026-01-12-03-47-29-664000, or None.""" + try: + dt = datetime.strptime(name, _EPISODE_HASH_FORMAT) + except ValueError: + return None + return int(dt.replace(tzinfo=timezone.utc).timestamp() * 1000) + + +def _matrix_shape(value: Any) -> tuple[int, ...] | None: + try: + return np.asarray(value, dtype=np.float64).shape + except (TypeError, ValueError): + return None + + +def camera_errors(intrinsics: Any, extrinsics: Any) -> list[str]: + """Problems with the intrinsics / extrinsics dicts (§6.4).""" + errors = [] + if not isinstance(intrinsics, dict) or not intrinsics: + errors.append( + "intrinsics must be a non-empty {camera_key: 3x4 K matrix} dict, " + 'e.g. {"front_1": K}. See CONTRIBUTING_DATA.md §6.4.' + ) + else: + if not any("front" in str(k).lower() for k in intrinsics): + errors.append( + f"intrinsics has no front-camera entry (keys: {list(intrinsics)}); " + 'key the egocentric camera "front_1". See CONTRIBUTING_DATA.md §6.4.' + ) + for cam, K in intrinsics.items(): + if (shape := _matrix_shape(K)) != (3, 4): + errors.append( + f"intrinsics[{cam!r}] must be a 3x4 K matrix (pad a bare 3x3 " + f"with np.hstack([K, np.zeros((3, 1))])), got {shape or K!r}. " + "See CONTRIBUTING_DATA.md §6.4." + ) + continue + K = np.asarray(K, dtype=np.float64) + if not np.isfinite(K).all(): + errors.append(f"intrinsics[{cam!r}] contains NaN or inf") + elif not (K[0, 0] > 0 and K[1, 1] > 0): + errors.append( + f"intrinsics[{cam!r}] must have positive focal lengths " + f"(fx={K[0, 0]}, fy={K[1, 1]}). See CONTRIBUTING_DATA.md §6.4." + ) + if extrinsics is None: + return errors + if not isinstance(extrinsics, dict) or not extrinsics: + errors.append( + "extrinsics must be None or a non-empty {key: 4x4} dict (robots key " + 'per-arm, e.g. {"left": T, "right": T}). See CONTRIBUTING_DATA.md §6.4.' + ) + return errors + for key, T in extrinsics.items(): + if (shape := _matrix_shape(T)) != (4, 4): + errors.append( + f"extrinsics[{key!r}] must be a 4x4 transform matrix, got " + f"{shape or T!r}. See CONTRIBUTING_DATA.md §6.4." + ) + elif not np.isfinite(np.asarray(T, dtype=np.float64)).all(): + errors.append(f"extrinsics[{key!r}] contains NaN or inf") + return errors + + +def matrices_to_jsonable(matrices: dict) -> dict: + """{key: matrix} as plain nested float lists for zarr.json.""" + return {k: np.asarray(v, dtype=np.float64).tolist() for k, v in matrices.items()} diff --git a/egomimic/rldb/zarr/required_metadata.py b/egomimic/rldb/zarr/required_metadata.py new file mode 100644 index 0000000000..8e99dd49a0 --- /dev/null +++ b/egomimic/rldb/zarr/required_metadata.py @@ -0,0 +1,278 @@ +""" +Metadata required in every contributed zarr episode. + +See CONTRIBUTING_DATA.md §6.3. ZarrWriter takes each REQUIRED_METADATA_FIELDS +entry as a keyword argument with no default and checks it in write(); the +episode validator (egomimic.rldb.zarr.validate) applies the same rules to a +finished store. task_name, operator and lab are always required; scene and +objects can be left out only with an explicit MissingMetadata(). +""" + +import argparse +import re +from collections.abc import Iterable +from typing import Any + +REQUIRED_METADATA_FIELDS = ("task_name", "operator", "lab", "scene", "objects") +WAIVABLE_FIELDS = ("scene", "objects") + +# Values that mean "I didn't fill this in". Rejected so a placeholder can't +# stand in for a real value (compared case-insensitively after strip). +_PLACEHOLDER_VALUES = frozenset( + { + "", + "unknown", + "none", + "null", + "n/a", + "na", + "tbd", + "todo", + "debug", + "test", + "placeholder", + "template", + "missing", + "nil", + "nan", + "-", + "?", + "...", + } +) + +_LAB_RE = re.compile(r"[a-z0-9][a-z0-9_-]*") + + +def is_placeholder(value: str) -> bool: + return value.strip().lower() in _PLACEHOLDER_VALUES + + +class MissingMetadata: + """Explicit opt-out for ``scene`` or ``objects`` when the source data has no value. + + Pass it in place of the value, e.g. ``scene=MissingMetadata()``. task_name, + operator and lab cannot be waived. ``None``, + ``""`` and placeholders like ``"unknown"`` are rejected, so a field can only + be left out on purpose. The field is stored as ``null`` in zarr.attrs. + + Only use this if absolutely necessary, i.e. the source data truly has no + way to recover the value. Episodes with missing metadata are much harder + to filter and train on, and fail validation without --allow-missing. + """ + + def __repr__(self) -> str: + return "MissingMetadata()" + + +def objects_list(value: str | list[str] | tuple[str, ...]) -> list[str]: + """``objects`` as a list: a str is a comma-separated list of names.""" + if isinstance(value, str): + return [o.strip() for o in value.split(",")] + return list(value) + + +def required_metadata_error(field: str, value: Any) -> str | None: + """Return why ``value`` is not acceptable for required metadata ``field``, or None. + + Shared by ZarrWriter.write and the episode validator so both + apply the same rules. ``value`` is a concrete value, not MissingMetadata. + """ + hint = "See CONTRIBUTING_DATA.md §6.3." + if field in WAIVABLE_FIELDS: + waive = f"If the source data has no {field}, pass {field}=MissingMetadata(). " + else: + waive = f"{field} is always required and cannot be waived. " + if field == "objects" and isinstance(value, (list, tuple, str)): + if isinstance(value, str) and value != value.strip(): + return f"objects={value!r} has leading/trailing whitespace. {hint}" + names = objects_list(value) + if not names or names == [""]: + return f"objects must name at least one object. {waive}{hint}" + for obj in names: + if not isinstance(obj, str) or is_placeholder(obj): + return f"objects entries must be real names, got {obj!r}. {hint}" + if obj != obj.strip(): + return f"objects entry {obj!r} has leading/trailing whitespace. {hint}" + if not obj.isprintable(): + return f"objects entry {obj!r} has non-printable characters. {hint}" + return None + if not isinstance(value, str): + extra = " or list[str]" if field == "objects" else "" + return ( + f"{field} must be a str{extra}, got {type(value).__name__} ({value!r}). " + f"{waive}{hint}" + ) + if is_placeholder(value): + return f"{field}={value!r} is a placeholder, not a value. {waive}{hint}" + if value != value.strip(): + return f"{field}={value!r} has leading/trailing whitespace. {hint}" + if not value.isprintable(): + return f"{field}={value!r} has non-printable characters. {hint}" + if field == "lab" and not _LAB_RE.fullmatch(value): + return ( + f"lab={value!r} must be a short, stable, lowercase identifier " + f"(letters, digits, '_' or '-'), e.g. 'mecka'. {hint}" + ) + return None + + +def source_recording_id_error(value: Any) -> str | None: + """Why ``value`` is not a usable source_recording_id (§6.6), or None. + + It names the raw recording under s3://rldb/raw/// + and, with lab, gives the Lance recording_id, so it is one path component. + """ + hint = "See CONTRIBUTING_DATA.md §6.6." + if not isinstance(value, str) or is_placeholder(value): + return ( + "source_recording_id must name the raw recording this episode was cut " + f"from, got {value!r}. {hint}" + ) + if value != value.strip() or not value.isprintable() or "/" in value: + return ( + f"source_recording_id={value!r} must be a single path component with no " + f"surrounding whitespace or non-printable characters. {hint}" + ) + return None + + +def check_stored_metadata( + meta: dict[str, Any], *, allow_missing: bool = False +) -> tuple[list[str], list[str], list[str]]: + """(errors, warnings, passed field names) for a finished episode's metadata. + + Shared by the zarr and MCAP validators. An absent key is always an error; a + ``None`` value is an explicit waiver (MissingMetadata), an error unless + ``allow_missing`` and the field is in WAIVABLE_FIELDS. Concrete values go + through ``required_metadata_error``. + """ + errors, warnings, passed = [], [], [] + for name in REQUIRED_METADATA_FIELDS: + if name not in meta: + errors.append( + f"Missing metadata field: {name}. See CONTRIBUTING_DATA.md §6.3." + ) + elif meta[name] is None and name not in WAIVABLE_FIELDS: + errors.append( + f"{name} is null: it is always required and cannot be waived. " + "See CONTRIBUTING_DATA.md §6.3." + ) + elif meta[name] is None: + msg = f"{name}: marked as not available (MissingMetadata)" + if allow_missing: + warnings.append(msg) + else: + errors.append( + msg + ". We heavily recommend providing it; pass " + "--allow-missing to accept the episode anyway." + ) + elif (err := required_metadata_error(name, meta[name])) is not None: + errors.append(err) + else: + passed.append(name) + return errors, warnings, passed + + +def add_required_metadata_args(parser: argparse.ArgumentParser) -> None: + """Add --operator/--lab/--scene/--objects/--missing-metadata to a converter CLI. + + The converter supplies its own --task-name. None of these have usable + defaults: an omitted field fails in ZarrWriter.write unless it is scene or + objects waived with ``--missing-metadata FIELD``. + """ + group = parser.add_argument_group( + "required episode metadata (CONTRIBUTING_DATA.md §6.3)" + ) + group.add_argument("--operator", type=str, help="ID of the collecting operator") + group.add_argument("--lab", type=str, help="Short lowercase source id, e.g. rl2") + group.add_argument("--scene", type=str, help="Scene / location label") + group.add_argument("--objects", type=str, help="Objects manipulated") + group.add_argument( + "--missing-metadata", + action="append", + default=[], + choices=WAIVABLE_FIELDS, + metavar="FIELD", + help="Explicitly mark FIELD (scene or objects) as not available in the " + "source data. " + "Repeatable. Only use this if absolutely necessary: provide the value " + "whenever the source data has it.", + ) + + +def _build_required_metadata( + values: dict[str, Any], missing: Iterable[str] = () +) -> dict[str, Any]: + """Merge plain values and the names of waived fields into ZarrWriter kwargs. + + Both inputs are plain data so they can travel through Ray / argparse. + A field named in ``missing`` becomes MissingMetadata(); giving a field both + a value and a waiver is an error. + """ + if errors := waiver_errors(values, missing): + raise ValueError(errors[0]) + missing = set(missing) + return { + field: MissingMetadata() if field in missing else values.get(field) + for field in REQUIRED_METADATA_FIELDS + } + + +def waiver_errors(values: dict[str, Any], missing: Iterable[str]) -> list[str]: + """Waived fields that are unknown or not waivable, and fields with both a + value and a waiver.""" + missing = set(missing) + errors = [] + unknown = missing - set(REQUIRED_METADATA_FIELDS) + if unknown: + errors.append( + f"missing_metadata names unknown fields {sorted(unknown)}; " + f"expected a subset of {list(WAIVABLE_FIELDS)}" + ) + if unwaivable := sorted( + (missing & set(REQUIRED_METADATA_FIELDS)) - set(WAIVABLE_FIELDS) + ): + errors.append( + f"{unwaivable} cannot be waived; only {list(WAIVABLE_FIELDS)} can be " + "marked missing. See CONTRIBUTING_DATA.md §6.3." + ) + for field in REQUIRED_METADATA_FIELDS: + value = values.get(field) + empty = value is None or (hasattr(value, "__len__") and len(value) == 0) + if field in missing and not empty: + errors.append( + f"{field} has both a value ({value!r}) and is marked missing; " + "pass one or the other" + ) + return errors + + +def check_writer_metadata(kwargs: dict[str, Any]) -> None: + """Raise ValueError on the first unacceptable value in ZarrWriter metadata kwargs. + + The same check ZarrWriter.write runs; call it before an expensive extraction + so a bad value fails fast. + """ + for field in REQUIRED_METADATA_FIELDS: + value = kwargs.get(field) + if isinstance(value, MissingMetadata): + if field in WAIVABLE_FIELDS: + continue + raise ValueError( + f"{field}=MissingMetadata(): {field} is always required and " + "cannot be waived. See CONTRIBUTING_DATA.md §6.3." + ) + if (err := required_metadata_error(field, value)) is not None: + raise ValueError(err) + + +def required_metadata_from_args(args: Any) -> dict[str, Any]: + """ZarrWriter required-metadata kwargs from an argparse/SimpleNamespace. + + ``args.missing_metadata`` is an iterable of waived field names (repeated + --missing-metadata on the CLI, or a list from run_conversion). + """ + missing = getattr(args, "missing_metadata", None) or () + values = {f: getattr(args, f, None) for f in REQUIRED_METADATA_FIELDS} + return _build_required_metadata(values, missing) diff --git a/egomimic/rldb/zarr/validate.py b/egomimic/rldb/zarr/validate.py new file mode 100644 index 0000000000..5b23ef7365 --- /dev/null +++ b/egomimic/rldb/zarr/validate.py @@ -0,0 +1,306 @@ +""" +Validate a contributed Zarr episode against CONTRIBUTING_DATA.md §6 / §11.1. + +Run on every episode before uploading (vendors) and on ingest (RL2): + + python -m egomimic.rldb.zarr.validate ep1.zarr ep2.zarr ... + +Exits nonzero if any episode has errors. scene or objects written as +MissingMetadata() (stored as null) are errors unless --allow-missing is passed. +Episodes without a schema_version predate the required metadata rules: their +metadata problems are reported as warnings. +""" + +import argparse +import json +import sys +from dataclasses import dataclass, field +from pathlib import Path + +import numpy as np +import simplejpeg +import zarr + +from egomimic.rldb.zarr.episode_rules import ( + QUAT_UNIT_TOL, + SCHEMA_VERSION, + arm_sides, + camera_errors, + embodiment_error, + episode_hash_start_ms, + fps_error, + start_time_error, +) +from egomimic.rldb.zarr.required_metadata import ( + check_stored_metadata, + source_recording_id_error, +) + +_POSE_KEYS = ( + "left.obs_ee_pose", + "right.obs_ee_pose", + "left.obs_wrist_pose", + "right.obs_wrist_pose", + "obs_head_pose", + "left.cmd_ee_pose", + "right.cmd_ee_pose", +) +_GRIPPER_KEYS = ( + "left.obs_gripper", + "right.obs_gripper", + "left.gripper", + "right.gripper", +) +_KEYPOINT_KEYS = ("left.obs_keypoints", "right.obs_keypoints") + + +@dataclass +class ValidationReport: + path: Path + errors: list[str] = field(default_factory=list) + warnings: list[str] = field(default_factory=list) + successes: list[str] = field(default_factory=list) + + @property + def ok(self) -> bool: + return not self.errors + + +def _unwrap_bytes(raw): + """Unwrap nested 0-d object/bytes ndarrays down to raw bytes.""" + while isinstance(raw, np.ndarray): + raw = raw.item() if raw.shape == () else raw.flat[0] + if isinstance(raw, np.bytes_): + raw = bytes(raw) + return raw + + +def validate_episode( + zarr_path: str | Path, *, allow_missing: bool = False +) -> ValidationReport: + """Check a finished .zarr episode. Empty ``report.errors`` = pass.""" + report = ValidationReport(path=Path(zarr_path)) + try: + store = zarr.open_group(str(zarr_path), mode="r", zarr_format=3) + except Exception as e: + report.errors.append(f"cannot open as a zarr v3 group: {e}") + return report + try: + _validate_store(store, report, allow_missing) + except Exception as e: # malformed attrs the checks below assume are well-formed + report.errors.append(f"could not finish validating: {type(e).__name__}: {e}") + return report + + +def _validate_store( + store: zarr.Group, report: ValidationReport, allow_missing: bool +) -> None: + meta = dict(store.attrs) + _check_attrs(meta, report, allow_missing) + _check_episode_hash(report) + if "features" not in meta: + report.errors.append("Missing metadata field: features") + T = meta.get("total_frames") + if type(T) is not int or T <= 0: + report.errors.append(f"total_frames must be a positive int, got {T!r}") + return + features = meta.get("features") or {} + _check_arrays(store, features, meta.get("embodiment"), T, report.errors) + _check_annotations(store, features, T, report.errors) + _check_images(store, features, report.errors) + + +def _check_attrs(meta: dict, report: ValidationReport, allow_missing: bool) -> None: + for name, err in ( + ("fps", fps_error(meta.get("fps"))), + ("embodiment", embodiment_error(meta.get("embodiment"))), + ): + if err is None: + report.successes.append(f"{name}={meta[name]!r}") + else: + report.errors.append(err) + report.errors += camera_errors(meta.get("intrinsics"), meta.get("extrinsics")) + errors, warnings, passed = check_stored_metadata(meta, allow_missing=allow_missing) + if (err := source_recording_id_error(meta.get("source_recording_id"))) is None: + passed.append("source_recording_id") + else: + errors.append(err) + version = meta.get("schema_version") + if version is None: + warnings = [ + f"legacy episode (no schema_version): {e}" for e in errors + warnings + ] + errors = [] + report.warnings.append( + "legacy episode written before schema_version " + f"{SCHEMA_VERSION}; required metadata checked as warnings only" + ) + elif type(version) is not int or not 1 <= version <= SCHEMA_VERSION: + report.errors.append( + f"schema_version must be an int in 1..{SCHEMA_VERSION}, got {version!r}" + ) + else: + report.successes.append(f"schema_version={version}") + report.errors += errors + report.warnings += warnings + report.successes += [f"{name}={meta[name]!r}" for name in passed] + + +def _check_episode_hash(report: ValidationReport) -> None: + name = report.path.name.removesuffix(".zarr") + start_ms = episode_hash_start_ms(name) + if start_ms is None: + report.warnings.append( + f"{report.path.name} is not named .zarr. " + "See CONTRIBUTING_DATA.md §4." + ) + elif (err := start_time_error(start_ms)) is not None: + report.errors.append(f"episode hash {name}: {err}") + else: + report.successes.append(f"episode hash {name}") + + +def _check_arrays( + store: zarr.Group, features: dict, embodiment, T: int, errors: list[str] +) -> None: + array_keys = list(store.array_keys()) + for key in array_keys: + if key not in features: + errors.append(f"{key}: array has no entry in features") + if features.get(key, {}).get("dtype") == "json": + continue + if not store[key].shape: + errors.append(f"{key}: array is 0-dimensional, expected a time axis") + continue + if (arr_len := store[key].shape[0]) < T: + errors.append(f"{key}: array length {arr_len} < total_frames {T}") + for key in features: + if key not in array_keys: + errors.append(f"features[{key!r}] has no matching array") + + embodiment = str(embodiment or "") + required = ["images.front_1"] + if embodiment.lower().startswith("human_"): + required.append("obs_head_pose") # training re-expresses poses relative to it + required += [f"{side}.obs_ee_pose" for side in arm_sides(embodiment)] + for key in required: + if key not in store: + errors.append(f"Missing required key: {key}") + + for key in _POSE_KEYS: + if key not in store: + continue + arr = store[key][:T] + if arr.ndim != 2 or arr.shape[-1] != 7: + errors.append(f"{key}: expected shape (T, 7), got {arr.shape}") + continue + if not np.isfinite(arr).all(): + errors.append(f"{key}: contains NaN or inf") + continue + norms = np.linalg.norm(arr[:, 3:7], axis=1) + bad = np.where(np.abs(norms - 1.0) > QUAT_UNIT_TOL)[0] + if len(bad): + errors.append( + f"{key}: {len(bad)} frames with non-unit quaternions " + f"(e.g. frame {bad[0]}, norm={norms[bad[0]]:.6f})" + ) + + for key in _GRIPPER_KEYS: + if key in store: + shape = store[key].shape + if len(shape) != 2 or shape[-1] != 1: + errors.append(f"{key}: expected shape (T, 1), got {shape}") + for key in _KEYPOINT_KEYS: + if key in store and store[key].shape[-1] != 63: + errors.append( + f"{key}: expected last dim 63 (21x3), got {store[key].shape[-1]}" + ) + for key in _GRIPPER_KEYS + _KEYPOINT_KEYS: + if key in store and not np.isfinite(store[key][:T]).all(): + errors.append(f"{key}: contains NaN or inf") + + +def _annotation_error(raw, T: int) -> str | None: + try: + rec = json.loads( + raw.decode("utf-8") if isinstance(raw, (bytes, bytearray)) else raw + ) + except (TypeError, ValueError) as e: + return str(e) + if not isinstance(rec, dict): + return f"record is {type(rec).__name__}, expected dict" + for name, expected in (("text", str), ("start_idx", int), ("end_idx", int)): + if not isinstance(rec.get(name), expected): + return f"field {name!r} missing or not {expected}" + if not (0 <= rec["start_idx"] <= rec["end_idx"] <= T): + return ( + f"index range invalid: start={rec['start_idx']}, " + f"end={rec['end_idx']}, T={T}" + ) + return None + + +def _check_annotations( + store: zarr.Group, features: dict, T: int, errors: list[str] +) -> None: + for key, feat in features.items(): + if feat.get("dtype") != "json" or key not in store: + continue + node = store[key] + found = [ + (i, err) + for i in range(node.shape[0]) + if (err := _annotation_error(_unwrap_bytes(node[i]), T)) is not None + ] + if found: + errors.append( + f"{key}: {len(found)}/{node.shape[0]} annotations malformed " + f"(e.g. index {found[0][0]}: {found[0][1]})" + ) + + +def _check_images(store: zarr.Group, features: dict, errors: list[str]) -> None: + """Spot-check that frame 0 of each JPEG key decodes to RGB.""" + for key, feat in features.items(): + if feat.get("dtype") != "jpeg" or key not in store: + continue + try: + frame = simplejpeg.decode_jpeg( + bytes(_unwrap_bytes(store[key][0])), colorspace="RGB" + ) + except Exception as e: + errors.append(f"{key}: failed to decode frame 0: {e}") + continue + if frame.ndim != 3 or frame.shape[2] != 3: + errors.append(f"{key}: decoded frame has shape {frame.shape}") + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__.strip().splitlines()[0]) + parser.add_argument("episodes", nargs="+", type=Path, help=".zarr episode dirs") + parser.add_argument( + "--allow-missing", + action="store_true", + help="Accept scene/objects written as MissingMetadata (warn instead of fail)", + ) + parser.add_argument("-v", "--verbose", action="store_true", help="Print passes") + args = parser.parse_args(argv) + + n_failed = 0 + for path in args.episodes: + report = validate_episode(path, allow_missing=args.allow_missing) + print(f"{'PASS' if report.ok else 'FAIL'} {path}") + if args.verbose: + for s in report.successes: + print(f" ok: {s}") + for w in report.warnings: + print(f" warning: {w}") + for e in report.errors: + print(f" error: {e}") + n_failed += not report.ok + print(f"{len(args.episodes) - n_failed}/{len(args.episodes)} episodes passed") + return 1 if n_failed else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/egomimic/rldb/zarr/zarr_writer.py b/egomimic/rldb/zarr/zarr_writer.py index 4b8d698d2f..0057078b87 100644 --- a/egomimic/rldb/zarr/zarr_writer.py +++ b/egomimic/rldb/zarr/zarr_writer.py @@ -5,7 +5,11 @@ compatible with the ZarrEpisode reader. """ +import glob import json +import logging +import os +import shutil from pathlib import Path from typing import Any, Literal @@ -14,283 +18,38 @@ import zarr from zarr.core.dtype import VariableLengthBytes - -def _intrinsics_to_jsonable( - intrinsics: np.ndarray | list | dict, -) -> list | dict: - """Coerce intrinsics into a JSON-serializable form for zarr.json attrs.""" - if isinstance(intrinsics, np.ndarray): - return intrinsics.tolist() - if isinstance(intrinsics, dict): - return { - k: (v.tolist() if isinstance(v, np.ndarray) else v) - for k, v in intrinsics.items() - } - return intrinsics - - -class _IncrementalHandle: - """Context manager handle for incremental frame-by-frame Zarr writing.""" - - def __init__( - self, - writer: "ZarrWriter", - total_frames: int | None, - metadata_override: dict[str, Any] | None, - ): - self._writer = writer - self._total_frames = total_frames - self._metadata_override = metadata_override - self._cursor = 0 - self._capacity = total_frames if total_frames is not None else 0 - self._store: zarr.Group | None = None - self._initialized = False - self._numeric_info: dict[str, dict] = {} - self._image_info: dict[str, dict] = {} - - @property - def frames_written(self) -> int: - return self._cursor - - @property - def _padded_frames(self) -> int: - w = self._writer - if self._total_frames is None: - return self._capacity - padded = self._total_frames - if self._total_frames % w.chunk_timesteps != 0: - padded = ( - (self._total_frames + w.chunk_timesteps - 1) - // w.chunk_timesteps - * w.chunk_timesteps - ) - return padded - - def __enter__(self) -> "_IncrementalHandle": - if self._total_frames is not None: - self._writer.total_frames = self._total_frames - self._writer.episode_path.parent.mkdir(parents=True, exist_ok=True) - self._store = zarr.open(str(self._writer.episode_path), mode="w", zarr_format=3) - return self - - def _init_arrays( - self, numeric: dict[str, np.ndarray], images: dict[str, np.ndarray] - ) -> None: - """Create pre-allocated zarr arrays from first frame's schema.""" - dynamic_length = self._total_frames is None - if dynamic_length: - self._capacity = max(1, self._writer.chunk_timesteps) - - padded = self._padded_frames - w = self._writer - - for key, arr in numeric.items(): - frame_shape = arr.shape - self._numeric_info[key] = {"shape": frame_shape, "dtype": arr.dtype} - - full_shape = (padded,) + frame_shape - frames_per_chunk = max(1, min(w.chunk_timesteps, padded)) - chunk_shape = (frames_per_chunk,) + frame_shape - - if not dynamic_length: - self._store.create_array( - key, - shape=full_shape, - chunks=chunk_shape, - shards=full_shape, - dtype=arr.dtype, - fill_value=0, - ) - else: - self._store.create_array( - key, - shape=full_shape, - chunks=chunk_shape, - dtype=arr.dtype, - fill_value=0, - ) - - dimension_names = [f"dim_{i}" for i in range(len(frame_shape))] - w._features[key] = { - "dtype": str(arr.dtype), - "shape": list(frame_shape), - "names": dimension_names, - } - - for key, img in images.items(): - if img.ndim != 3 or img.shape[-1] != 3: - raise ValueError( - f"Image '{key}' must have shape (H, W, 3), got {img.shape}" - ) - self._image_info[key] = {"shape": img.shape} - - shape = (padded,) - chunk_shape = (1,) - - if not dynamic_length: - self._store.create_array( - key, - shape=shape, - chunks=chunk_shape, - shards=shape, - dtype=VariableLengthBytes(), - ) - else: - self._store.create_array( - key, - shape=shape, - chunks=chunk_shape, - dtype=VariableLengthBytes(), - ) - - w._features[key] = { - "dtype": "jpeg", - "shape": list(img.shape), - "names": ["height", "width", "channel"], - } - - self._initialized = True - - def _ensure_capacity(self, target_size: int) -> None: - if self._total_frames is not None: - if target_size > self._total_frames: - raise ValueError( - f"Write at frame {target_size} would exceed total_frames={self._total_frames}" - ) - return - - if target_size <= self._capacity: - return - - grown = max( - target_size, - self._capacity * 2 if self._capacity > 0 else self._writer.chunk_timesteps, - 1, - ) - for key, info in self._numeric_info.items(): - self._store[key].resize((grown,) + info["shape"]) - for key in self._image_info: - self._store[key].resize((grown,)) - self._capacity = grown - - def add_frame( - self, - numeric: dict[str, np.ndarray] | None = None, - images: dict[str, np.ndarray] | None = None, - ) -> None: - """ - Write a single frame. - - Args: - numeric: Dict of per-frame numeric arrays, each with shape matching - the feature dimensions (e.g. shape (D,) for a D-dim vector). - images: Dict of per-frame images, each with shape (H, W, 3) uint8. - """ - numeric = numeric or {} - images = images or {} - - if not self._initialized: - self._init_arrays(numeric, images) - - self._ensure_capacity(self._cursor + 1) - - for key, arr in numeric.items(): - self._store[key][self._cursor] = arr - - for key, img in images.items(): - jpeg_bytes = simplejpeg.encode_jpeg( - img, quality=ZarrWriter.JPEG_QUALITY, colorspace="RGB" - ) - self._store[key][self._cursor] = jpeg_bytes - - self._cursor += 1 - - def add_frames( - self, - numeric: dict[str, np.ndarray] | None = None, - images: dict[str, np.ndarray] | None = None, - ) -> None: - """ - Write a batch of frames. - - Args: - numeric: Dict of numeric arrays with shape (B, ...) where B is batch size. - images: Dict of image arrays with shape (B, H, W, 3) uint8. - """ - numeric = numeric or {} - images = images or {} - - batch_size = None - for arr in (*numeric.values(), *images.values()): - if batch_size is None: - batch_size = len(arr) - elif len(arr) != batch_size: - raise ValueError("All arrays in a batch must have the same length") - - if batch_size is None: - return - - if not self._initialized: - first_numeric = {k: v[0] for k, v in numeric.items()} - first_images = {k: v[0] for k, v in images.items()} - self._init_arrays(first_numeric, first_images) - - end = self._cursor + batch_size - self._ensure_capacity(end) - - for key, arr in numeric.items(): - self._store[key][self._cursor : end] = arr - - for key, img_batch in images.items(): - encoded = np.empty((batch_size,), dtype=object) - for i in range(batch_size): - encoded[i] = simplejpeg.encode_jpeg( - img_batch[i], quality=ZarrWriter.JPEG_QUALITY, colorspace="RGB" - ) - self._store[key][self._cursor : end] = encoded - - self._cursor += batch_size - - def __exit__(self, exc_type, exc_val, exc_tb) -> bool: - if exc_type is not None: - return False - - if self._total_frames is not None and self._cursor != self._total_frames: - raise ValueError( - f"Expected {self._total_frames} frames but wrote {self._cursor}" - ) - - if self._cursor == 0: - raise ValueError("Expected at least one frame, but wrote 0") - - self._writer.total_frames = self._cursor - - if self._total_frames is None: - # Unknown-length mode grows arrays dynamically; trim slack at close. - for key, info in self._numeric_info.items(): - self._store[key].resize((self._cursor,) + info["shape"]) - for key in self._image_info: - self._store[key].resize((self._cursor,)) - else: - # Pad image arrays for sharding alignment (numeric arrays use fill_value=0) - padded = self._padded_frames - if padded > self._total_frames and self._image_info: - pad_len = padded - self._total_frames - for key in self._image_info: - last_jpeg = self._store[key][self._total_frames - 1] - padding = np.empty((pad_len,), dtype=object) - padding[:] = last_jpeg - self._store[key][self._total_frames : padded] = padding - - # Write language annotations - if self._writer.annotations is not None: - self._writer._write_annotations(self._store, self._writer.annotations) - - # Write metadata - metadata = self._writer._build_metadata(self._metadata_override) - self._store.attrs.update(metadata) - - return False +from egomimic.rldb.zarr.episode_rules import ( + SCHEMA_VERSION, + camera_errors, + embodiment_error, + fps_error, + matrices_to_jsonable, +) +from egomimic.rldb.zarr.required_metadata import ( + REQUIRED_METADATA_FIELDS, + MissingMetadata, + check_writer_metadata, + objects_list, + source_recording_id_error, +) + +logger = logging.getLogger(__name__) + +# attrs the writer derives itself; metadata_override may not name them. +_WRITER_OWNED_ATTRS = frozenset( + { + "embodiment", + "total_frames", + "fps", + "task_description", + "features", + "intrinsics", + "extrinsics", + "schema_version", + "source_recording_id", + *REQUIRED_METADATA_FIELDS, + } +) class ZarrWriter: @@ -306,50 +65,70 @@ class ZarrWriter: def __init__( self, episode_path: str | Path, - embodiment: str = "", + *, + embodiment: str, + intrinsics: dict, + task_name: "str | MissingMetadata", + operator: "str | MissingMetadata", + lab: "str | MissingMetadata", + scene: "str | MissingMetadata", + objects: "str | list[str] | MissingMetadata", + source_recording_id: str, fps: int = 30, - task_name: str = "debug", task_description: str = "", annotations: list[tuple[str, int, int]] | None = None, chunk_timesteps: int = 100, - intrinsics: dict | None = None, extrinsics: dict | None = None, - verbose: bool = False, ): """ Initialize ZarrWriter. + The camera and metadata arguments are keyword-only with no default, so + forgetting one is a TypeError at the call site; their values are + checked by write() before anything touches disk. To edit an existing + episode (annotations, removing a key) use the module-level + write_annotations / remove_key / key_exists instead. + Args: episode_path: Path to episode .zarr directory. - embodiment: Robot type identifier (e.g., "eva_bimanual"). + embodiment: Embodiment identifier (e.g. "human_bimanual"); must be + a valid EMBODIMENT name (CONTRIBUTING_DATA.md §9). fps: Frames per second for playback (default: 30). - task_name: Task name. task_description: Task description. annotations: List of (text, start_idx, end_idx) tuples describing language annotations. chunk_timesteps: Number of timesteps per chunk for numeric arrays (default: 100). - intrinsics: Camera intrinsics as a {camera_key: 3x4 K matrix} dict + intrinsics: REQUIRED: {camera_key: 3x4 K matrix} dict (single-camera = one entry, e.g. {"front_1": K}). Stored under the - "intrinsics" key in zarr.json metadata. Optional in this low-level - constructor; create_and_write requires it. + "intrinsics" key in zarr.json metadata. extrinsics: Optional camera extrinsics. None (egocentric human data) or - a dict mapping a key to its transform matrix; robots key per-arm + a non-empty dict of 4x4 transforms; robots key per-arm (e.g. {"left": world<-cam, "right": world<-cam}). Stored under the - "extrinsics" key in zarr.json metadata. create_and_write enforces - the None-or-non-empty-dict contract. + "extrinsics" key in zarr.json metadata. + task_name, operator, lab, scene, objects: REQUIRED episode + metadata; rules in CONTRIBUTING_DATA.md §6.3. objects may be a + list or a comma-separated str and is stored as a list. + task_name, operator and lab are always required. scene and + objects may be MissingMetadata() only if the source data has no + value; it is stored as null. + source_recording_id: REQUIRED: your ID for the raw recording this + episode was cut from (CONTRIBUTING_DATA.md §6.6). """ self.episode_path = Path(episode_path) - - # Store parameters + self.source_recording_id = source_recording_id self.fps = fps self.embodiment = embodiment - self.task_name = task_name self.task_description = task_description self.annotations = annotations if annotations is not None else [] self.chunk_timesteps = chunk_timesteps self.intrinsics = intrinsics self.extrinsics = extrinsics - self.verbose = verbose - # Track image shapes for metadata + self.required_metadata = { + "task_name": task_name, + "operator": operator, + "lab": lab, + "scene": scene, + "objects": objects, + } self._features: dict[str, dict[str, Any]] = {} def write( @@ -358,9 +137,9 @@ def write( image_data: dict[str, np.ndarray] | None = None, pre_encoded_image_data: dict[str, tuple[np.ndarray, list[int]]] | None = None, metadata_override: dict[str, Any] | None = None, - ) -> None: + ) -> Path: """ - Write episode data to Zarr store. + Validate the episode configuration, then write it to the Zarr store. Args: numeric_data: Dictionary of numeric arrays (state, actions, etc.). @@ -370,21 +149,33 @@ def write( pre_encoded_image_data: Dictionary mapping key to (encoded_array, image_shape). encoded_array is np.ndarray(dtype=object) of JPEG bytes. image_shape is [H, W, 3]. Skips internal JPEG encoding. - metadata_override: Optional metadata overrides to apply after building metadata. + metadata_override: Extra keys to add to zarr.attrs. Must not name + a key the writer sets itself (embodiment, fps, intrinsics, the + required metadata fields, ...); pass those as ZarrWriter + arguments instead. + + Returns: + Path to the written episode. Raises: - ValueError: If arrays have inconsistent frame counts. - ValueError: If total_frames was not set and cannot be inferred. + ValueError: If the data, cameras, embodiment, fps, required + metadata (§6.3) or metadata_override are invalid. Nothing is + written, and an existing episode at episode_path is left as it + was, when write() raises. """ + _remove_stale_siblings(self.episode_path) + self._validate() numeric_data = numeric_data or {} image_data = image_data or {} pre_encoded_image_data = pre_encoded_image_data or {} + self._features = {} if not numeric_data and not image_data and not pre_encoded_image_data: raise ValueError( "Must provide at least one of numeric_data, image_data, " "or pre_encoded_image_data" ) + self._validate_inputs(image_data, pre_encoded_image_data, metadata_override) # Infer total_frames from data all_lengths: list[int] = [] @@ -398,74 +189,97 @@ def write( if len(set(all_lengths)) > 1: raise ValueError(f"Inconsistent frame counts across arrays: {all_lengths}") - self.total_frames = all_lengths[0] + total_frames = all_lengths[0] + if total_frames == 0: + raise ValueError("Episode has zero frames") - # Calculate padded frame count for sharding alignment - padded_frames = self.total_frames - if self.total_frames % self.chunk_timesteps != 0: - padded_frames = ( - (self.total_frames + self.chunk_timesteps - 1) // self.chunk_timesteps - ) * self.chunk_timesteps + # Padded to a whole number of chunks for sharding alignment. + padded_frames = -(-total_frames // self.chunk_timesteps) * self.chunk_timesteps - # Create parent directory self.episode_path.parent.mkdir(parents=True, exist_ok=True) - # Open Zarr v3 store - mode = "w" if self.episode_path.exists() else "w" - store = zarr.open(str(self.episode_path), mode=mode, zarr_format=3) - - # Write numeric arrays - for key, arr in numeric_data.items(): - self._write_numeric_array(store, key, arr, padded_frames) + # Build in a sibling temp dir and swap it in at the end, so a failure + # never leaves a half-written episode or wipes an existing one. + tmp_path = _sibling(self.episode_path, "tmp") + try: + store = zarr.open(str(tmp_path), mode="w", zarr_format=3) - # Write image arrays (with internal JPEG encoding) - for key, arr in image_data.items(): - self._write_image_array(store, key, arr, padded_frames) - - # Write pre-encoded image arrays (skip JPEG encoding) - for key, (enc_arr, img_shape) in pre_encoded_image_data.items(): - self._write_pre_encoded_image_array( - store, key, enc_arr, img_shape, padded_frames - ) + for key, arr in numeric_data.items(): + self._write_numeric_array(store, key, arr, padded_frames) - # Write language annotations if provided - if self.annotations is not None: - self._write_annotations(store, self.annotations) + for key, arr in image_data.items(): + self._write_image_array(store, key, arr, padded_frames) - # Build and attach metadata - metadata = self._build_metadata(metadata_override) - store.attrs.update(metadata) - - def write_incremental( - self, - total_frames: int | None = None, - metadata_override: dict[str, Any] | None = None, - ) -> _IncrementalHandle: - """ - Begin incremental writing to avoid loading all data into memory. - - Use as a context manager. Array schemas are inferred automatically - from the first add_frame() or add_frames() call. - - Args: - total_frames: Optional total number of frames to write. If omitted, - arrays grow dynamically and are trimmed to final size at close. - metadata_override: Optional metadata overrides. + for key, (enc_arr, img_shape) in pre_encoded_image_data.items(): + self._write_pre_encoded_image_array( + store, key, enc_arr, img_shape, padded_frames + ) - Returns: - Context manager with add_frame() and add_frames() methods. + self._features["annotations"] = _write_annotation_array( + store, self.annotations, "annotations" + ) - Example:: + store.attrs.update(self._build_metadata(total_frames, metadata_override)) + _swap_in(tmp_path, self.episode_path) + except BaseException: + shutil.rmtree(tmp_path, ignore_errors=True) + raise + return self.episode_path - writer = ZarrWriter(episode_path="ep.zarr", embodiment="eva_bimanual") - with writer.write_incremental() as inc: - for i in range(1000): - inc.add_frame( - numeric={"actions": actions[i]}, - images={"cam_left": images[i]}, - ) - """ - return _IncrementalHandle(self, total_frames, metadata_override) + @staticmethod + def _validate_inputs( + image_data: dict[str, np.ndarray], + pre_encoded_image_data: dict[str, tuple[np.ndarray, list[int]]], + metadata_override: dict[str, Any] | None, + ) -> None: + """Checks that need the data, run before anything touches disk.""" + for key, arr in image_data.items(): + if arr.ndim != 4 or arr.shape[-1] != 3 or arr.dtype != np.uint8: + raise ValueError( + f"Image array '{key}' must be uint8 with shape (T, H, W, 3), " + f"got {arr.dtype} {arr.shape}" + ) + for key, (_enc, img_shape) in pre_encoded_image_data.items(): + if len(img_shape) != 3 or img_shape[-1] != 3: + raise ValueError( + f"Pre-encoded image '{key}' shape must be [H, W, 3], " + f"got {img_shape}" + ) + if metadata_override: + owned = sorted(_WRITER_OWNED_ATTRS & metadata_override.keys()) + if owned: + raise ValueError( + f"metadata_override must not set {owned}: the writer sets " + "these itself. Pass them as ZarrWriter arguments (total_frames " + "and features come from the data)." + ) + try: + json.dumps(metadata_override) + except TypeError as e: + raise ValueError( + f"metadata_override is not JSON-serializable: {e}" + ) from e + + def _validate(self) -> None: + """Check embodiment, fps, cameras, required metadata and the target path.""" + errors = [ + e + for e in (embodiment_error(self.embodiment), fps_error(self.fps)) + if e is not None + ] + errors += camera_errors(self.intrinsics, self.extrinsics) + if (err := source_recording_id_error(self.source_recording_id)) is not None: + errors.append(err) + if errors: + raise ValueError("; ".join(errors)) + if self.episode_path.is_symlink() or self.episode_path.is_file(): + raise ValueError( + f"{self.episode_path} exists and is not an episode directory" + ) + check_writer_metadata(self.required_metadata) + for field, value in self.required_metadata.items(): + if isinstance(value, MissingMetadata): + logger.warning("Writing %s without %s", self.episode_path, field) def _write_numeric_array( self, store: zarr.Group, key: str, arr: np.ndarray, padded_frames: int @@ -528,12 +342,6 @@ def _write_image_array( image_arr: Image array with shape (T, H, W, 3). padded_frames: Target frame count after padding (for sharding alignment). """ - # Validate shape - if image_arr.ndim != 4 or image_arr.shape[-1] != 3: - raise ValueError( - f"Image array '{key}' must have shape (T, H, W, 3), got {image_arr.shape}" - ) - # Encode each frame as JPEG num_frames = len(image_arr) @@ -615,93 +423,8 @@ def _write_pre_encoded_image_array( "names": ["height", "width", "channel"], } - def check_key_exists(self, episode_path: str, key: str) -> bool: - return key in zarr.open(str(episode_path), mode="r", zarr_format=3) - - def remove_key(self, episode_path: str, key: str): - store = zarr.open(str(episode_path), mode="a", zarr_format=3) - del store[key] - - def append_annotations( - self, - annotation_key: str, - annotations: list[tuple[str, int, int]], - mode: Literal["w", "a"] = "a", - ) -> None: - """ - Write language annotations to the Zarr store. - - Args: - annotation_key: Key to write annotations to. - annotations: List of (text, start_idx, end_idx) tuples. - """ - annotation_exists = self.check_key_exists(self.episode_path, annotation_key) - if annotation_exists: - if mode == "w": - if self.verbose: - episode_name = self.episode_path.stem - print(f"({episode_name}) Removing key: {annotation_key}") - self.remove_key(self.episode_path, annotation_key) - store = zarr.open(str(self.episode_path), mode="a", zarr_format=3) - self._write_annotations(store, annotations, annotation_key) - - def _write_annotations( - self, - store: zarr.Group, - annotations: list[tuple[str, int, int]], - annotation_key: str = "annotations", - ) -> None: - """ - Write language annotations as JSON-encoded bytes. - - Args: - store: Zarr group to write to. - annotations: List of (text, start_idx, end_idx) tuples. - """ - encoded = np.array( - [ - json.dumps( - { - "text": text, - "start_idx": int(start_idx), - "end_idx": int(end_idx), - }, - ensure_ascii=False, - separators=(",", ":"), - ).encode("utf-8") - for text, start_idx, end_idx in annotations - ], - dtype=object, - ) - - n_annotations = len(annotations) - # Both the chunk and the SHARD edge must be >= 1 even when there are no - # annotations: zarr v3 rejects a zero-length chunk grid, so an episode - # written with shards=(0,) cannot be enumerated (group.keys() raises - # "integer chunk edge length must be >= 1") even though the array itself - # reads back fine. shape=(0,) with a >=1 edge is the correct encoding of - # "empty" -- no chunk objects are written either way. - chunk_shape = (max(1, n_annotations),) - store.create_array( - annotation_key, - shape=encoded.shape, - chunks=chunk_shape, - shards=(max(1, n_annotations),), - dtype=VariableLengthBytes(), - ) - if n_annotations > 0: - store[annotation_key][:] = encoded - - # Track in features - self._features[annotation_key] = { - "dtype": "json", - "shape": [n_annotations], - "names": ["json"], - "format": "annotation_v1", - } - def _build_metadata( - self, metadata_override: dict[str, Any] | None = None + self, total_frames: int, metadata_override: dict[str, Any] | None = None ) -> dict[str, Any]: """ Build episode metadata dictionary. @@ -713,32 +436,30 @@ def _build_metadata( Metadata dictionary. """ metadata = { + "schema_version": SCHEMA_VERSION, "embodiment": self.embodiment, - "total_frames": self.total_frames, + "total_frames": total_frames, "fps": self.fps, - "task_name": self.task_name, "task_description": self.task_description, + "source_recording_id": self.source_recording_id, "features": self._features, } - if self.intrinsics is not None: - metadata["intrinsics"] = _intrinsics_to_jsonable(self.intrinsics) - + metadata["intrinsics"] = matrices_to_jsonable(self.intrinsics) if self.extrinsics is not None: - metadata["extrinsics"] = _intrinsics_to_jsonable(self.extrinsics) + metadata["extrinsics"] = matrices_to_jsonable(self.extrinsics) - # Apply overrides — but NEVER let them clobber the validated camera - # metadata. intrinsics/extrinsics are validated in create_and_write; a - # converter's metadata_override that happens to carry a stale or empty - # "intrinsics"/"extrinsics" key must not silently overwrite the validated - # values (this was the Mecka clobber bug → empty intrinsics in zarr.json). + for field in REQUIRED_METADATA_FIELDS: + value = self.required_metadata[field] + if isinstance(value, MissingMetadata): + value = None + elif field == "objects": + value = objects_list(value) + metadata[field] = value + + # write() has already rejected overrides that name a writer-owned key. if metadata_override: - override = { - k: v - for k, v in metadata_override.items() - if k not in ("intrinsics", "extrinsics") - } - metadata.update(override) + metadata.update(metadata_override) return metadata @@ -748,116 +469,197 @@ def create_and_write( numeric_data: dict[str, np.ndarray] | None = None, image_data: dict[str, np.ndarray] | None = None, pre_encoded_image_data: dict[str, tuple[np.ndarray, list[int]]] | None = None, - embodiment: str = "", + *, + embodiment: str, + intrinsics: dict, + task_name: "str | MissingMetadata", + operator: "str | MissingMetadata", + lab: "str | MissingMetadata", + scene: "str | MissingMetadata", + objects: "str | list[str] | MissingMetadata", + source_recording_id: str, fps: int = 30, - task_name: str = "debug", task_description: str = "", annotations: list[tuple[str, int, int]] | None = None, chunk_timesteps: int = 100, - *, - intrinsics: dict, extrinsics: dict | None = None, metadata_override: dict[str, Any] | None = None, ) -> Path: - """ - Convenience method: create writer and write in one call. + """Build a ZarrWriter and write the episode in one call. - Args: - episode_path: Path to episode .zarr directory. - numeric_data: Dictionary of numeric arrays (state, actions, etc.). - image_data: Dictionary of image arrays with shape (T, H, W, 3). - pre_encoded_image_data: Dict mapping key to (encoded_array, image_shape). - Skips internal JPEG encoding for these keys. - embodiment: Robot type identifier. - fps: Frames per second (default: 30). - task_name: Task name. - task_description: Task description. - annotations: List of (text, start_idx, end_idx) tuples describing language annotations. - chunk_timesteps: Number of timesteps per chunk for numeric arrays (default: 100). - intrinsics: REQUIRED {camera_key: 3x4 K matrix} dict (single-camera = - one entry, e.g. {"front_1": K}). Stored in zarr.json metadata. - extrinsics: Optional camera extrinsics. Either None (e.g. egocentric - human data) or a non-empty dict mapping a key to its transform - matrix (robots key per-arm, e.g. {"left": T, "right": T}). Stored - in zarr.json metadata. - metadata_override: Optional metadata overrides. - - Returns: - Path to created episode. - - Raises: - ValueError: If no data is provided, if embodiment is not a valid - identifier (see §9), if intrinsics is not a non-empty - {camera_key: 3x4 K matrix} dict, or if extrinsics is neither None - nor a non-empty {key: 4x4 transform} dict. + Arguments are as in ZarrWriter.__init__ and ZarrWriter.write; all + validation happens in write(). Returns the episode path. """ - # Validate the embodiment up front (it is guaranteed to be consumed by - # the reader via get_embodiment_id) so a bad/empty/typo'd value fails - # here with an actionable message instead of an opaque KeyError at load. - from egomimic.rldb.embodiment.embodiment import EMBODIMENT - - if not embodiment or embodiment.upper() not in EMBODIMENT.__members__: - raise ValueError( - "embodiment must be one of " - f"{[m.name.lower() for m in EMBODIMENT]}, got {embodiment!r}. " - "See CONTRIBUTING_DATA.md §9." - ) - if not isinstance(intrinsics, dict) or not intrinsics: - raise ValueError( - "Camera intrinsics must be a non-empty {camera_key: 3x4 K matrix} " - 'dict for a contributed zarr episode, e.g. {"front_1": K}. ' - "See CONTRIBUTING_DATA.md §6.4." - ) - for cam_key, K in intrinsics.items(): - try: - K_arr = np.asarray(K, dtype=np.float64) - except (TypeError, ValueError): - K_arr = None - if K_arr is None or K_arr.shape != (3, 4): - raise ValueError( - f"intrinsics[{cam_key!r}] must be a 3x4 K matrix (the 3x3 " - "pinhole K with a zero last column; pad a bare 3x3 with " - "np.hstack([K, np.zeros((3, 1))])), got " - f"{getattr(K_arr, 'shape', type(K).__name__)}. " - "See CONTRIBUTING_DATA.md §6.4." - ) - if extrinsics is not None and ( - not isinstance(extrinsics, dict) or not extrinsics - ): - raise ValueError( - "Camera extrinsics must be None or a non-empty dict mapping a key " - 'to its transform matrix (robots key per-arm, e.g. {"left": T, ' - '"right": T}). See CONTRIBUTING_DATA.md §6.3.' - ) - if extrinsics is not None: - for ext_key, T in extrinsics.items(): - try: - T_arr = np.asarray(T, dtype=np.float64) - except (TypeError, ValueError): - T_arr = None - if T_arr is None or T_arr.shape != (4, 4): - raise ValueError( - f"extrinsics[{ext_key!r}] must be a 4x4 transform matrix, " - f"got {getattr(T_arr, 'shape', type(T).__name__)}. " - "See CONTRIBUTING_DATA.md §6.3." - ) - writer = ZarrWriter( - episode_path=episode_path, + return ZarrWriter( + episode_path, embodiment=embodiment, - fps=fps, + intrinsics=intrinsics, task_name=task_name, + operator=operator, + lab=lab, + scene=scene, + objects=objects, + source_recording_id=source_recording_id, + fps=fps, task_description=task_description, annotations=annotations, chunk_timesteps=chunk_timesteps, - intrinsics=intrinsics, extrinsics=extrinsics, - ) - - writer.write( + ).write( numeric_data=numeric_data, image_data=image_data, pre_encoded_image_data=pre_encoded_image_data, metadata_override=metadata_override, ) - return writer.episode_path + +def _sibling(episode_path: Path, kind: str) -> Path: + return episode_path.with_name(f"{episode_path.name}.{kind}-{os.getpid()}") + + +def _remove_stale_siblings(episode_path: Path) -> None: + """Drop temp/old dirs left by a writer that was killed mid-write (OOM, + SIGKILL), whose own cleanup never ran. Assumes one writer per episode.""" + name = glob.escape(episode_path.name) + for stale in episode_path.parent.glob(f"{name}.tmp-*"): + _remove(stale) + olds = sorted( + episode_path.parent.glob(f"{name}.old-*"), + key=lambda p: p.stat().st_mtime, + ) + # Killed between the two renames in _swap_in: the .old copy is the episode. + if olds and not episode_path.exists(): + os.replace(olds.pop(), episode_path) + for stale in olds: + _remove(stale) + + +def _remove(path: Path) -> None: + if path.is_dir() and not path.is_symlink(): + shutil.rmtree(path, ignore_errors=True) + else: + path.unlink(missing_ok=True) + + +def _swap_in(new: Path, target: Path) -> None: + """Replace ``target`` with ``new``; the old episode stays on disk until the + new one is in place.""" + if not target.exists(): + os.replace(new, target) + return + old = _sibling(target, "old") + os.replace(target, old) + try: + os.replace(new, target) + except BaseException: + os.replace(old, target) + raise + shutil.rmtree(old, ignore_errors=True) + + +def _write_annotation_array( + store: zarr.Group, + annotations: list[tuple[str, int, int]], + annotation_key: str, +) -> dict[str, Any]: + """Write (text, start_idx, end_idx) annotations as JSON-encoded bytes. + + Returns the array's ``features`` entry. + """ + encoded = np.array( + [ + json.dumps( + { + "text": text, + "start_idx": int(start_idx), + "end_idx": int(end_idx), + }, + ensure_ascii=False, + separators=(",", ":"), + ).encode("utf-8") + for text, start_idx, end_idx in annotations + ], + dtype=object, + ) + + n_annotations = len(annotations) + # Both the chunk and the SHARD edge must be >= 1 even when there are no + # annotations: zarr v3 rejects a zero-length chunk grid, so an episode + # written with shards=(0,) cannot be enumerated (group.keys() raises + # "integer chunk edge length must be >= 1") even though the array itself + # reads back fine. shape=(0,) with a >=1 edge is the correct encoding of + # "empty" -- no chunk objects are written either way. + store.create_array( + annotation_key, + shape=encoded.shape, + chunks=(max(1, n_annotations),), + shards=(max(1, n_annotations),), + dtype=VariableLengthBytes(), + ) + if n_annotations > 0: + store[annotation_key][:] = encoded + + return { + "dtype": "json", + "shape": [n_annotations], + "names": ["json"], + "format": "annotation_v1", + } + + +# ── Editing an existing episode ───────────────────────────────────────────── +# These work on an episode that is already written, so they need none of the +# metadata a ZarrWriter requires. Each keeps zarr.attrs["features"] in sync +# with the arrays (the episode validator checks that). + + +def key_exists(episode_path: str | Path, key: str) -> bool: + """Whether ``key`` is an array/group in the episode store.""" + return key in zarr.open(str(episode_path), mode="r", zarr_format=3) + + +def remove_key(episode_path: str | Path, key: str) -> None: + """Delete ``key`` from the episode store and from its features. + + Raises KeyError if the episode has no such key. + """ + store = zarr.open_group(str(episode_path), mode="a", zarr_format=3) + _remove_key(store, key) + + +def _remove_key(store: zarr.Group, key: str) -> None: + if key not in store: + raise KeyError(f"{key!r} is not in {store.store}") + del store[key] + features = dict(store.attrs.get("features", {})) + if features.pop(key, None) is not None: + store.attrs["features"] = features + + +def write_annotations( + episode_path: str | Path, + annotations: list[tuple[str, int, int]], + annotation_key: str = "annotations", + mode: Literal["w", "a"] = "a", +) -> None: + """Write language annotations into an existing episode. + + Args: + episode_path: Path to the episode .zarr directory. + annotations: List of (text, start_idx, end_idx) tuples. + annotation_key: Array key to write (default "annotations"). + mode: "w" replaces an existing ``annotation_key``; "a" only adds a new + key and raises if it already exists. + """ + store = zarr.open_group(str(episode_path), mode="a", zarr_format=3) + if annotation_key in store: + if mode != "w": + raise ValueError( + f"{annotation_key!r} already exists in {episode_path}; " + 'pass mode="w" to replace it' + ) + _remove_key(store, annotation_key) + feature = _write_annotation_array(store, annotations, annotation_key) + features = dict(store.attrs.get("features", {})) + features[annotation_key] = feature + store.attrs["features"] = features diff --git a/egomimic/scripts/aria_process/aria_to_zarr.py b/egomimic/scripts/aria_process/aria_to_zarr.py index 9548b58f85..496b57cf6b 100644 --- a/egomimic/scripts/aria_process/aria_to_zarr.py +++ b/egomimic/scripts/aria_process/aria_to_zarr.py @@ -8,6 +8,11 @@ import numpy as np from egomimic.rldb.embodiment.human import ARIA_INTRINSICS +from egomimic.rldb.zarr.required_metadata import ( + add_required_metadata_args, + check_writer_metadata, + required_metadata_from_args, +) from egomimic.rldb.zarr.zarr_writer import ZarrWriter from egomimic.scripts.aria_process.aria_utils import AriaVRSExtractor from egomimic.utils.aws.aws_sql import timestamp_ms_to_episode_hash @@ -34,7 +39,7 @@ class DatasetConverter: Whether the images are compressed, by default True. Methods ------- - extract_episode(episode_path, task_name='', output_dir='.', dataset_name='', chunk_timesteps=100) + extract_episode(episode_path, required_metadata=..., output_dir='.', dataset_name='', chunk_timesteps=100) Extracts frames from a single episode and saves it with a description. main(args) Main function to convert the dataset. @@ -122,7 +127,8 @@ def __init__( def extract_episode( self, episode_path, - task_name: str = "", + *, + required_metadata: dict, task_description: str = "", output_dir: Path = Path("."), dataset_name: str = "", @@ -136,6 +142,9 @@ def extract_episode( The path to the episode file. task_description : str, optional A description of the task associated with the episode (default is an empty string). + required_metadata : dict + task_name/operator/lab/scene/objects kwargs for create_and_write (see + egomimic.rldb.zarr.required_metadata.required_metadata_from_args). Returns ------- None @@ -176,10 +185,11 @@ def extract_episode( image_data=image_data if image_data else None, fps=self.fps, embodiment=self.embodiment, - task_name=task_name, task_description=task_description, chunk_timesteps=chunk_timesteps, intrinsics={"front_1": ARIA_INTRINSICS}, + source_recording_id=Path(episode_path).stem, + **required_metadata, ) if self.save_mp4: mp4_path = output_dir / f"{episode_name}.mp4" @@ -201,6 +211,8 @@ def main(args) -> None: try: episode_hash = timestamp_ms_to_episode_hash(Path(args.raw_path).stem) + required_metadata = required_metadata_from_args(args) + check_writer_metadata(required_metadata) converter = DatasetConverter( raw_path=Path(args.raw_path), @@ -224,10 +236,10 @@ def main(args) -> None: ctypes.CDLL("libc.so.6").malloc_trim(0) zarr_path, mp4_path = converter.extract_episode( episode_path=Path(args.raw_path), - task_name=args.task_name, task_description=args.task_description, output_dir=Path(args.output_dir), dataset_name=episode_hash, + required_metadata=required_metadata, ) return zarr_path, mp4_path except Exception: @@ -256,8 +268,8 @@ def argument_parse(): parser.add_argument( "--task-name", type=str, - default="Aria recorded dataset.", - help="Task name of the data.", + default=None, + help="Task name of the data (must match the DB row's task).", ) parser.add_argument( "--task-description", @@ -272,6 +284,7 @@ def argument_parse(): default="both", help="Specify the arm for processing.", ) + add_required_metadata_args(parser) parser.add_argument( "--image-compressed", type=str2bool, diff --git a/egomimic/scripts/convert_template/template_to_zarr.py b/egomimic/scripts/convert_template/template_to_zarr.py new file mode 100644 index 0000000000..47aad6af24 --- /dev/null +++ b/egomimic/scripts/convert_template/template_to_zarr.py @@ -0,0 +1,345 @@ +""" +Reference converter: your raw episodes -> EgoVerse zarr episodes. + +Copy this file and replace the two functions marked YOUR CODE: + +- ``find_raw_episodes(raw_root)`` — list one entry per raw episode. +- ``load_episode(raw_path)`` — read one raw episode into a ``RawEpisode``: + arrays in the shapes / frames below plus that episode's metadata. + +Everything else (writing the zarr, the episode hash name, validation, the MP4 +preview, batch conversion) is done for you. As shipped, ``load_episode`` +returns synthetic data with ``lab="template"``, which validation rejects, so +every episode fails until you put in your own metadata: + + python egomimic/scripts/convert_template/template_to_zarr.py \\ + --raw-root /path/to/raw --output-dir /path/to/out --workers 8 + +or from your own batch script: + + from template_to_zarr import convert_all + ok, failed = convert_all(paths, output_dir, workers=8) + +Each episode is validated right after it is written (CONTRIBUTING_DATA.md +§11.1); an episode that fails is deleted and reported, so whatever ends up in +``output_dir`` is ready to upload. An episode with scene or objects set to +MissingMetadata() fails unless you pass ``--allow-missing`` +(``allow_missing=True``). The MP4 preview needs ffmpeg; if it fails the episode +is kept and a warning is logged (``--no-mp4`` skips it). Spec: +CONTRIBUTING_DATA.md §4 (episode hash), §6 (arrays + metadata), §7 (coordinate +frames). +""" + +import argparse +import fcntl +import hashlib +import logging +import os +import shutil +import traceback +from concurrent.futures import ProcessPoolExecutor +from contextlib import contextmanager +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any + +import numpy as np + +from egomimic.rldb.zarr.episode_rules import start_time_error +from egomimic.rldb.zarr.required_metadata import ( + MissingMetadata, # noqa: F401 (see load_episode) +) +from egomimic.rldb.zarr.validate import validate_episode +from egomimic.rldb.zarr.zarr_writer import ZarrWriter +from egomimic.utils.aws.aws_sql import timestamp_ms_to_episode_hash +from egomimic.utils.video_utils import save_preview_mp4 + +logger = logging.getLogger(__name__) + + +@dataclass +class RawEpisode: + """One episode, already in EgoVerse conventions. + + Poses are (T, 7) = xyz + quaternion **wxyz**, expressed in your SLAM world + frame (§7.1), at the RGB frame rate — interpolate first if your tracker is + slower (§3.1). Leave a hand as None if it isn't tracked. + """ + + start_time_ms: int # UTC epoch ms at recording start -> episode hash (§4) + fps: int # actual capture rate of `images` (30 or 60) + images: np.ndarray # (T, H, W, 3) uint8 RGB, egocentric camera + intrinsics: np.ndarray # 3x3 or 3x4 pinhole K at the (H, W) of `images` + head_pose: np.ndarray # (T, 7) world <- camera (§6.2) + metadata: dict[str, Any] # task_name, operator, lab, scene, objects (§6.3) + # Your ID for the raw recording this episode was cut from, so it can be + # re-converted from the raw upload later (§6.6). + source_recording_id: str + left_ee_pose: np.ndarray | None = None # (T, 7) + right_ee_pose: np.ndarray | None = None # (T, 7) + left_keypoints: np.ndarray | None = None # (T, 63) = 21 MANO kps x xyz + right_keypoints: np.ndarray | None = None # (T, 63) + task_description: str = "" + # Language spans: (text, start_frame, end_frame), end exclusive (§8.1) + annotations: list[tuple[str, int, int]] = field(default_factory=list) + + +# ── YOUR CODE ─────────────────────────────────────────────────────────────── + + +def find_raw_episodes(raw_root: Path) -> list[Path]: + """Return one path per raw episode under ``raw_root``. + + Replace with however your raw data is laid out (a folder per episode, + one file per episode, rows of a manifest, ...). + """ + return sorted(p for p in Path(raw_root).iterdir() if not p.name.startswith(".")) + + +def load_episode(raw_path: Path) -> RawEpisode: + """Read one raw episode. Replace this body with your own loader. + + ``metadata`` is per episode: read the operator, scene and objects from your + own capture logs rather than hardcoding them for the whole batch — they + are what lets people filter and train on the data. task_name, operator + and lab are always required. If your source truly has no scene or + objects, use MissingMetadata() for it — only if absolutely necessary, and + convert with --allow-missing. + """ + # Synthetic stand-in so the template runs as shipped. + seed = int(hashlib.sha256(Path(raw_path).name.encode()).hexdigest()[:8], 16) + rng = np.random.default_rng(seed) + T, H, W = 60, 360, 640 + return RawEpisode( + start_time_ms=1_767_225_600_000 + seed, # 2026-01-01 UTC + offset + fps=30, + images=rng.integers(0, 255, (T, H, W, 3), dtype=np.uint8), + intrinsics=np.array( + [[250.0, 0.0, W / 2], [0.0, 250.0, H / 2], [0.0, 0.0, 1.0]] + ), + head_pose=_synthetic_trajectory(T, rng), + left_ee_pose=_synthetic_trajectory(T, rng), + right_ee_pose=_synthetic_trajectory(T, rng), + task_description="folding a towel on a table", + source_recording_id=Path(raw_path).name, + annotations=[("fold the towel", 0, T)], + metadata={ + "task_name": "fold_clothes", + "operator": f"operator_{seed % 5}", + "lab": "template", # rejected: replace with your lab id + "scene": "kitchen_a", + "objects": ["towel"], + # Only if absolutely necessary, e.g. your capture logs have no scene: + # "scene": MissingMetadata(), + }, + ) + + +# ── Conversion (no changes needed below) ─────────────────────────────────── + + +def convert_episode( + raw_path: Path, + output_dir: Path, + *, + save_mp4: bool = True, + allow_missing: bool = False, +) -> Path: + """Convert and validate one raw episode; returns the written .zarr path. + + Raises (and leaves nothing behind) if the episode fails validation. + """ + ep = load_episode(Path(raw_path)) + if (err := start_time_error(ep.start_time_ms)) is not None: + raise ValueError(f"{raw_path}: {err}") + output_dir = Path(output_dir) + episode_hash = timestamp_ms_to_episode_hash(ep.start_time_ms) + zarr_path = output_dir / f"{episode_hash}.zarr" + mp4_path = output_dir / f"{episode_hash}.mp4" + + numeric = {"obs_head_pose": ep.head_pose} + for side in ("left", "right"): + if getattr(ep, f"{side}_ee_pose") is not None: + numeric[f"{side}.obs_ee_pose"] = getattr(ep, f"{side}_ee_pose") + if getattr(ep, f"{side}_keypoints") is not None: + numeric[f"{side}.obs_keypoints"] = getattr(ep, f"{side}_keypoints") + + K = np.asarray(ep.intrinsics, dtype=np.float64) + if K.shape == (3, 3): + K = np.hstack([K, np.zeros((3, 1))]) + + output_dir.mkdir(parents=True, exist_ok=True) + lock_path = output_dir / f"{episode_hash}.lock" + with _hash_lock(lock_path, zarr_path): + if zarr_path.exists(): + raise FileExistsError(f"{zarr_path} already exists (duplicate start time?)") + _write_and_validate( + ep, zarr_path, mp4_path, numeric, K, save_mp4, allow_missing + ) + return zarr_path + + +@contextmanager +def _hash_lock(lock_path: Path, zarr_path: Path): + """Exclusive per-hash lock so parallel workers that share a start time can't + overwrite each other. flock is released if the worker is killed, so a crash + never blocks a rerun.""" + while True: + lock = open(lock_path, "w") + try: + fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) + except BlockingIOError: + lock.close() + raise FileExistsError( + f"{zarr_path} is being written by another worker " + "(duplicate start time?)" + ) from None + # Another worker may have unlinked the file between our open and flock; + # a lock on an unlinked inode excludes nobody, so take a fresh one. + try: + if os.stat(lock_path).st_ino == os.fstat(lock.fileno()).st_ino: + break + except FileNotFoundError: + pass + lock.close() + try: + yield + finally: + lock_path.unlink(missing_ok=True) + lock.close() + + +def _write_and_validate( + ep: RawEpisode, + zarr_path: Path, + mp4_path: Path, + numeric: dict[str, np.ndarray], + K: np.ndarray, + save_mp4: bool, + allow_missing: bool, +) -> None: + """Write and validate one episode, removing it on failure; then the preview.""" + try: + ZarrWriter.create_and_write( + episode_path=zarr_path, + numeric_data={k: np.asarray(v) for k, v in numeric.items()}, + image_data={"images.front_1": ep.images}, + embodiment=_embodiment(ep), + fps=ep.fps, + task_description=ep.task_description, + annotations=ep.annotations or None, + intrinsics={"front_1": K}, + source_recording_id=ep.source_recording_id, + **ep.metadata, + ) + report = validate_episode(zarr_path, allow_missing=allow_missing) + if not report.ok: + raise ValueError( + f"{zarr_path.name} failed validation:\n " + "\n ".join(report.errors) + ) + for warning in report.warnings: + logger.warning("%s: %s", zarr_path.name, warning) + except BaseException: + shutil.rmtree(zarr_path, ignore_errors=True) + raise + if save_mp4: # sibling preview for review (§6.5); the zarr is valid without it + try: + save_preview_mp4(ep.images.transpose(0, 3, 1, 2), mp4_path, ep.fps) + except Exception as e: + mp4_path.unlink(missing_ok=True) + logger.warning("%s: MP4 preview not written: %s", zarr_path.name, e) + + +def convert_all( + raw_paths: list[Path], + output_dir: Path, + *, + workers: int = 1, + save_mp4: bool = True, + allow_missing: bool = False, +) -> tuple[list[Path], dict[Path, str]]: + """Convert many episodes; one bad episode never stops the batch. + + Returns (written zarr paths, {raw_path: error message} for failures). + """ + output_dir = Path(output_dir) + output_dir.mkdir(parents=True, exist_ok=True) + written: list[Path] = [] + failed: dict[Path, str] = {} + + def _record(raw_path, fn): + try: + written.append(fn()) + logger.info("ok %s -> %s", raw_path, written[-1].name) + except Exception: + failed[raw_path] = traceback.format_exc() + logger.error("FAILED %s\n%s", raw_path, failed[raw_path]) + + opts = {"save_mp4": save_mp4, "allow_missing": allow_missing} + if workers <= 1: + for p in raw_paths: + _record(p, lambda p=p: convert_episode(p, output_dir, **opts)) + else: + with ProcessPoolExecutor(max_workers=workers) as pool: + futures = { + p: pool.submit(convert_episode, p, output_dir, **opts) + for p in raw_paths + } + for p, fut in futures.items(): + _record(p, fut.result) + return written, failed + + +def _embodiment(ep: RawEpisode) -> str: + # All human data shares one embodiment; your lab goes in `lab` (§9). + has_left, has_right = ep.left_ee_pose is not None, ep.right_ee_pose is not None + if has_left and has_right: + return "human_bimanual" + if has_left or has_right: + return "human_left_arm" if has_left else "human_right_arm" + raise ValueError("episode has no tracked hand (left_ee_pose / right_ee_pose)") + + +def _synthetic_trajectory(T: int, rng: np.random.Generator) -> np.ndarray: + """Smooth (T, 7) xyz + wxyz trajectory: drift plus a slow yaw.""" + xyz = np.cumsum(rng.normal(0, 0.002, (T, 3)), axis=0) + rng.normal(0, 0.3, 3) + yaw = np.linspace(0, rng.uniform(-0.5, 0.5), T) + quat = np.stack( + [np.cos(yaw / 2), np.zeros(T), np.zeros(T), np.sin(yaw / 2)], axis=1 + ) + return np.concatenate([xyz, quat], axis=1) + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__.strip().splitlines()[0]) + parser.add_argument("--raw-root", type=Path, required=True) + parser.add_argument("--output-dir", type=Path, required=True) + parser.add_argument("--workers", type=int, default=1) + parser.add_argument( + "--no-mp4", action="store_true", help="Skip MP4 previews (need ffmpeg)" + ) + parser.add_argument( + "--allow-missing", + action="store_true", + help="Keep episodes with scene/objects set to MissingMetadata(). Only " + "if absolutely necessary.", + ) + args = parser.parse_args(argv) + + raw_paths = find_raw_episodes(args.raw_root) + written, failed = convert_all( + raw_paths, + args.output_dir, + workers=args.workers, + save_mp4=not args.no_mp4, + allow_missing=args.allow_missing, + ) + print(f"{len(written)}/{len(raw_paths)} episodes written to {args.output_dir}") + for p in failed: + print(f" FAILED {p}") + return 1 if failed else 0 + + +if __name__ == "__main__": + logging.basicConfig(level=logging.INFO, format="%(message)s") + raise SystemExit(main()) diff --git a/egomimic/scripts/eva_process/eva_to_zarr.py b/egomimic/scripts/eva_process/eva_to_zarr.py index 01316a9bcc..b7568b08b2 100644 --- a/egomimic/scripts/eva_process/eva_to_zarr.py +++ b/egomimic/scripts/eva_process/eva_to_zarr.py @@ -14,6 +14,11 @@ from scipy.spatial.transform import Rotation as R from egomimic.rldb.embodiment.eva import Eva +from egomimic.rldb.zarr.required_metadata import ( + add_required_metadata_args, + check_writer_metadata, + required_metadata_from_args, +) from egomimic.rldb.zarr.zarr_writer import ZarrWriter from egomimic.scripts.eva_process.eva_utils import EvaHD5Extractor from egomimic.utils.aws.aws_sql import timestamp_ms_to_episode_hash @@ -161,13 +166,17 @@ def convert_episode( dataset_name: str, arm: str, fps: int, - task_name: str = "", + *, + required_metadata: dict, task_description: str = "", save_mp4: bool = False, chunk_timesteps: int = 100, ) -> tuple[Path, Path]: """Process one HDF5 file and write a .zarr episode. + ``required_metadata`` holds the task_name/operator/lab/scene/objects kwargs for + create_and_write (see egomimic.rldb.zarr.required_metadata.required_metadata_from_args). + Returns the zarr episode path on success. """ @@ -192,11 +201,12 @@ def convert_episode( image_data=image_data or None, embodiment=embodiment, fps=fps, - task_name=task_name, task_description=task_description, chunk_timesteps=chunk_timesteps, intrinsics={"front_1": Eva.INTRINSICS}, extrinsics=Eva.EXTRINSICS, + source_recording_id=Path(raw_path).stem, + **required_metadata, ) logger.info("Wrote zarr episode: %s", zarr_path) @@ -230,16 +240,18 @@ def main(args) -> None: try: episode_hash = timestamp_ms_to_episode_hash(Path(args.raw_path).stem) + required_metadata = required_metadata_from_args(args) + check_writer_metadata(required_metadata) zarr_path, mp4_path = convert_episode( raw_path=Path(args.raw_path), output_dir=Path(args.output_dir), dataset_name=episode_hash, arm=args.arm, fps=args.fps, - task_name=args.task_name, task_description=args.task_description, save_mp4=args.save_mp4, chunk_timesteps=args.chunk_timesteps, + required_metadata=required_metadata, ) return zarr_path, mp4_path except Exception: @@ -272,8 +284,11 @@ def argument_parse(): help="Timesteps per zarr chunk for numeric arrays.", ) parser.add_argument("--debug", action="store_true") - parser.add_argument("--task-name", type=str, default="") + parser.add_argument( + "--task-name", type=str, default=None, help="Must match the DB row's task." + ) parser.add_argument("--task-description", type=str, default="") + add_required_metadata_args(parser) return parser.parse_args() diff --git a/egomimic/scripts/language_process/bucket_to_zarr_annotation_parallel.py b/egomimic/scripts/language_process/bucket_to_zarr_annotation_parallel.py index 9d7f71ffba..bcea99044d 100644 --- a/egomimic/scripts/language_process/bucket_to_zarr_annotation_parallel.py +++ b/egomimic/scripts/language_process/bucket_to_zarr_annotation_parallel.py @@ -2,7 +2,7 @@ Ray-parallel: download per-episode annotation JSONs from a bucket and write into Zarr. Counterpart to scale_to_bucket_annotation_parallel.py. Each Ray task: - download s3://{bucket}/{prefix}/{episode_hash}_{annotation_key}.json -> ZarrWriter. + download s3://{bucket}/{prefix}/{episode_hash}_{annotation_key}.json -> zarr_writer.write_annotations. Episodes that already have ``annotation_key`` in their Zarr are skipped (unless --overwrite). @@ -51,11 +51,10 @@ def process_episode( overwrite: bool = False, ) -> str: """Download annotation JSON from bucket and write into the episode's Zarr.""" - from egomimic.rldb.zarr.zarr_writer import ZarrWriter + from egomimic.rldb.zarr.zarr_writer import key_exists, write_annotations from egomimic.utils.aws.aws_data_utils import get_boto3_s3_client - writer = ZarrWriter(episode_path=episode_path, verbose=True) - if not overwrite and writer.check_key_exists(episode_path, annotation_key): + if not overwrite and key_exists(episode_path, annotation_key): print(f"[SKIP] {episode_hash} -> {annotation_key} already in zarr") return episode_hash @@ -76,8 +75,8 @@ def process_episode( for entry in payload ] - writer.append_annotations( - annotation_key=annotation_key, annotations=annotations, mode="w" + write_annotations( + episode_path, annotations, annotation_key=annotation_key, mode="w" ) print(f"[OK] {episode_hash} -> {episode_path} ({len(annotations)} entries)") return episode_hash diff --git a/egomimic/scripts/language_process/scale_to_zarr_annotation.py b/egomimic/scripts/language_process/scale_to_zarr_annotation.py index a1a4209cdf..a845b0f69a 100644 --- a/egomimic/scripts/language_process/scale_to_zarr_annotation.py +++ b/egomimic/scripts/language_process/scale_to_zarr_annotation.py @@ -17,7 +17,7 @@ from omegaconf import OmegaConf from scaleapi import ScaleClient -from egomimic.rldb.zarr.zarr_writer import ZarrWriter +from egomimic.rldb.zarr.zarr_writer import write_annotations from egomimic.scripts.language_process.converter import ( HardCodedConverter, PickPlaceLLMConverter, @@ -119,12 +119,8 @@ def download_scale_annotation_csv(dest_path: str): if tid is None: continue annotation = converter.convert(tid) - writer = ZarrWriter( - episode_path=zarr_dataset.episode_path, - verbose=True, - ) - writer.append_annotations( - annotation_key="annotations", annotations=annotation, mode="w" + write_annotations( + zarr_dataset.episode_path, annotation, "annotations", mode="w" ) for dataset_name in valid_datasets: @@ -133,10 +129,6 @@ def download_scale_annotation_csv(dest_path: str): if tid is None: continue annotation = converter.convert(tid) - writer = ZarrWriter( - episode_path=zarr_dataset.episode_path, - verbose=True, - ) - writer.append_annotations( - annotation_key="annotations", annotations=annotation, mode="w" + write_annotations( + zarr_dataset.episode_path, annotation, "annotations", mode="w" ) diff --git a/egomimic/scripts/language_process/scale_to_zarr_annotation_parallel.py b/egomimic/scripts/language_process/scale_to_zarr_annotation_parallel.py index 98b54201f2..c840e3faf2 100644 --- a/egomimic/scripts/language_process/scale_to_zarr_annotation_parallel.py +++ b/egomimic/scripts/language_process/scale_to_zarr_annotation_parallel.py @@ -90,10 +90,9 @@ def process_episode( Self-contained Ray task: download, convert, and write one episode's annotations. Returns the episode_hash on success. """ - from egomimic.rldb.zarr.zarr_writer import ZarrWriter + from egomimic.rldb.zarr.zarr_writer import key_exists, write_annotations - writer = ZarrWriter(episode_path=episode_path, verbose=True) - if not overwrite and writer.check_key_exists(episode_path, annotation_key): + if not overwrite and key_exists(episode_path, annotation_key): print(f"[SKIP] {episode_hash} -> {annotation_key} already exists") return episode_hash @@ -107,9 +106,7 @@ def process_episode( ) annotation = converter.convert(tid) - writer.append_annotations( - annotation_key=annotation_key, annotations=annotation, mode="w" - ) + write_annotations(episode_path, annotation, annotation_key=annotation_key, mode="w") print(f"[OK] {episode_hash} -> {episode_path}") return episode_hash diff --git a/egomimic/scripts/mecka_process/mecka_to_zarr.py b/egomimic/scripts/mecka_process/mecka_to_zarr.py index f00cebe38e..486287194f 100644 --- a/egomimic/scripts/mecka_process/mecka_to_zarr.py +++ b/egomimic/scripts/mecka_process/mecka_to_zarr.py @@ -706,6 +706,14 @@ def _extract_head_poses(egomotion: np.ndarray) -> np.ndarray: return np.asarray(adjusted_head_poses) +# The pose helpers above are still used by training (left_wrist_convention); +# only the zarr conversion path is disabled. +_CONVERTER_DISABLED = ( + "mecka_to_zarr is disabled until it passes the required episode metadata " + "and intrinsics to ZarrWriter; see INTERNAL_DATA.md §4.3." +) + + class MeckaDatasetConverter: """Convert Mecka episodes to Zarr dataset format using ZarrWriter.""" @@ -734,6 +742,7 @@ def __init__( local_data_dir: Optional path to pre-downloaded episode files; skips download. task_description: Optional task label for annotations. """ + raise NotImplementedError(_CONVERTER_DISABLED) self.episode_json_path = episode_json_path self.repo_id = repo_id self.arm = arm @@ -934,6 +943,7 @@ def main() -> None: local data directory. Creates MeckaDatasetConverter and runs extract_episode. Removes existing output directory before conversion. Exits with traceback on error. """ + raise NotImplementedError(_CONVERTER_DISABLED) parser = argparse.ArgumentParser(description="Convert Mecka RL2 to LeRobot format") parser.add_argument( "--episode-json", required=True, help="Path to episode JSON file" diff --git a/egomimic/scripts/ray_helper.py b/egomimic/scripts/ray_helper.py index 1652054be2..27ff9d9e3a 100644 --- a/egomimic/scripts/ray_helper.py +++ b/egomimic/scripts/ray_helper.py @@ -240,7 +240,7 @@ def iter_bundles(self): vrs = vrs_by_name[filename] jsonf = root / f"{filename}.json" mps_dir = root / f"mps_{filename}_vrs" - # arm, task_name, task_description are inferred from row in run_converion.py + # arm, task_name, task_description, required_metadata are inferred from row in run_conversion.py args = { "processed_local_root": str(self.processed_local_root), "processed_remote_prefix": self.processed_remote_prefix, @@ -274,6 +274,7 @@ def convert_one_bundle( fps: int, task_name: str, task_description: str, + required_metadata: dict, chunk_timesteps: int, image_compressed: bool, save_mp4: bool, @@ -356,6 +357,7 @@ def convert_one_bundle( fps=fps, task_name=task_name, task_description=task_description, + required_metadata=required_metadata, chunk_timesteps=chunk_timesteps, image_compressed=image_compressed, save_mp4=save_mp4, @@ -441,6 +443,8 @@ def zarr_job( raw_path: str | Path, output_dir: str | Path, arm: str, + *, + required_metadata: dict, fps: int = 30, task_name: str = "", task_description: str = "", @@ -455,6 +459,7 @@ def zarr_job( fps=fps, task_name=task_name, task_description=task_description, + **required_metadata, chunk_timesteps=chunk_timesteps, image_compressed=image_compressed, save_mp4=save_mp4, @@ -526,6 +531,7 @@ def convert_one_bundle( fps: int, task_name: str, task_description: str, + required_metadata: dict, chunk_timesteps: int, save_mp4: bool, ) -> tuple[str, str, int]: @@ -586,6 +592,7 @@ def convert_one_bundle( fps=fps, task_name=task_name, task_description=task_description, + required_metadata=required_metadata, chunk_timesteps=chunk_timesteps, save_mp4=save_mp4, ) @@ -668,6 +675,8 @@ def zarr_job( raw_path: str | Path, output_dir: str | Path, arm: str, + *, + required_metadata: dict, fps: int = 30, save_mp4: bool = True, task_name: str = "", @@ -688,6 +697,7 @@ def zarr_job( save_mp4=save_mp4, task_name=task_name, task_description=task_description, + **required_metadata, chunk_timesteps=chunk_timesteps, ) diff --git a/egomimic/scripts/run_conversion.py b/egomimic/scripts/run_conversion.py index 93ce4db8e8..6c8bd2ca13 100644 --- a/egomimic/scripts/run_conversion.py +++ b/egomimic/scripts/run_conversion.py @@ -7,12 +7,18 @@ import os import time from pathlib import Path +from types import SimpleNamespace from typing import Any, Dict import ray from cloudpathlib import S3Path from ray.exceptions import OutOfMemoryError, RayTaskError, WorkerCrashedError +from egomimic.rldb.zarr.required_metadata import ( + WAIVABLE_FIELDS, + check_writer_metadata, + required_metadata_from_args, +) from egomimic.scripts.ray_helper import AriaRay, EmbodimentRay, EvaRay from egomimic.utils.aws.aws_data_utils import ( get_cloudpathlib_s3_client, @@ -89,10 +95,68 @@ def infer_arm_from_row(row: TableRow) -> str: def infer_task_name_from_row(row: TableRow) -> str: """ - Infer task name from SQL row.task_name (e.g., 'task_name'). - Falls back to ''. + Infer task name from SQL row.task (e.g., 'fold_clothes'). + Falls back to '', which required_metadata_row_error rejects. + """ + return (row.task or "").strip() + + +_NULL_OBJECTS = frozenset({"", "none", "null"}) + + +def parse_db_objects(raw) -> list[str]: + """Object names from the DB objects column. + + Most rows hold a Postgres array literal ('{"pink cup",bag}'), some a plain + comma-separated string; a driver may also hand back a list. None/NULL + entries are dropped. + """ + if isinstance(raw, (list, tuple)): + entries = [str(e) for e in raw if e is not None] + elif isinstance(raw, str): + text = raw.strip() + if text.startswith("{") and text.endswith("}"): + text = text[1:-1] + reader = csv.reader([text], skipinitialspace=True, escapechar="\\") + entries = next(reader, []) + else: + entries = [] + return [e.strip() for e in entries if e.strip().lower() not in _NULL_OBJECTS] + + +def infer_required_metadata_from_row(row: TableRow) -> dict: """ - return row.task or "unknown" + operator/lab/scene/objects for the converter, plus a missing_metadata list + naming the waivable fields (scene, objects) the row left empty, so they are + written as an explicit MissingMetadata(). An empty task, operator or lab is + left as None and rejected by required_metadata_row_error. Plain data so it + travels through Ray. + """ + values = { + "task_name": infer_task_name_from_row(row), + "operator": (row.operator or "").strip(), + "lab": (row.lab or "").strip(), + "scene": (row.scene or "").strip(), + "objects": parse_db_objects(row.objects), + } + missing = [field for field in WAIVABLE_FIELDS if not values[field]] + del values["task_name"] + return {**{k: v or None for k, v in values.items()}, "missing_metadata": missing} + + +def required_metadata_row_error( + task_name: str | None, required_metadata: dict +) -> str | None: + """First reason the row's metadata would be rejected by ZarrWriter, or None.""" + try: + check_writer_metadata( + required_metadata_from_args( + SimpleNamespace(task_name=task_name, **required_metadata) + ) + ) + except ValueError as e: + return str(e) + return None def infer_task_description_from_row(row: TableRow) -> str: @@ -256,8 +320,21 @@ def launch( # continue args["arm"] = arm - args["task_name"] = task_name + args["task_name"] = task_name or None + args["required_metadata"] = infer_required_metadata_from_row(row) args["task_description"] = task_description + # Reject here, not after a full extraction, and record why. + metadata_error = required_metadata_row_error( + args["task_name"], args["required_metadata"] + ) + if metadata_error is not None: + print(f"[SKIP] {name}: {metadata_error}", flush=True) + if not dry and ( + table_row := episode_hash_to_table_row(engine, episode_key) + ): + table_row.zarr_processing_error = f"Metadata: {metadata_error}" + update_episode(engine, table_row) + continue start_time = time.time() ref = submit_convert(embodiment_ray, "small", **args) pending[ref] = { diff --git a/external/scale/scripts/sfs_to_egoverse_zarr.py b/external/scale/scripts/sfs_to_egoverse_zarr.py index 144ac1899e..e782cf4bcc 100644 --- a/external/scale/scripts/sfs_to_egoverse_zarr.py +++ b/external/scale/scripts/sfs_to_egoverse_zarr.py @@ -524,6 +524,10 @@ def convert_task_to_zarr( Returns a dict with keys: episodes, folder, task_desc, total_frames, output_dir. """ + raise NotImplementedError( + "sfs_to_egoverse_zarr is disabled until it passes the required episode " + "metadata and intrinsics to ZarrWriter; see INTERNAL_DATA.md §4.3." + ) t_start = time.perf_counter() if img_workers is None: img_workers = min(os.cpu_count() or 4, 8) diff --git a/external/scale/scripts/sfs_zarr_pipeline.py b/external/scale/scripts/sfs_zarr_pipeline.py index 77f4ca0613..af5d9806eb 100644 --- a/external/scale/scripts/sfs_zarr_pipeline.py +++ b/external/scale/scripts/sfs_zarr_pipeline.py @@ -275,6 +275,11 @@ def log_task_result( def main() -> int: + raise NotImplementedError( + "sfs_zarr_pipeline is disabled with sfs_to_egoverse_zarr until it passes " + "the required episode metadata and intrinsics to ZarrWriter; see " + "INTERNAL_DATA.md §4.3." + ) parser = argparse.ArgumentParser( description="End-to-end Scale SFS -> EgoVerse Zarr pipeline", formatter_class=argparse.RawDescriptionHelpFormatter, diff --git a/pyproject.toml b/pyproject.toml index 9d1213ec3b..1502ccec89 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -65,6 +65,10 @@ dependencies = [ "openai", "scikit-learn>=1.8.0", "scipy", + # Vendor MCAP deliveries (egomimic/rldb/mcap) + "mcap==1.5.0", + "mcap-protobuf-support==0.5.4", + "foxglove-schemas-protobuf==0.4.0", ] # Not declared: `openpi` (egomimic/algo/pi.py). The fork at external/openpi pins @@ -84,7 +88,11 @@ where = ["."] include = ["egomimic*"] [tool.setuptools.package-data] -egomimic = ["hydra_configs/**/*.yaml", "resources/*"] +egomimic = [ + "hydra_configs/**/*.yaml", + "resources/*", + "rldb/mcap/proto/*.proto", +] [tool.pytest.ini_options] testpaths = ["tests"] diff --git a/tests/fixtures/mcap_episodes.py b/tests/fixtures/mcap_episodes.py new file mode 100644 index 0000000000..92ebd7d0ae --- /dev/null +++ b/tests/fixtures/mcap_episodes.py @@ -0,0 +1,542 @@ +"""Tiny synthetic raw MCAP episodes (spec 0.1), valid unless a breakage is asked for. + +`write_mcap_episode` writes `episode_/episode.mcap` with every topic the spec +requires for the embodiment: ~2 s of 64x48 H.264 video, 200 Hz IMU, 30 Hz poses, etc. +It also writes the optional label topics (two /task spans with a /subtask-annotation +each, in annotation.mcap by default) and, for humans, /eye-gaze. +Each keyword-only switch breaks exactly one thing, so a test can assert the validator +reports that problem and nothing else. +""" + +from __future__ import annotations + +from collections.abc import Callable +from fractions import Fraction +from functools import lru_cache +from pathlib import Path + +import numpy as np +from foxglove_schemas_protobuf.CameraCalibration_pb2 import CameraCalibration +from foxglove_schemas_protobuf.CompressedVideo_pb2 import CompressedVideo +from foxglove_schemas_protobuf.FrameTransforms_pb2 import FrameTransforms +from foxglove_schemas_protobuf.PoseInFrame_pb2 import PoseInFrame +from mcap_protobuf.writer import Writer + +from egomimic.rldb.mcap import schemas, spec + +T0 = 1_760_000_000 * 10**9 # 2025-10-09, absolute Unix ns +DURATION_S = 2 +WIDTH, HEIGHT = 64, 48 +IMU_HZ = 200 +POSE_HZ = 30 +VIDEO_FPS = 30 + + +@lru_cache(maxsize=None) +def h264_packets(n_frames: int, b_frames: bool = False) -> tuple[bytes, ...]: + """Annex B H.264, one frame per packet, keyframe every 2 frames, SPS/PPS repeated. + With b_frames, x264's defaults instead (B-frames, keyframe every 30 frames).""" + import av + + enc = av.CodecContext.create("libx264", "w") + enc.width, enc.height, enc.pix_fmt = WIDTH, HEIGHT, "yuv420p" + enc.time_base = Fraction(1, VIDEO_FPS) + enc.options = ( + {"g": "30", "repeat-headers": "1"} + if b_frames + else {"bf": "0", "g": "2", "repeat-headers": "1", "tune": "zerolatency"} + ) + packets = [] + for i in range(n_frames): + img = np.full((HEIGHT, WIDTH, 3), i * 4 % 255, dtype=np.uint8) + frame = av.VideoFrame.from_ndarray(img, format="rgb24").reformat( + format="yuv420p" + ) + frame.pts = i + packets += [bytes(p) for p in enc.encode(frame)] + packets += [bytes(p) for p in enc.encode(None)] + return tuple(packets) + + +def _times(hz: float) -> list[int]: + return [T0 + round(i * 1e9 / hz) for i in range(int(DURATION_S * hz) + 1)] + + +def _metadata(embodiment: str, declared: str) -> dict[str, str]: + info = { + "task_name": "fold_clothes", + "operator": "op_7", + "lab": "rl2", + "scene": "kitchen_A", + "objects": "shirt,basket", + "task_description": "fold the shirt", + "embodiment": declared, + "rig_name": "test_rig", + "device_serial": "SN0001", + "start_time_unix": str(T0 // 10**6), + "end_time_unix": str((T0 + DURATION_S * 10**9) // 10**6), + "source_recording_id": "acme-rec-0001", + "spec_version": spec.SPEC_VERSION, + "license": "CC-BY-4.0", + } + if spec.profile_for(embodiment) is spec.HUMAN: + info["camera_timestamp_ref"] = "mid" + info["hand_joints"] = ",".join(spec.MANO_JOINTS) + info["shutter"] = "rolling:16ms" + return info + + +def _tf(links: list[tuple[str, str]], bad_rotation: bool) -> FrameTransforms: + tf = FrameTransforms() + for i, (parent, child) in enumerate(links): + x = tf.transforms.add(parent_frame_id=parent, child_frame_id=child) + x.translation.z = 0.1 + x.rotation.w = 2.0 if bad_rotation and i == 0 else 1.0 + return tf + + +def _rigid(z: float) -> list[float]: + T = np.eye(4) + T[:3, 3] = (0.3, 0.0, z) + return T.ravel().tolist() + + +def write_mcap_episode( + root: Path, + embodiment: str = "human_bimanual", + episode_id: str = "0", + *, + # 1. everything there + drop_topic: str | None = None, + drop_metadata_key: str | None = None, + declared_embodiment: str | None = None, + metadata_override: dict[str, str] | None = None, + unknown_hand_joint: bool = False, + empty_arm_pose: bool = False, + # 2. video + video_format: str = "h264", + skip_first_keyframe: bool = False, + b_frames: bool = False, + calibration_width: int = WIDTH, + video_fps: int = VIDEO_FPS, + # 3. clocks + backwards_head_pose: bool = False, + hand_gap_s: float = 0.0, + device_clock_imu_info: bool = False, + # 4. calibration + unlink_main_camera: bool = False, + bad_tf_rotation: bool = False, + bad_upright_rotation: bool = False, + # 5. imu + imu_hz: int = IMU_HZ, + gravity_removed: bool = False, + # 6. poses + nan_head_pose: bool = False, + zeroed_left_hand: bool = False, + gripper_out_of_range: bool = False, + bad_arm_pose: bool = False, + bad_hand_confidence: bool = False, + hands_wrist_only: bool = False, + bad_wrist_pose: bool = False, + nan_eye_gaze: bool = False, + joint_names_mismatch: bool = False, + multi_frame_video: bool = False, + # 7. tasks & labels + labels_in: str | None = "annotation", + task_past_end: bool = False, + label_outside_tasks: bool = False, + stray_annotation_topic: bool = False, + label_crosses_tasks: bool = False, + labels_without_tasks: bool = False, + label_past_video: bool = False, + label_at_last_frame: bool = False, + annotation_metadata: bool = False, + calibration_twice: bool = False, + principal_x: float = WIDTH / 2, + repeat_frame_time: bool = False, + tamper: Callable | None = None, + distortion_model: str = "", + distortion_coeffs: tuple[float, ...] = (), + bad_annotation_type: bool = False, +) -> Path: + """Write `root/episode_/episode.mcap` and return the episode dir. + + With no switches set, the episode passes the validator. Each switch breaks one + thing: + + drop_topic leave this topic out (e.g. "/imu-info", "/main-camera-info") + drop_metadata_key leave this episode-info key out + declared_embodiment write this as the "embodiment" key, while the topics + follow `embodiment` + metadata_override episode-info values to set as given + unknown_hand_joint "hand_joints" names "thumb_base", not in the vocabulary + empty_arm_pose robot: one arm's state `pose` left empty (as in ABC's + release) + video_format CompressedVideo.format on every camera + skip_first_keyframe cameras start on a P-frame + b_frames cameras are encoded with B-frames (x264 defaults) + calibration_width width in /main-camera-info (frames stay 64 wide) + video_fps frame rate of every camera + backwards_head_pose two /head-pose messages swap timestamps + hand_gap_s hole of this length in /left-hand-keypoints + device_clock_imu_info /imu-info stamped with device time (1 s), not Unix time + unlink_main_camera /tf_static has no transform for the main camera + bad_tf_rotation first /tf_static transform has a non-unit quaternion + bad_upright_rotation every camera's -info-extra says to turn it 45 degrees + imu_hz /imu sample rate + gravity_removed /imu reads ~0 m/s^2 at rest + nan_head_pose one /head-pose position is NaN + zeroed_left_hand /left-hand-keypoints all zero with valid=true + gripper_out_of_range robot: one /-ee-state value is 1.5 + bad_arm_pose robot: one /-arm-state pose has a scaled rotation + bad_hand_confidence one /left-hand-keypoints confidence is 1.5 + hands_wrist_only valid: hands carry only wrist_pose, no keypoints + bad_wrist_pose one /left-hand-keypoints wrist_pose has a non-unit rotation + joint_names_mismatch robot: one arm's state lists 5 joint names for 6 joints + multi_frame_video /main-camera (or /top-camera) packs 2 frames per message + nan_eye_gaze one valid /eye-gaze message has a NaN yaw + labels_in where the label topics go: "annotation" (annotation.mcap), + "episode" (inline, as ABC does) or None (left out) + task_past_end the second /task ends 3 s after the recorded data + label_outside_tasks only the first /task is written, so the second label is + in no task + stray_annotation_topic annotation.mcap also carries /imu + label_crosses_tasks the first label runs on into the second task + labels_without_tasks valid: labels but no /task (the episode info is the task) + label_past_video the point label sits after the last video frame + label_at_last_frame valid: the point label sits on the last video frame + annotation_metadata annotation.mcap carries its own session-metadata record + calibration_twice every camera's -info is published twice + principal_x cx in every camera's K (default: the image centre) + repeat_frame_time frame 21 of every camera repeats frame 20's timestamp + tamper tamper(topic, msg, t) -> None | (msg, t), applied to every + message before it is written (both files): any breakage + distortion_model, distortion_coeffs every camera's -info lens model and D + bad_annotation_type the labels are typed "summary" (only "language" is known) + """ + human = spec.profile_for(embodiment) is spec.HUMAN + sides = spec.arm_sides(embodiment) + out = Path(root) / f"episode_{episode_id}" + out.mkdir(parents=True, exist_ok=True) + + info = _metadata(embodiment, declared_embodiment or embodiment) + if drop_metadata_key: + info.pop(drop_metadata_key) + info.update(metadata_override or {}) + if unknown_hand_joint: + info["hand_joints"] = info["hand_joints"].replace("thumb_cmc", "thumb_base") + + def keep(topic: str) -> bool: + return topic != drop_topic + + n_frames = DURATION_S * video_fps + packets = h264_packets(n_frames, b_frames) + if skip_first_keyframe: + packets = packets[1:] + video_times = [ + T0 + round((i + skip_first_keyframe) * 1e9 / video_fps) + for i in range(len(packets)) + ] + if repeat_frame_time: + video_times[21] = video_times[20] + + if human: + cameras = {"/main-camera": "main-camera"} + links = [("head", "main-camera"), ("head", "imu"), ("head", "et")] + if unlink_main_camera: + links.remove(("head", "main-camera")) + else: + cameras = {"/top-camera": "top-camera"} + links = [("world", "top-camera")] + for s in sides: + cameras[f"/{s}-wrist-camera"] = f"{s}-wrist-camera" + links += [("world", f"{s}_arm_base"), (f"{s}_ee", f"{s}-wrist-camera")] + + with open(out / spec.EPISODE_FILE, "wb") as f, Writer(f) as w: + + def write(topic: str, msg, t: int) -> None: + if tamper and (changed := tamper(topic, msg, t)): + msg, t = changed + if keep(topic): + w.write_message(topic, msg, log_time=t, publish_time=t) + + # mcap_protobuf's Writer has no metadata call; its underlying mcap writer does. + w._writer.add_metadata(spec.METADATA_RECORD, info) + + for topic, frame_id in cameras.items(): + width = calibration_width if topic == "/main-camera" else WIDTH + write( + topic + spec.CALIBRATION_SUFFIX, + CameraCalibration( + frame_id=frame_id, + width=width, + height=HEIGHT, + K=[50, 0, principal_x, 0, 50, HEIGHT / 2, 0, 0, 1], + distortion_model=distortion_model, + D=list(distortion_coeffs), + ), + T0, + ) + if calibration_twice: + write( + topic + spec.CALIBRATION_SUFFIX, + CameraCalibration(frame_id=frame_id, width=width, height=HEIGHT), + T0 + 1, + ) + write( + topic + spec.CAMERA_EXTRA_SUFFIX, + schemas.CameraInfoExtra( + frame_id=frame_id, + valid_radius=0.0, + upright_rotation_cw_deg=45 if bad_upright_rotation else 0, + ), + T0, + ) + pairs = list(zip(packets, video_times)) + if multi_frame_video and topic in ("/main-camera", "/top-camera"): + pairs = [ + (b"".join(d for d, _ in pairs[i : i + 2]), pairs[i][1]) + for i in range(0, len(pairs), 2) + ] + for data, t in pairs: + write( + topic, + CompressedVideo(frame_id=frame_id, data=data, format=video_format), + t, + ) + + write("/tf_static", _tf(links, bad_tf_rotation), T0) + + if human: + _write_human( + write, + sides, + backwards_head_pose=backwards_head_pose, + hand_gap_s=hand_gap_s, + device_clock_imu_info=device_clock_imu_info, + imu_hz=imu_hz, + gravity_removed=gravity_removed, + nan_head_pose=nan_head_pose, + zeroed_left_hand=zeroed_left_hand, + bad_hand_confidence=bad_hand_confidence, + hands_wrist_only=hands_wrist_only, + bad_wrist_pose=bad_wrist_pose, + nan_eye_gaze=nan_eye_gaze, + ) + else: + _write_robot( + write, + sides, + empty_arm_pose=empty_arm_pose, + gripper_out_of_range=gripper_out_of_range, + bad_arm_pose=bad_arm_pose, + joint_names_mismatch=joint_names_mismatch, + ) + if labels_in == "episode": + _write_labels( + write, + task_past_end, + label_outside_tasks, + label_crosses_tasks, + labels_without_tasks, + label_past_video, + label_at_last_frame, + bad_annotation_type, + ) + + if labels_in == "annotation": + with open(out / spec.ANNOTATION_FILE, "wb") as f, Writer(f) as w: + + def write_label(topic: str, msg, t: int) -> None: + if tamper and (changed := tamper(topic, msg, t)): + msg, t = changed + w.write_message(topic, msg, log_time=t, publish_time=t) + + _write_labels( + write_label, + task_past_end, + label_outside_tasks, + label_crosses_tasks, + labels_without_tasks, + label_past_video, + label_at_last_frame, + bad_annotation_type, + ) + if annotation_metadata: + w._writer.add_metadata(spec.METADATA_RECORD, {"lab": "other"}) + if stray_annotation_topic: + write_label("/imu", schemas.Imu(frame_id="imu"), T0) + return out + + +def _write_labels( + write, + task_past_end: bool, + label_outside_tasks: bool, + label_crosses_tasks: bool, + labels_without_tasks: bool, + label_past_video: bool, + label_at_last_frame: bool, + bad_annotation_type: bool, +) -> None: + """Two back-to-back tasks, one ranged label inside the first and a point label + inside the second.""" + half = DURATION_S * 10**9 // 2 + end = T0 + 2 * half + (3 * 10**9 if task_past_end else 0) + tasks = [ + (T0, T0 + half, "fold_clothes", ["shirt"]), + (T0 + half, end, "pack_clothes", ["shirt", "basket"]), + ] + for start, stop, name, objects in tasks[ + : 0 if labels_without_tasks else 1 if label_outside_tasks else 2 + ]: + write( + "/task", + schemas.Task( + name=name, + description=name.replace("_", " "), + objects=objects, + end_time_unix_ns=stop, + ), + start, + ) + kind = "summary" if bad_annotation_type else "language" + first_end = T0 + half + half // 2 if label_crosses_tasks else T0 + half // 2 + write( + "/subtask-annotation", + schemas.Annotation( + data="pick up the shirt", + end_time_unix_ns=first_end, + annotation_type=kind, + annotation_source="acme", + ), + T0 + half // 5, + ) + write( + "/subtask-annotation", + schemas.Annotation(data="drop it in the basket", annotation_type=kind), + T0 + 2 * half - 10**6 + if label_past_video + else T0 + round((DURATION_S * VIDEO_FPS - 1) * 1e9 / VIDEO_FPS) + if label_at_last_frame + else T0 + half + half // 3, + ) + + +def _write_human( + write, + sides, + *, + backwards_head_pose, + hand_gap_s, + device_clock_imu_info, + imu_hz, + gravity_removed, + nan_head_pose, + zeroed_left_hand, + bad_hand_confidence, + hands_wrist_only, + bad_wrist_pose, + nan_eye_gaze, +) -> None: + write( + "/imu-info", + schemas.ImuInfo( + frame_id="imu", + accel_noise_density=1e-3, + accel_random_walk=1e-4, + gyro_noise_density=1e-4, + gyro_random_walk=1e-5, + ), + 10**9 if device_clock_imu_info else T0, + ) + for i, t in enumerate(_times(imu_hz)): + m = schemas.Imu(frame_id="imu") + m.linear_acceleration.z = 0.02 if gravity_removed else spec.GRAVITY + m.angular_velocity.x = 0.01 * np.sin(i / 20) + write("/imu", m, t) + + pose_times = _times(POSE_HZ) + if backwards_head_pose: + pose_times[10], pose_times[11] = pose_times[11], pose_times[10] + for i, t in enumerate(pose_times): + p = PoseInFrame(frame_id=spec.WORLD_FRAME) + p.pose.position.z = 1.5 + p.pose.orientation.w = 1.0 + if nan_head_pose and i == 5: + p.pose.position.x = float("nan") + write("/head-pose", p, t) + + for i, t in enumerate(_times(POSE_HZ)): + g = schemas.EyeGaze(frame_id="et", yaw=0.1, pitch=-0.3, depth=0.8, valid=True) + if nan_eye_gaze and i == 5: + g.yaw = float("nan") + write("/eye-gaze", g, t) + + ok = schemas.TrackingQuality.OK + hand = np.linspace(0.1, 0.5, 63).tolist() + for t in _times(POSE_HZ): + write("/head-pose-quality", schemas.TrackingQuality(state=ok), t) + for s in sides: + in_gap = s == "left" and T0 + 0.5e9 < t < T0 + (0.5 + hand_gap_s) * 1e9 + if in_gap: + continue + zero = zeroed_left_hand and s == "left" + k = schemas.HandKeypoints( + frame_id=spec.WORLD_FRAME, + positions=[] if hands_wrist_only else [0.0] * 63 if zero else hand, + confidence=1.5 + if bad_hand_confidence and s == "left" and t == T0 + else 0.9, + joint_confidence=[] if hands_wrist_only else [0.9] * 21, + valid=True, + ) + k.wrist_pose.position.z = 1.2 + k.wrist_pose.orientation.w = ( + 2.0 if bad_wrist_pose and s == "left" and t == T0 else 1.0 + ) + write(f"/{s}-hand-keypoints", k, t) + + +def _write_robot( + write, + sides, + *, + empty_arm_pose, + gripper_out_of_range, + bad_arm_pose, + joint_names_mismatch, +): + joints = [0.1, -0.2, 0.3, 0.0, 0.5, -0.1] + names = [f"joint{j}" for j in range(1, 7)] + for i, t in enumerate(_times(POSE_HZ)): + for s in sides: + pose = _rigid(0.2 + 0.001 * i) + if bad_arm_pose and s == sides[-1] and i == 5: + pose[0] = 2.0 # R[0, 0]: no longer a rotation + last = s == sides[-1] + write( + f"/{s}-arm-state", + schemas.ArmState( + frame_id=f"{s}_arm_base", + joint_names=names[:5] if joint_names_mismatch and last else names, + position=joints, + velocity=[0.0] * 6, + torque=[0.5] * 6, + ee_pose=[] if empty_arm_pose and last else pose, + ), + t, + ) + write( + f"/{s}-arm-action", + schemas.ArmAction( + frame_id=f"{s}_arm_base", + joint_names=names, + position=joints, + ee_pose=_rigid(0.2 + 0.001 * (i + 1)), + ), + t, + ) + g = 1.5 if gripper_out_of_range and last and i == 5 else 0.4 + write(f"/{s}-ee-state", schemas.GripperState(position=g), t) + write(f"/{s}-ee-action", schemas.GripperState(position=0.4), t) diff --git a/tests/fixtures/synthetic_episodes.py b/tests/fixtures/synthetic_episodes.py index b4317a057f..714e1c7cda 100644 --- a/tests/fixtures/synthetic_episodes.py +++ b/tests/fixtures/synthetic_episodes.py @@ -108,6 +108,11 @@ def write_episode( embodiment=v.embodiment, fps=30, task_name="synthetic", + operator=f"op_{seed:02d}", + lab="synthetic", + scene="test_scene", + objects=["red_cube", "bin"], + source_recording_id=f"{vendor}_session_{seed:02d}", annotations=[ ("pick up the red cube", 0, T // 2), ("place it in the bin", T // 2, T - 1), diff --git a/tests/integration/test_mcap_abc.py b/tests/integration/test_mcap_abc.py new file mode 100644 index 0000000000..2fa1766e44 --- /dev/null +++ b/tests/integration/test_mcap_abc.py @@ -0,0 +1,46 @@ +"""Vendor MCAP validator on real episodes. + + EGOVERSE_TEST_ABC_DIR= pytest --integration tests/integration/test_mcap_abc.py + EGOVERSE_TEST_MCAP_DIR= pytest --integration tests/integration/test_mcap_abc.py + +ABC-130k predates the spec (own robot types, no /tf_static or full episode info), so its +episodes must be read end to end and reported, not pass. Episodes delivered to the spec +(e.g. Aria recordings converted to it) must pass. +""" + +import os +from pathlib import Path + +import pytest + +from egomimic.rldb.mcap import spec +from egomimic.rldb.mcap.validate import validate_episode + + +def _episodes(var: str) -> list[Path]: + root = os.environ.get(var) + if not root: + pytest.skip(f"set {var} to a directory of episode_*/ dirs") + eps = sorted(p for p in Path(root).glob("episode_*") if p.is_dir()) + if not eps: + pytest.skip(f"no episode_*/ dirs under {root}") + return eps + + +def test_abc_episodes_are_read_and_reported(): + for ep in _episodes("EGOVERSE_TEST_ABC_DIR"): + r = validate_episode(ep) + unreadable = [p for p in r.problems if p.message.startswith("cannot read")] + assert not unreadable, f"{ep}: {unreadable}" + assert r.problems, f"{ep}: ABC predates the spec, expected problems" + print(f"{ep.name}: {len(r.problems)} problems") + for p in r.problems: + print(f" {p}") + + +def test_spec_deliveries_pass(): + for ep in _episodes("EGOVERSE_TEST_MCAP_DIR"): + r = validate_episode(ep) + assert r.problems == [], f"{ep}:\n" + "\n".join(map(str, r.problems)) + assert r.recording_id, ep + assert (ep / spec.EPISODE_FILE).exists() diff --git a/tests/unit/test_convert_template.py b/tests/unit/test_convert_template.py new file mode 100644 index 0000000000..c7d106a174 --- /dev/null +++ b/tests/unit/test_convert_template.py @@ -0,0 +1,152 @@ +"""The reference converter vendors copy must keep producing valid episodes.""" + +import numpy as np +import pytest +import zarr + +from egomimic.scripts.convert_template import template_to_zarr as tmpl + + +@pytest.fixture(autouse=True) +def _demo_lab(monkeypatch): + """The shipped lab="template" is rejected on purpose; tests stand in a lab.""" + shipped = tmpl.load_episode + + def load(raw_path): + ep = shipped(raw_path) + ep.metadata["lab"] = "demo_lab" + return ep + + monkeypatch.setattr(tmpl, "load_episode", load) + return shipped + + +def test_shipped_template_metadata_is_rejected(tmp_path, monkeypatch, _demo_lab): + monkeypatch.setattr(tmpl, "load_episode", _demo_lab) + raw = _raw_paths(tmp_path, 1) + written, failed = tmpl.convert_all(raw, tmp_path / "out", save_mp4=False) + assert not written + assert "lab='template' is a placeholder" in failed[raw[0]] + + +def _raw_paths(tmp_path, n): + paths = [tmp_path / "raw" / f"session_{i}" for i in range(n)] + for p in paths: + p.mkdir(parents=True) + return paths + + +def test_template_converts_and_validates(tmp_path): + raw = _raw_paths(tmp_path, 3) + assert tmpl.find_raw_episodes(tmp_path / "raw") == raw + written, failed = tmpl.convert_all(raw, tmp_path / "out", save_mp4=False) + assert not failed + assert len(written) == 3 + for path in written: + # named by the §4 episode hash, e.g. 2026-01-01-00-00-12-345000.zarr + assert len(path.stem.split("-")) == 7 + attrs = zarr.open_group(str(path), mode="r").attrs + assert attrs["embodiment"] == "human_bimanual" + assert attrs["lab"] == "demo_lab" + + +def test_template_cli_and_workers(tmp_path): + _raw_paths(tmp_path, 2) + argv = ["--raw-root", str(tmp_path / "raw"), "--output-dir", str(tmp_path / "out")] + assert tmpl.main([*argv, "--no-mp4", "--workers", "2"]) == 0 + assert len(list((tmp_path / "out").glob("*.zarr"))) == 2 + + +def test_bad_episode_is_reported_and_removed(tmp_path, monkeypatch): + good_load = tmpl.load_episode + + def bad_load(raw_path): + ep = good_load(raw_path) + if raw_path.name == "session_1": + ep.head_pose = ep.head_pose * np.array( + [1, 1, 1, 2, 2, 2, 2] + ) # non-unit quats + return ep + + monkeypatch.setattr(tmpl, "load_episode", bad_load) + raw = _raw_paths(tmp_path, 2) + written, failed = tmpl.convert_all(raw, tmp_path / "out", save_mp4=False) + assert len(written) == 1 + assert list(failed) == [raw[1]] + assert "quaternion" in failed[raw[1]] + assert sorted(p.name for p in (tmp_path / "out").iterdir()) == [written[0].name] + + +def test_waived_field_needs_allow_missing(tmp_path, monkeypatch): + good_load = tmpl.load_episode + + def waived_load(raw_path): + ep = good_load(raw_path) + ep.metadata["scene"] = tmpl.MissingMetadata() + return ep + + monkeypatch.setattr(tmpl, "load_episode", waived_load) + raw = _raw_paths(tmp_path, 1) + written, failed = tmpl.convert_all(raw, tmp_path / "strict", save_mp4=False) + assert not written + assert "--allow-missing" in failed[raw[0]] + assert not list((tmp_path / "strict").iterdir()) + + written, failed = tmpl.convert_all( + raw, tmp_path / "waived", save_mp4=False, allow_missing=True + ) + assert not failed + assert zarr.open_group(str(written[0]), mode="r").attrs["scene"] is None + + +def test_operator_cannot_be_waived(tmp_path, monkeypatch): + good_load = tmpl.load_episode + + def waived_load(raw_path): + ep = good_load(raw_path) + ep.metadata["operator"] = tmpl.MissingMetadata() + return ep + + monkeypatch.setattr(tmpl, "load_episode", waived_load) + raw = _raw_paths(tmp_path, 1) + written, failed = tmpl.convert_all( + raw, tmp_path / "out", save_mp4=False, allow_missing=True + ) + assert not written + assert "cannot be waived" in failed[raw[0]] + + +def test_whole_episode_annotation_covers_every_frame(tmp_path): + written, _ = tmpl.convert_all( + _raw_paths(tmp_path, 1), tmp_path / "out", save_mp4=False + ) + attrs = zarr.open_group(str(written[0]), mode="r").attrs + ep = tmpl.load_episode(tmp_path / "raw" / "session_0") + assert ep.annotations[0][1:] == (0, attrs["total_frames"]) + + +def test_failed_preview_keeps_the_episode(tmp_path, monkeypatch): + def no_ffmpeg(*args, **kwargs): + raise RuntimeError("ffmpeg not found") + + monkeypatch.setattr(tmpl, "save_preview_mp4", no_ffmpeg) + raw = _raw_paths(tmp_path, 1) + written, failed = tmpl.convert_all(raw, tmp_path / "out") + assert not failed + assert [p.name for p in (tmp_path / "out").iterdir()] == [written[0].name] + + +@pytest.mark.parametrize("scale", [1e-3, 1e6]) # seconds, nanoseconds +def test_start_time_in_wrong_unit_is_rejected(tmp_path, monkeypatch, scale): + good_load = tmpl.load_episode + + def load(raw_path): + ep = good_load(raw_path) + ep.start_time_ms = int(ep.start_time_ms * scale) + return ep + + monkeypatch.setattr(tmpl, "load_episode", load) + raw = _raw_paths(tmp_path, 1) + written, failed = tmpl.convert_all(raw, tmp_path / "out", save_mp4=False) + assert not written + assert "epoch milliseconds" in failed[raw[0]] diff --git a/tests/unit/test_mcap_validate.py b/tests/unit/test_mcap_validate.py new file mode 100644 index 0000000000..577027d712 --- /dev/null +++ b/tests/unit/test_mcap_validate.py @@ -0,0 +1,543 @@ +"""Vendor MCAP validator (egomimic/rldb/mcap) on tiny synthetic episodes. + +Valid episodes of every profile pass; each breakage in fixtures.mcap_episodes produces +exactly its expected problems and nothing else. +""" + +from pathlib import Path + +import pytest +from fixtures.mcap_episodes import T0, write_mcap_episode + +from egomimic.rldb.mcap import schemas, spec +from egomimic.rldb.mcap.validate import main, validate_episode + +PROTO = Path(schemas.__file__).parent / "proto" / "egoverse.proto" + + +def problems(tmp_path, embodiment="human_bimanual", **kw) -> list[tuple]: + r = validate_episode(write_mcap_episode(tmp_path, embodiment, **kw)) + return [(p.group, p.topic, p.message) for p in r.problems] + + +@pytest.mark.parametrize( + "embodiment, kw", + [ + ("human_bimanual", {}), + ("human_right_arm", {}), + ("human_left_arm", {}), + ("eva_bimanual", {}), + ("eva_left_arm", {}), + ("human_bimanual", {"labels_in": "episode"}), + ("eva_bimanual", {"labels_in": None}), + ("human_bimanual", {"hands_wrist_only": True}), + ("human_bimanual", {"drop_topic": "/main-camera-info-extra"}), + ("human_bimanual", {"labels_without_tasks": True}), + ("human_bimanual", {"labels_without_tasks": True, "label_at_last_frame": True}), + ( + "human_bimanual", + {"distortion_model": "fisheye624", "distortion_coeffs": (0.1,) * 12}, + ), + ], +) +def test_valid_episode_passes(tmp_path, embodiment, kw): + r = validate_episode(write_mcap_episode(tmp_path, embodiment, **kw)) + assert r.problems == [] + assert r.embodiment == embodiment + assert r.recording_id == spec.recording_id("rl2", "acme-rec-0001") + + +# (embodiment, breakage, [(group, topic, substring of the message), ...]) +BREAKAGES = [ + # 1. everything there + ("human_bimanual", {"drop_topic": "/imu-info"}, [("present", "/imu-info", "missing required topic")]), + ("human_bimanual", {"drop_metadata_key": "operator"}, [("present", None, "Missing metadata field: operator")]), + ("human_bimanual", {"metadata_override": {"scene": "unknown"}}, [("present", None, "scene='unknown' is a placeholder")]), + ("human_bimanual", {"metadata_override": {"objects": "shirt,,basket"}}, [("present", None, "objects entries must be real names")]), + ("human_bimanual", {"metadata_override": {"objects": " shirt, basket"}}, [("present", None, "objects=' shirt, basket' has leading/trailing whitespace")]), + ("human_bimanual", {"metadata_override": {"missing_metadata": "objects"}}, [("present", None, "objects has both a value")]), + ("human_bimanual", {"metadata_override": {"missing_metadata": "scen"}}, [("present", None, "names unknown fields ['scen']")]), + ("human_bimanual", {"drop_metadata_key": "license"}, [("present", None, "'license' missing")]), + ("human_bimanual", {"drop_metadata_key": "source_recording_id"}, [("present", None, "'source_recording_id' missing")]), + ("human_bimanual", {"drop_metadata_key": "hand_joints"}, [("present", None, "'hand_joints' missing")]), + ("human_bimanual", {"metadata_override": {"missing_metadata": "lab"}}, [("present", None, "['lab'] cannot be waived"), ("present", None, "lab has both a value")]), + ("human_bimanual", {"metadata_override": {"missing_metadata": "operator"}}, [("present", None, "['operator'] cannot be waived"), ("present", None, "operator has both a value")]), + ("human_bimanual", {"metadata_override": {"missing_metadata": "task_name"}}, [("present", None, "['task_name'] cannot be waived"), ("present", None, "task_name has both a value")]), + ("human_bimanual", {"metadata_override": {"site": "rl2"}}, [("present", None, "unknown episode info key 'site'")]), + ("human_bimanual", {"metadata_override": {"rig-name": "test_rig"}}, [("present", None, "unknown episode info key 'rig-name'; did you mean 'rig_name'?")]), + ("human_bimanual", {"metadata_override": {"license": "tbd"}}, [("present", None, "'license' = 'tbd' is a placeholder")]), + ("human_bimanual", {"metadata_override": {"lab": "RL2 Lab"}}, [("present", None, "lab='RL2 Lab' must be a short, stable, lowercase identifier")]), + ("human_bimanual", {"metadata_override": {"shutter": "rolling"}}, [("present", None, "expected 'global' or 'rolling:ms'")]), + ("human_bimanual", {"metadata_override": {"camera_timestamp_ref": "middle"}}, [("present", None, "'camera_timestamp_ref' = 'middle'")]), + ("human_bimanual", {"annotation_metadata": True}, [("present", None, "annotation.mcap carries metadata records")]), + ("human_bimanual", {"metadata_override": {"start_time_unix": "--1760000000000"}}, [("present", None, "is not an integer")]), + ("human_bimanual", {"metadata_override": {"end_time_unix": "1760000002000²"}}, [("present", None, "is not an integer")]), + ("human_bimanual", {"unknown_hand_joint": True}, [("present", None, "unknown joints ['thumb_base']")]), + ("human_bimanual", {"declared_embodiment": "robot_dog"}, [("present", None, "'embodiment' = 'robot_dog'")]), + ("human_bimanual", {"declared_embodiment": "human_right_arm"}, [("present", "/left-hand-keypoints", "does not fit embodiment")]), + ("human_bimanual", {"stray_annotation_topic": True}, [("present", "/imu", "belong in annotation.mcap")]), + ("eva_bimanual", {"empty_arm_pose": True}, [ + ("present", "/right-arm-state", "'ee_pose' empty on 61/61"), + ("poses", "/right-arm-state", "not a finite rigid 4x4"), + ]), + # 2. video + ("human_bimanual", {"video_format": "vp9"}, [("video", "/main-camera", "format 'vp9'")]), + ("human_bimanual", {"skip_first_keyframe": True}, [("video", "/main-camera", "not a keyframe")]), + ("human_bimanual", {"b_frames": True}, [("video", "/main-camera", "uses B-frames; re-encode without them (x264: -bf 0)")]), + ("eva_bimanual", {"b_frames": True}, [("video", c, "uses B-frames") for c in ("/left-wrist-camera", "/right-wrist-camera", "/top-camera")]), + ("human_bimanual", {"calibration_width": 128}, [("video", "/main-camera", "calibration says 128x48")]), + ("human_bimanual", {"video_fps": 15}, [("video", "/main-camera", "15.0 fps")]), + ("human_bimanual", {"multi_frame_video": True}, [("video", "/main-camera", "exactly one frame")]), + ("eva_bimanual", {"multi_frame_video": True}, [("video", "/top-camera", "exactly one frame")]), + # 3. clocks + ("human_bimanual", {"backwards_head_pose": True}, [("clocks", "/head-pose", "backwards 1 times")]), + ("human_bimanual", {"hand_gap_s": 0.8}, [("clocks", "/left-hand-keypoints", "0.80 s gap")]), + ("human_bimanual", {"device_clock_imu_info": True}, [("clocks", "/imu-info", "before 2015")]), + ("human_bimanual", {"repeat_frame_time": True}, [("clocks", "/main-camera", "1 frames repeat a timestamp")]), + # 4. calibration + ("human_bimanual", {"drop_topic": "/main-camera-info"}, [("calibration", "/main-camera", "no /main-camera-info")]), + ("human_bimanual", {"unlink_main_camera": True}, [("calibration", "/main-camera", "no path to 'world'")]), + ("human_bimanual", {"bad_tf_rotation": True}, [("calibration", "/tf_static", "non-unit rotation")]), + ("human_bimanual", {"bad_upright_rotation": True}, [("calibration", "/main-camera", "upright_rotation_cw_deg 45")]), + ("human_bimanual", {"calibration_twice": True}, [("calibration", "/main-camera-info", "got 2 foxglove.CameraCalibration")]), + ("human_bimanual", {"principal_x": 500.0}, [("calibration", "/main-camera", "principal point (500, 24) is outside")]), + ("human_bimanual", {"distortion_model": "equidistant", "distortion_coeffs": (0.1,) * 4}, [("calibration", "/main-camera", "distortion_model 'equidistant'")]), + ("human_bimanual", {"distortion_model": "fisheye624", "distortion_coeffs": (0.1,) * 5}, [("calibration", "/main-camera", "needs 12 D coefficients, got 5")]), + # 5. imu + ("human_bimanual", {"imu_hz": 100}, [("imu", "/imu", "100 Hz")]), + ("human_bimanual", {"gravity_removed": True}, [("imu", "/imu", "gravity removed")]), + # 6. poses + ("human_bimanual", {"nan_head_pose": True}, [("poses", "/head-pose", "1/61 poses")]), + ("human_bimanual", {"zeroed_left_hand": True}, [("poses", "/left-hand-keypoints", "all-zero with valid=true")]), + ("human_bimanual", {"bad_hand_confidence": True}, [("poses", "/left-hand-keypoints", "confidence outside [0, 1]")]), + ("human_bimanual", {"bad_wrist_pose": True}, [("poses", "/left-hand-keypoints", "wrist poses")]), + ("human_bimanual", {"nan_eye_gaze": True}, [("poses", "/eye-gaze", "NaN/Inf")]), + ("eva_bimanual", {"gripper_out_of_range": True}, [("poses", "/right-ee-state", "outside [0, 1]")]), + ("eva_bimanual", {"bad_arm_pose": True}, [("poses", "/right-arm-state", "1/61 poses")]), + ("eva_bimanual", {"joint_names_mismatch": True}, [("poses", "/right-arm-state", "do not match joint_names")]), + # 7. tasks & labels + ("human_bimanual", {"task_past_end": True}, [("labels", "/task", "outside the recorded data")]), + ("eva_bimanual", {"label_outside_tasks": True}, [("labels", "/subtask-annotation", "inside one /task")]), + ("human_bimanual", {"label_crosses_tasks": True}, [("labels", "/subtask-annotation", "inside one /task")]), + ("human_bimanual", {"bad_annotation_type": True}, [("labels", "/subtask-annotation", "['summary']")]), + ("human_bimanual", {"labels_without_tasks": True, "label_past_video": True}, [("labels", "/subtask-annotation", "outside /main-camera's frames")]), +] # fmt: skip + + +@pytest.mark.parametrize( + "embodiment, kw, expected", + BREAKAGES, + ids=[f"{e}-{next(iter(kw))}={next(iter(kw.values()))}" for e, kw, _ in BREAKAGES], +) +def test_breakage_reports_exactly_its_problem(tmp_path, embodiment, kw, expected): + found = problems(tmp_path, embodiment, **kw) + assert len(found) == len(expected), found + for (group, topic, message), (g, t, sub) in zip(sorted(found), sorted(expected)): + assert (group, topic) == (g, t), found + assert sub in message, found + + +def test_cli_pass_fail_and_summary(tmp_path, capsys): + good = write_mcap_episode(tmp_path, episode_id="good") + bad = write_mcap_episode(tmp_path, episode_id="bad", imu_hz=100) + assert main([str(good)]) == 0 + assert main([str(good), str(bad)]) == 1 + out = capsys.readouterr().out + assert f"PASS {good}" in out + assert f"FAIL {bad}" in out and "[imu] /imu: 100 Hz" in out + assert "1/2 passed" in out + assert spec.recording_id("rl2", "acme-rec-0001") in out + + +def test_waived_metadata_needs_allow_missing(tmp_path, capsys): + ep = write_mcap_episode( + tmp_path, + drop_metadata_key="scene", + metadata_override={"missing_metadata": "scene"}, + ) + strict = validate_episode(ep) + assert [p.message for p in strict.problems] == [ + "scene: marked as not available (MissingMetadata). We heavily recommend " + "providing it; pass --allow-missing to accept the episode anyway." + ] + lenient = validate_episode(ep, allow_missing=True) + assert lenient.ok and lenient.warnings == [ + "scene: marked as not available (MissingMetadata)" + ] + assert main([str(ep)]) == 1 + assert main([str(ep), "--allow-missing"]) == 0 + assert "warning: scene: marked as not available" in capsys.readouterr().out + + +@pytest.mark.parametrize("name", ["lab", "task_name", "operator"]) +def test_unwaivable_metadata_fails_even_with_allow_missing(tmp_path, name): + ep = write_mcap_episode( + tmp_path, drop_metadata_key=name, metadata_override={"missing_metadata": name} + ) + r = validate_episode(ep, allow_missing=True) + assert not r.ok + assert any(f"['{name}'] cannot be waived" in p.message for p in r.problems) + + +def test_missing_and_unreadable_episodes(tmp_path): + r = validate_episode(tmp_path / "episode_nope") + assert [p.message for p in r.problems] == [ + f"no episode.mcap in {tmp_path / 'episode_nope'}" + ] + junk = tmp_path / "episode_junk" + junk.mkdir() + (junk / "episode.mcap").write_bytes(b"not an mcap") + (p,) = validate_episode(junk).problems + assert p.message.startswith("cannot read episode.mcap") + + +def test_unreadable_annotation_file(tmp_path): + ep = write_mcap_episode(tmp_path, labels_in=None) + (ep / spec.ANNOTATION_FILE).write_bytes(b"garbage") + (p,) = validate_episode(ep).problems + assert p.message.startswith("cannot read annotation.mcap") + + +def test_proto_file_matches_schemas(): + assert PROTO.read_text() == schemas.render_proto(), ( + "regenerate: python -m egomimic.rldb.mcap.schemas > " + str(PROTO) + ) + + +def test_proto_field_numbers_are_unique(): + for m in schemas.MESSAGES: + numbers = [f.number for f in m.fields] + assert len(set(numbers)) == len(numbers), f"{m.name} reuses a field number" + assert all(n >= 1 for n in numbers), m.name + + +def test_only_scene_and_objects_can_be_waived(): + from egomimic.rldb.zarr.required_metadata import REQUIRED_METADATA_FIELDS + + assert set(spec.UNWAIVABLE_FIELDS) == {"task_name", "operator", "lab"} + waivable = set(REQUIRED_METADATA_FIELDS) - set(spec.UNWAIVABLE_FIELDS) + assert waivable == {"scene", "objects"} + + +def test_embodiments_match_enum(): + from egomimic.rldb.embodiment.embodiment import EMBODIMENT + from egomimic.rldb.zarr import episode_rules + + assert set(episode_rules.EMBODIMENTS) == {e.name.lower() for e in EMBODIMENT} + + +def test_mano_joints_match_human_finger_edges(): + # Canonical MANO: wrist, then 4 joints per finger, thumb to pinky. + assert len(spec.MANO_JOINTS) == 21 and spec.MANO_JOINTS[0] == "wrist" + assert spec.MANO_JOINTS[1:5] == ("thumb_cmc", "thumb_mcp", "thumb_ip", "thumb_tip") + assert spec.MANO_JOINTS[17:] == ("pinky_mcp", "pinky_pip", "pinky_dip", "pinky_tip") + + +def test_lance_ids_and_annotation_types_match_lance_schema(): + lance_schema = pytest.importorskip("egomimic.rldb.lance.schema") + assert spec.recording_id("aria", "x") == lance_schema.recording_id("aria", "x") + if hasattr(lance_schema, "ANNOTATION_TYPES"): + assert spec.ANNOTATION_TYPES == lance_schema.ANNOTATION_TYPES + + +def _late_hand_frame(topic, msg, t): + if topic == "/left-hand-keypoints" and t > T0 + 10**9: + msg.frame_id = "tracker" # message 0 is fine; later ones are not linked + return msg, t + + +def _head_in_odom(topic, msg, t): + if topic == "/head-pose": + msg.frame_id = "odom" + return msg, t + if ( + topic == "/tf_static" + ): # odom does reach world, but /head-pose must be T_world_head + x = msg.transforms.add(parent_frame_id="world", child_frame_id="odom") + x.rotation.w = 1.0 + return msg, t + + +def _nan_static_translation(topic, msg, t): + if topic == "/tf_static": + msg.transforms[0].translation.x = float("nan") + return msg, t + + +def _main_camera_fixed_in_world(topic, msg, t): + """A consistent camera frame, rigidly mounted to the world instead of the head.""" + if topic.startswith("/main-camera"): + msg.frame_id = "room_cam" + return msg, t + if topic == "/tf_static": + x = msg.transforms.add(parent_frame_id="world", child_frame_id="room_cam") + x.rotation.w = 1.0 + return msg, t + + +def _point_label_at_task_end(topic, msg, t): + if topic == "/subtask-annotation" and not msg.end_time_unix_ns: + return msg, T0 + 10**9 - 10_000_000 # 10 ms before task 1 ends: not a frame + + +@pytest.mark.parametrize( + "tamper, expected", + [ + ( + _late_hand_frame, + ("calibration", "/left-hand-keypoints", "'tracker' has no path"), + ), + (_head_in_odom, ("calibration", "/head-pose", "must be in frame 'world'")), + ( + _nan_static_translation, + ("calibration", "/tf_static", "1 transforms have a NaN / Inf translation"), + ), + ( + _main_camera_fixed_in_world, + ("calibration", "/main-camera", "no fixed path to 'head'"), + ), + ( + _point_label_at_task_end, + ("labels", "/subtask-annotation", "inside one /task"), + ), + ], +) +def test_tampered_episode_reports_exactly_its_problem(tmp_path, tamper, expected): + found = problems(tmp_path, tamper=tamper) + assert len(found) == 1, found + group, topic, message = found[0] + assert (group, topic) == expected[:2] and expected[2] in message, found + + +def test_calibration_naming_another_frame_than_the_video_is_caught(tmp_path): + """The calibration (and -info-extra) moved to `world`, the video frames not: the + camera has two frames, and the steps camera is no longer on the head.""" + + def calib_in_world(topic, msg, t): + if topic in ("/main-camera-info", "/main-camera-info-extra"): + msg.frame_id = "world" + return msg, t + + found = problems(tmp_path, tamper=calib_in_world) + assert ("calibration", "/main-camera") == found[0][:2] + assert any("video frames name ['main-camera']" in m for *_, m in found), found + assert any("no fixed path to 'head'" in m for *_, m in found), found + + +def test_wrong_message_type_is_reported_not_raised(tmp_path): + from egomimic.rldb.mcap import schemas + + def imu_as_task(topic, msg, t): + if topic == "/imu": + return schemas.Task(name="not an imu"), t + + found = problems(tmp_path, tamper=imu_as_task) + assert ( + "present", + "/imu", + "type is 'egoverse.Task', expected 'egoverse.Imu'", + ) in found + assert any(g == "imu" and "could not check" in m for g, _, m in found), found + + +def _rewrite(path: Path, extra=None, **writer_options) -> None: + """Rewrite an MCAP record by record, uncompressed (so a chunk's CRC is its only + guard). extra(writer, messages, channel_ids, schema_ids) adds records at the end; + writer_options go to mcap's Writer.""" + from mcap.reader import make_reader + from mcap.writer import CompressionType, Writer + + with open(path, "rb") as f: + reader = make_reader(f) + records = [(r.name, dict(r.metadata)) for r in reader.iter_metadata()] + messages = list(reader.iter_messages(log_time_order=False)) + schema_ids, channel_ids = {}, {} + with open(path, "wb") as f: + w = Writer(f, compression=CompressionType.NONE, **writer_options) + w.start() + for name, data in records: + w.add_metadata(name, data) + for schema, channel, message in messages: + if schema.id not in schema_ids: + schema_ids[schema.id] = w.register_schema( + schema.name, schema.encoding, schema.data + ) + if channel.id not in channel_ids: + channel_ids[channel.id] = w.register_channel( + channel.topic, channel.message_encoding, schema_ids[schema.id] + ) + w.add_message( + channel_ids[channel.id], + message.log_time, + message.data, + message.publish_time, + ) + if extra: + extra(w, messages, channel_ids, schema_ids) + w.finish() + + +def test_corrupt_chunk_fails_crc(tmp_path): + from fixtures.mcap_episodes import h264_packets + + ep = write_mcap_episode(tmp_path) + path = ep / spec.EPISODE_FILE + _rewrite(path) + data = bytearray(path.read_bytes()) + i = data.find(h264_packets(60)[5][-16:]) + assert i > 0 + data[i] ^= 0x01 + path.write_bytes(bytes(data)) + (p,) = validate_episode(ep).problems + assert p.message.startswith("cannot read episode.mcap") and "CRC" in p.message + + +def test_corrupt_unchunked_file_fails_crc(tmp_path): + from fixtures.mcap_episodes import h264_packets + + ep = write_mcap_episode(tmp_path) + path = ep / spec.EPISODE_FILE + _rewrite(path, use_chunking=False, enable_data_crcs=True) + data = bytearray(path.read_bytes()) + i = data.find(h264_packets(60)[5][-16:]) + assert i > 0 + data[i] ^= 0x01 + path.write_bytes(bytes(data)) + (p,) = validate_episode(ep).problems + assert p.message.startswith("cannot read episode.mcap"), p.message + assert "crc" in p.message.lower(), p.message + + +def test_file_without_metadata_index_passes(tmp_path): + from mcap.writer import IndexType + + ep = write_mcap_episode(tmp_path) + no_metadata = IndexType.ATTACHMENT | IndexType.CHUNK | IndexType.MESSAGE + _rewrite(ep / spec.EPISODE_FILE, index_types=no_metadata) + assert validate_episode(ep).problems == [] + + +def test_unknown_topics_are_not_decoded_and_series_keep_no_messages(tmp_path): + from egomimic.rldb.mcap.validate import _read + + ep = write_mcap_episode(tmp_path) + + def garbage_topic(w, messages, channel_ids, schema_ids): + imu_schema = next(s.id for s, c, _ in messages if c.topic == "/imu") + ch = w.register_channel("/vendor-debug", "protobuf", schema_ids[imu_schema]) + for i in range(3): + w.add_message(ch, T0 + i, b"\xff not protobuf", T0 + i) + + _rewrite(ep / spec.EPISODE_FILE, extra=garbage_topic) + assert validate_episode(ep).problems == [] + streams = _read(ep / spec.EPISODE_FILE).streams + assert streams["/vendor-debug"].n == 3 and not streams["/vendor-debug"].msgs + for topic in ("/imu", "/head-pose", "/left-hand-keypoints", "/head-pose-quality"): + assert streams[topic].n > 0 and not streams[topic].msgs, topic + assert len(streams["/main-camera"].msgs) == spec.VIDEO_SAMPLE_FRAMES + + +def _after_sample(change): + """Tamper every /main-camera message past the decoded sample.""" + seen = {"n": 0} + + def tamper(topic, msg, t): + if topic == "/main-camera": + seen["n"] += 1 + if seen["n"] > spec.VIDEO_SAMPLE_FRAMES: + change(msg) + return msg, t + + return tamper + + +def _empty_data(msg): + msg.data = b"" + + +def _no_picture(msg): + msg.data = b"\x00\x00\x01\x06garbage" # an SEI NAL only + + +def _other_frame(msg): + msg.frame_id = "other" + + +def _other_format(msg): + msg.format = "h265" + + +@pytest.mark.parametrize( + "change, expected", + [ + (_empty_data, ("present", "/main-camera", "'data' empty on 60/120 messages")), + (_no_picture, ("video", "/main-camera", "60/120 messages hold no h264 picture")), + (_other_frame, ("calibration", "/main-camera", "video frames name ['main-camera', 'other']")), + (_other_format, ("video", "/main-camera", "format changes within the stream")), + ], +) # fmt: skip +def test_video_is_checked_past_the_decoded_sample(tmp_path, change, expected): + found = problems(tmp_path, video_fps=60, tamper=_after_sample(change)) + assert any(f[:2] == expected[:2] and expected[2] in f[2] for f in found), found + + +def test_video_format_is_case_sensitive(tmp_path): + found = problems(tmp_path, video_format="H264") + assert ("video", "/main-camera") in [f[:2] for f in found], found + assert any("format 'H264'" in m for *_, m in found), found + + +def test_duplicate_records_channels_and_once_topics(tmp_path): + ep = write_mcap_episode(tmp_path) + + def duplicates(w, messages, channel_ids, schema_ids): + info = {"lab": "rl2"} + w.add_metadata(spec.METADATA_RECORD, info) + for schema, channel, m in messages: + if channel.topic == "/tf_static": + w.add_message( + channel_ids[channel.id], m.log_time + 1, m.data, m.log_time + ) + if channel.topic == "/imu-info": + second = w.register_channel( + "/imu-info", "protobuf", schema_ids[schema.id] + ) + w.add_message(second, m.log_time, m.data, m.log_time) + + _rewrite(ep / spec.EPISODE_FILE, extra=duplicates) + found = [(p.group, p.topic, p.message) for p in validate_episode(ep).problems] + assert any("2 'session-metadata' records" in m for *_, m in found), found + assert ("present", "/tf_static", "published 2 times, expected once") in found + assert any( + t == "/imu-info" and "published on 2 channels" in m for _, t, m in found + ), found + + +def test_wrong_typed_optional_calibration_does_not_hide_other_faults(tmp_path): + def slam_info_as_task(topic, msg, t): + if topic == "/main-camera-info": + return schemas.Task(name="not a calibration"), t + + found = problems(tmp_path, unlink_main_camera=True, tamper=slam_info_as_task) + assert any( + g == "calibration" + and t == "/main-camera" + and "not foxglove.CameraCalibration" in m + for g, t, m in found + ), found + assert not any("could not check" in m for *_, m in found), found + + +def test_corrupt_slice_inside_the_sample_fails_decoding(tmp_path): + seen = {"n": 0} + + def corrupt_frame_10(topic, msg, t): + if topic == "/main-camera": + seen["n"] += 1 + if seen["n"] == 11: + msg.data = b"\x00\x00\x01\x41\x9a\x00" + b"\x5a" * 200 + return msg, t + + found = problems(tmp_path, tamper=corrupt_frame_10) + assert [f[:2] for f in found] == [("video", "/main-camera")], found + assert "decode error" in found[0][2] diff --git a/tests/unit/test_norm_cache.py b/tests/unit/test_norm_cache.py index 8af26e89b3..f015f9955a 100644 --- a/tests/unit/test_norm_cache.py +++ b/tests/unit/test_norm_cache.py @@ -19,7 +19,7 @@ import egomimic.trainHydra as train_hydra from egomimic.rldb.zarr import norm_cache from egomimic.rldb.zarr.zarr_dataset_multi import MultiDataset -from egomimic.rldb.zarr.zarr_writer import ZarrWriter +from egomimic.rldb.zarr.zarr_writer import write_annotations CFG = OmegaConf.create( { @@ -151,11 +151,10 @@ def test_episode_fingerprint_changes_when_episode_is_rewritten(tmp_path) -> None def test_episode_fingerprint_changes_when_annotations_are_rewritten(tmp_path) -> None: - """Re-annotating in place (scale_to_zarr_annotation.py) and adding a key - (zarr_key_transform.py) touch only a child array, not the root zarr.json.""" + """Re-annotating in place and adding an annotation key, as + scale_to_zarr_annotation.py does through write_annotations.""" write_episode(tmp_path, "aria", seed=0) ep = tmp_path / "aria_00.zarr" - writer = ZarrWriter(episode_path=ep) seen = [norm_cache.episode_fingerprint(ep)] for mode, annotations in [ ("w", [("pick", 0, 5)]), @@ -164,7 +163,7 @@ def test_episode_fingerprint_changes_when_annotations_are_rewritten(tmp_path) -> ]: time.sleep(0.02) # past the filesystem's mtime granularity key = "annotations" if mode == "w" else "annotations_v2" - writer.append_annotations(key, annotations, mode=mode) + write_annotations(ep, annotations, key, mode=mode) seen.append(norm_cache.episode_fingerprint(ep)) assert len(set(seen)) == len(seen) diff --git a/tests/unit/test_packaging.py b/tests/unit/test_packaging.py index dd035fc212..bcca6ae48a 100644 --- a/tests/unit/test_packaging.py +++ b/tests/unit/test_packaging.py @@ -77,3 +77,8 @@ def test_wheel_ships_robot_resources(wheel_files): # XML relative to egomimic.__file__. assert "egomimic/resources/model_x5.xml" in wheel_files assert "egomimic/resources/model_arx.urdf" in wheel_files + + +def test_wheel_ships_mcap_proto(wheel_files): + # The .proto handed to vendors; egomimic.rldb.mcap.schemas renders the same text. + assert "egomimic/rldb/mcap/proto/egoverse.proto" in wheel_files diff --git a/tests/unit/test_run_conversion_metadata.py b/tests/unit/test_run_conversion_metadata.py new file mode 100644 index 0000000000..bf177525dd --- /dev/null +++ b/tests/unit/test_run_conversion_metadata.py @@ -0,0 +1,73 @@ +"""DB row -> required metadata for the internal conversion driver.""" + +from types import SimpleNamespace + +import pytest + +run_conversion = pytest.importorskip("egomimic.scripts.run_conversion") + + +@pytest.mark.parametrize( + "raw, expected", + [ + ('{"pink cup","white saucer"}', ["pink cup", "white saucer"]), + ("{bag,fruits}", ["bag", "fruits"]), + ("{None}", []), + ("{bag,NULL}", ["bag"]), + ("cup, towel", ["cup", "towel"]), + ("", []), + (None, []), + (float("nan"), []), + (["cup", None, "bowl"], ["cup", "bowl"]), + ('{"say \\"hi\\"",cup}', ['say "hi"', "cup"]), + ], +) +def test_parse_db_objects(raw, expected): + assert run_conversion.parse_db_objects(raw) == expected + + +def _row(**overrides): + fields = { + "task": "fold_clothes", + "operator": "op_7", + "lab": "rl2", + "scene": "kitchen_a", + "objects": "{shirt,basket}", + } + return SimpleNamespace(**{**fields, **overrides}) + + +def test_row_metadata_is_accepted(): + row = _row() + meta = run_conversion.infer_required_metadata_from_row(row) + assert meta["objects"] == ["shirt", "basket"] + assert meta["missing_metadata"] == [] + assert run_conversion.required_metadata_row_error("fold_clothes", meta) is None + + +def test_null_objects_become_a_waiver(): + meta = run_conversion.infer_required_metadata_from_row(_row(objects="{None}")) + assert meta["missing_metadata"] == ["objects"] + assert run_conversion.required_metadata_row_error("fold_clothes", meta) is None + + +def test_empty_scene_is_a_waiver(): + meta = run_conversion.infer_required_metadata_from_row(_row(scene=None)) + assert meta["missing_metadata"] == ["scene"] + assert run_conversion.required_metadata_row_error("fold_clothes", meta) is None + + +@pytest.mark.parametrize( + "overrides, task_name, match", + [ + ({"operator": None}, "fold_clothes", "operator is always required"), + ({"lab": ""}, "fold_clothes", "lab is always required"), + ({}, "", "task_name is always required"), + ({"operator": "unknown"}, "fold_clothes", "operator"), + ({"lab": "RL2"}, "fold_clothes", "lab"), + ({}, "debug", "task_name"), + ], +) +def test_bad_row_metadata_is_reported(overrides, task_name, match): + meta = run_conversion.infer_required_metadata_from_row(_row(**overrides)) + assert match in run_conversion.required_metadata_row_error(task_name, meta) diff --git a/tests/unit/test_zarr_required_metadata.py b/tests/unit/test_zarr_required_metadata.py new file mode 100644 index 0000000000..7f83bfdbdb --- /dev/null +++ b/tests/unit/test_zarr_required_metadata.py @@ -0,0 +1,452 @@ +"""Required metadata (operator/lab/scene/objects/task_name) in +ZarrWriter.write and the episode validator.""" + +import argparse + +import numpy as np +import pytest +import zarr +from fixtures.synthetic_episodes import write_episode + +from egomimic.rldb.zarr import validate +from egomimic.rldb.zarr.episode_rules import SCHEMA_VERSION +from egomimic.rldb.zarr.required_metadata import ( + MissingMetadata, + add_required_metadata_args, + required_metadata_from_args, +) +from egomimic.rldb.zarr.zarr_writer import ( + ZarrWriter, + key_exists, + remove_key, + write_annotations, +) + +T = 8 +K = np.array([[100.0, 0, 32, 0], [0, 100.0, 32, 0], [0, 0, 1.0, 0]]) +GOOD = { + "task_name": "fold_clothes", + "operator": "op_7", + "lab": "rl2", + "scene": "kitchen_a", + "objects": ["shirt"], + "source_recording_id": "session_0412", +} + + +_DROP = object() + + +def _write(path, metadata_override=None, **overrides): + kwargs = {**GOOD, **overrides} + kwargs = {k: v for k, v in kwargs.items() if v is not _DROP} + return ZarrWriter( + path, + embodiment="human_bimanual", + intrinsics={"front_1": K}, + **kwargs, + ).write( + numeric_data={"actions": np.zeros((T, 3), np.float32)}, + metadata_override=metadata_override, + ) + + +def _attrs(path): + return dict(zarr.open_group(str(path), mode="r").attrs) + + +@pytest.mark.parametrize("field", list(GOOD)) +def test_omitting_a_field_is_a_type_error(tmp_path, field): + with pytest.raises(TypeError, match=field): + _write(tmp_path / "ep.zarr", **{field: _DROP}) + assert not (tmp_path / "ep.zarr").exists() + + +def test_create_and_write_wraps_write(tmp_path): + ep = ZarrWriter.create_and_write( + tmp_path / "ep.zarr", + numeric_data={"actions": np.zeros((T, 3), np.float32)}, + embodiment="human_bimanual", + intrinsics={"front_1": K}, + **GOOD, + ) + assert _attrs(ep)["operator"] == GOOD["operator"] + with pytest.raises(ValueError, match="operator"): + ZarrWriter.create_and_write( + tmp_path / "bad.zarr", + numeric_data={"actions": np.zeros((T, 3), np.float32)}, + embodiment="human_bimanual", + intrinsics={"front_1": K}, + **{**GOOD, "operator": "unknown"}, + ) + assert not (tmp_path / "bad.zarr").exists() + + +@pytest.mark.parametrize( + "kwargs, match", + [ + ({"embodiment": "mecka_bimanual"}, "embodiment"), + ({"intrinsics": None}, "intrinsics"), + ({"intrinsics": {"front_1": np.eye(3)}}, "3x4"), + ({"extrinsics": {}}, "extrinsics"), + ({"extrinsics": {"left": np.eye(3)}}, "4x4"), + ({"intrinsics": {"cam0": K}}, "front-camera"), + ({"fps": 25}, "fps"), + ({"fps": np.int64(30)}, "fps"), + ({"fps": 30.0}, "fps"), + ], +) +def test_write_validates_camera_and_embodiment(tmp_path, kwargs, match): + base = {"embodiment": "human_bimanual", "intrinsics": {"front_1": K}, **GOOD} + writer = ZarrWriter(tmp_path / "ep.zarr", **{**base, **kwargs}) + with pytest.raises(ValueError, match=match): + writer.write(numeric_data={"actions": np.zeros((T, 3), np.float32)}) + assert not (tmp_path / "ep.zarr").exists() + + +def test_bare_writer_is_a_type_error(tmp_path): + with pytest.raises(TypeError): + ZarrWriter(tmp_path / "ep.zarr") + + +def test_store_edits_keep_features_in_sync(tmp_path): + ep = write_episode(tmp_path, "aria", T=16, H=32, W=32) + write_annotations(ep, [("pick", 0, 3)], "annotations", mode="w") + write_annotations(ep, [("place", 4, 9)], "annotations_v2") + assert key_exists(ep, "annotations_v2") + assert _attrs(ep)["features"]["annotations_v2"]["shape"] == [1] + assert validate.validate_episode(ep).ok + + with pytest.raises(ValueError, match="already exists"): + write_annotations(ep, [("x", 0, 1)], "annotations_v2", mode="a") + + remove_key(ep, "annotations_v2") + assert not key_exists(ep, "annotations_v2") + assert "annotations_v2" not in _attrs(ep)["features"] + assert validate.validate_episode(ep).ok + + +@pytest.mark.parametrize("field", list(GOOD)) +@pytest.mark.parametrize("bad", [None, "", " ", "unknown", "N/A", "debug", 3]) +def test_placeholders_are_rejected(tmp_path, field, bad): + with pytest.raises(ValueError, match=field): + _write(tmp_path / "ep.zarr", **{field: bad}) + assert not (tmp_path / "ep.zarr").exists() + + +@pytest.mark.parametrize("bad", ["RL2", "rl 2", "_rl2", "rl2\n", "rl2 "]) +def test_lab_must_be_lowercase_identifier(tmp_path, bad): + with pytest.raises(ValueError, match="lab"): + _write(tmp_path / "ep.zarr", lab=bad) + + +def test_objects_list_rejects_empty_and_placeholder_entries(tmp_path): + with pytest.raises(ValueError, match="objects"): + _write(tmp_path / "a.zarr", objects=[]) + with pytest.raises(ValueError, match="objects"): + _write(tmp_path / "b.zarr", objects=["shirt", "unknown"]) + + +def test_values_land_in_attrs(tmp_path): + attrs = _attrs(_write(tmp_path / "ep.zarr")) + for k, v in GOOD.items(): + assert attrs[k] == v + + +def test_missing_metadata_is_stored_as_null(tmp_path): + attrs = _attrs(_write(tmp_path / "ep.zarr", scene=MissingMetadata())) + assert attrs["scene"] is None + assert "missing_metadata" not in attrs + + +@pytest.mark.parametrize("field", ["task_name", "operator", "lab"]) +def test_core_fields_cannot_be_waived(tmp_path, field): + with pytest.raises(ValueError, match="cannot be waived"): + _write(tmp_path / "ep.zarr", **{field: MissingMetadata()}) + assert not (tmp_path / "ep.zarr").exists() + with pytest.raises(ValueError, match="cannot be waived"): + required_metadata_from_args(argparse.Namespace(missing_metadata=[field])) + + +def test_schema_version_is_stamped(tmp_path): + assert _attrs(_write(tmp_path / "ep.zarr"))["schema_version"] == SCHEMA_VERSION + + +@pytest.mark.parametrize("key", ["operator", "intrinsics", "embodiment", "features"]) +def test_metadata_override_rejects_writer_owned_keys(tmp_path, key): + with pytest.raises(ValueError, match=f"must not set \\['{key}'\\]"): + _write(tmp_path / "ep.zarr", metadata_override={key: "x"}) + assert not (tmp_path / "ep.zarr").exists() + + +def test_metadata_override_adds_extra_keys(tmp_path): + attrs = _attrs( + _write(tmp_path / "ep.zarr", metadata_override={"hand_used": "left"}) + ) + assert attrs["hand_used"] == "left" + + +def test_failed_rewrite_keeps_existing_episode(tmp_path): + path = _write(tmp_path / "ep.zarr") + before = _attrs(path) + writer = ZarrWriter( + path, embodiment="human_bimanual", intrinsics={"front_1": K}, **GOOD + ) + with pytest.raises(ValueError, match="images.front_1"): + writer.write( + numeric_data={"actions": np.zeros((T, 3), np.float32)}, + image_data={"images.front_1": np.zeros((T, 4, 4), np.uint8)}, + ) + with pytest.raises(ValueError, match="JSON-serializable"): + writer.write( + numeric_data={"actions": np.zeros((T, 3), np.float32)}, + metadata_override={"x": object()}, + ) + assert _attrs(path) == before + assert [p.name for p in tmp_path.iterdir()] == ["ep.zarr"] + + +def test_rewrite_does_not_keep_stale_features(tmp_path): + writer = ZarrWriter( + tmp_path / "ep.zarr", + embodiment="human_bimanual", + intrinsics={"front_1": K}, + **GOOD, + ) + writer.write(numeric_data={"a": np.zeros((T, 3), np.float32)}) + path = writer.write(numeric_data={"b": np.zeros((T, 3), np.float32)}) + assert set(_attrs(path)["features"]) == {"b", "annotations"} + + +def test_waiver_rejects_value_and_unknown_fields(): + with pytest.raises(ValueError, match="both"): + required_metadata_from_args( + argparse.Namespace(scene="kitchen", missing_metadata=["scene"]) + ) + with pytest.raises(ValueError, match="unknown fields"): + required_metadata_from_args(argparse.Namespace(missing_metadata=["opertor"])) + + +def test_cli_args_feed_writer(tmp_path): + parser = argparse.ArgumentParser() + parser.add_argument("--task-name") + add_required_metadata_args(parser) + args = parser.parse_args( + [ + "--task-name=fold_clothes", + "--operator=op_7", + "--lab=rl2", + "--objects=shirt", + "--missing-metadata", + "scene", + ] + ) + prov = required_metadata_from_args(args) + assert isinstance(prov["scene"], MissingMetadata) + attrs = _attrs(_write(tmp_path / "ep.zarr", **prov)) + assert attrs["scene"] is None + + with pytest.raises(SystemExit): # typo'd field name is an argparse error + parser.parse_args(["--missing-metadata", "scen"]) + + # Forgetting a flag (no waiver) fails at write time. + bare = parser.parse_args(["--task-name=fold_clothes"]) + with pytest.raises(ValueError, match="operator"): + _write(tmp_path / "bare.zarr", **required_metadata_from_args(bare)) + + +def test_validator_passes_fixture_episode(tmp_path): + ep = write_episode(tmp_path, "aria", T=16, H=32, W=32) + report = validate.validate_episode(ep) + assert report.ok, report.errors + assert validate.main([str(ep)]) == 0 + + +def test_validator_flags_waived_and_absent_fields(tmp_path): + ep = write_episode(tmp_path, "aria", T=16, H=32, W=32) + group = zarr.open_group(str(ep), mode="r+") + group.attrs["objects"] = None + del group.attrs["scene"] + + strict = validate.validate_episode(ep) + assert any("objects" in e for e in strict.errors) + assert any("scene" in e for e in strict.errors) + + lenient = validate.validate_episode(ep, allow_missing=True) + assert any("objects" in w for w in lenient.warnings) + assert not any("objects" in e for e in lenient.errors) + assert any("scene" in e for e in lenient.errors) # absent != waived + assert validate.main([str(ep), "--allow-missing"]) == 1 + + +@pytest.mark.parametrize("field", ["task_name", "operator", "lab"]) +def test_validator_rejects_null_core_field_even_with_allow_missing(tmp_path, field): + ep = write_episode(tmp_path, "aria", T=16, H=32, W=32) + zarr.open_group(str(ep), mode="r+").attrs[field] = None + report = validate.validate_episode(ep, allow_missing=True) + assert any(field in e and "cannot be waived" in e for e in report.errors) + + +def test_legacy_episode_metadata_is_only_warned(tmp_path): + ep = write_episode(tmp_path, "aria", T=16, H=32, W=32) + group = zarr.open_group(str(ep), mode="r+") + attrs = dict(group.attrs) + for key in ("schema_version", "operator", "lab", "scene"): + del attrs[key] + attrs.update(task_name="debug", objects=[]) + group.attrs.clear() + group.attrs.update(attrs) + report = validate.validate_episode(ep) + assert report.ok, report.errors + assert any("legacy episode" in w for w in report.warnings) + assert any("operator" in w for w in report.warnings) + + group["left.obs_ee_pose"][0] = np.full(7, np.nan) # arrays are still checked + assert any("NaN" in e for e in validate.validate_episode(ep).errors) + + +@pytest.mark.parametrize("version", [0, SCHEMA_VERSION + 1, "1"]) +def test_validator_rejects_unknown_schema_version(tmp_path, version): + ep = write_episode(tmp_path, "aria", T=16, H=32, W=32) + zarr.open_group(str(ep), mode="r+").attrs["schema_version"] = version + report = validate.validate_episode(ep) + assert any("schema_version" in e for e in report.errors) + + +def test_validator_accepts_waiver_with_allow_missing(tmp_path): + ep = write_episode(tmp_path, "aria", T=16, H=32, W=32) + group = zarr.open_group(str(ep), mode="r+") + group.attrs["scene"] = None + assert validate.main([str(ep)]) == 1 + assert validate.main([str(ep), "--allow-missing"]) == 0 + + +@pytest.mark.parametrize("field", ["task_name", "operator", "scene"]) +def test_surrounding_whitespace_is_rejected(tmp_path, field): + with pytest.raises(ValueError, match="whitespace"): + _write(tmp_path / "ep.zarr", **{field: f" {GOOD[field]}\n"}) + + +@pytest.mark.parametrize( + "value", ["template", "Missing", "NaN", "...", "op\u200b7", "a\nb"] +) +def test_placeholders_and_nonprintable_values_are_rejected(tmp_path, value): + with pytest.raises(ValueError, match="placeholder|non-printable"): + _write(tmp_path / "ep.zarr", operator=value) + with pytest.raises(ValueError, match="objects"): + _write(tmp_path / "ep.zarr", objects=["shirt", value]) + + +def test_objects_string_is_split_and_stored_as_list(tmp_path): + attrs = _attrs(_write(tmp_path / "ep.zarr", objects="shirt, basket")) + assert attrs["objects"] == ["shirt", "basket"] + for bad in ("shirt, n/a", "shirt,,basket", ",", " shirt"): + with pytest.raises(ValueError, match="objects"): + _write(tmp_path / "bad.zarr", objects=bad) + + +def _corrupt(ep, **attrs): + zarr.open_group(str(ep), mode="a").attrs.update(attrs) + return validate.validate_episode(ep) + + +@pytest.mark.parametrize( + "attrs, match", + [ + ({"total_frames": None}, "total_frames"), + ({"intrinsics": {"front_1": [[1, 2], [3]]}}, "must be a 3x4 K matrix"), + ( + { + "intrinsics": { + "front_1": [[float("nan"), 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0]] + } + }, + "NaN or inf", + ), + ({"intrinsics": {"front_1": [[0] * 4] * 3}}, "positive focal lengths"), + ({"extrinsics": {"left": [[float("inf")] * 4] * 4}}, "NaN or inf"), + ({"features": {"images.front_1": "jpeg"}}, "could not finish"), + ], +) +def test_validator_reports_malformed_attrs(tmp_path, attrs, match): + report = _corrupt(write_episode(tmp_path, "aria", T=16, H=32, W=32), **attrs) + assert not report.ok + assert any(match in e for e in report.errors) + + +def test_validator_rejects_nan_poses(tmp_path): + ep = write_episode(tmp_path, "aria", T=16, H=32, W=32) + store = zarr.open_group(str(ep), mode="a") + pose = store["obs_head_pose"][:] + pose[3] = np.nan + store["obs_head_pose"][:] = pose + report = validate.validate_episode(ep) + assert any("obs_head_pose: contains NaN" in e for e in report.errors) + + +def test_float32_intrinsics_lists_are_stored(tmp_path): + K32 = [[np.float32(v) for v in row] for row in K] + path = ZarrWriter( + tmp_path / "ep.zarr", + embodiment="human_bimanual", + intrinsics={"front_1": K32}, + **GOOD, + ).write(numeric_data={"actions": np.zeros((T, 3), np.float32)}) + assert _attrs(path)["intrinsics"]["front_1"] == K.tolist() + + +def test_stale_temp_dirs_are_swept_and_non_dirs_rejected(tmp_path): + (tmp_path / "ep.zarr.tmp-1").mkdir() + (tmp_path / "ep.zarr.tmp-2").write_text("x") + (tmp_path / "ep.zarr.old-3").mkdir() + _write(tmp_path / "ep.zarr") + assert [p.name for p in tmp_path.iterdir()] == ["ep.zarr"] + + (tmp_path / "file.zarr").write_text("x") + (tmp_path / "link.zarr").symlink_to(tmp_path / "ep.zarr") + for name in ("file.zarr", "link.zarr"): + with pytest.raises(ValueError, match="not an episode directory"): + _write(tmp_path / name) + + +def test_orphaned_old_copy_is_restored_not_swept(tmp_path): + good = _write(tmp_path / "ep.zarr") + good.rename(tmp_path / "ep.zarr.old-99") # killed between the two renames + with pytest.raises(ValueError): + _write(tmp_path / "ep.zarr", operator="unknown") + assert _attrs(tmp_path / "ep.zarr")["operator"] == GOOD["operator"] + + +@pytest.mark.parametrize("value", ["", "unknown", "a/b", " s1", None]) +def test_source_recording_id_is_required(tmp_path, value): + with pytest.raises(ValueError, match="source_recording_id"): + _write(tmp_path / "ep.zarr", source_recording_id=value) + assert not (tmp_path / "ep.zarr").exists() + + +def test_validator_requires_source_recording_id_and_head_pose(tmp_path): + ep = write_episode(tmp_path, "aria", T=16, H=32, W=32) + group = zarr.open_group(str(ep), mode="r+") + attrs = dict(group.attrs) + del attrs["source_recording_id"] + group.attrs.clear() + group.attrs.update(attrs) + del group["obs_head_pose"] + errors = validate.validate_episode(ep).errors + assert any("source_recording_id" in e for e in errors) + assert any("Missing required key: obs_head_pose" in e for e in errors) + + +def test_robot_episode_needs_no_head_pose(tmp_path): + report = validate.validate_episode(write_episode(tmp_path, "eva", T=16, H=32, W=32)) + assert report.ok, report.errors + + +def test_validator_checks_episode_hash_name(tmp_path): + ep = write_episode(tmp_path, "aria", T=16, H=32, W=32) + assert any("episode_hash" in w for w in validate.validate_episode(ep).warnings) + good = ep.rename(tmp_path / "2026-01-12-03-47-29-664000.zarr") + assert validate.validate_episode(good).ok + old = good.rename(tmp_path / "1970-01-21-07-03-09-664000.zarr") + report = validate.validate_episode(old) + assert any("epoch milliseconds" in e for e in report.errors) diff --git a/uv.lock b/uv.lock index 13130da7b9..81faeb6b80 100644 --- a/uv.lock +++ b/uv.lock @@ -6,10 +6,10 @@ resolution-markers = [ "python_full_version >= '3.14' and sys_platform == 'emscripten'", "python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32'", "python_full_version == '3.13.*' and sys_platform == 'win32'", - "python_full_version == '3.12.*' and sys_platform == 'win32'", "python_full_version == '3.13.*' and sys_platform == 'emscripten'", - "python_full_version == '3.12.*' and sys_platform == 'emscripten'", "python_full_version == '3.13.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", + "python_full_version == '3.12.*' and sys_platform == 'win32'", + "python_full_version == '3.12.*' and sys_platform == 'emscripten'", "python_full_version == '3.12.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", "python_full_version < '3.12' and sys_platform == 'win32'", "python_full_version < '3.12' and sys_platform == 'emscripten'", @@ -1030,6 +1030,7 @@ dependencies = [ { name = "datasets" }, { name = "dm-control" }, { name = "einops" }, + { name = "foxglove-schemas-protobuf" }, { name = "geomloss" }, { name = "gpustat" }, { name = "h5py" }, @@ -1038,6 +1039,8 @@ dependencies = [ { name = "ipython" }, { name = "lightning" }, { name = "matplotlib" }, + { name = "mcap" }, + { name = "mcap-protobuf-support" }, { name = "mediapy" }, { name = "mink" }, { name = "mujoco" }, @@ -1092,6 +1095,7 @@ requires-dist = [ { name = "datasets", specifier = "==4.0.0" }, { name = "dm-control", specifier = "==1.0.8" }, { name = "einops" }, + { name = "foxglove-schemas-protobuf", specifier = "==0.4.0" }, { name = "geomloss" }, { name = "gpustat" }, { name = "h5py" }, @@ -1100,6 +1104,8 @@ requires-dist = [ { name = "ipython" }, { name = "lightning" }, { name = "matplotlib" }, + { name = "mcap", specifier = "==1.5.0" }, + { name = "mcap-protobuf-support", specifier = "==0.5.4" }, { name = "mediapy" }, { name = "mink", specifier = "==0.0.13" }, { name = "mujoco", specifier = "==3.4.0" }, @@ -1243,6 +1249,18 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/fd/ba/56147c165442cc5ba7e82ecf301c9a68353cede498185869e6e02b4c264f/fonttools-4.62.1-py3-none-any.whl", hash = "sha256:7487782e2113861f4ddcc07c3436450659e3caa5e470b27dc2177cade2d8e7fd", size = 1152647, upload-time = "2026-03-13T13:54:22.735Z" }, ] +[[package]] +name = "foxglove-schemas-protobuf" +version = "0.4.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "protobuf" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/47/d8/7ca265b4721e2ae912cd0fa5cc96d90357b5ae1ae15c5dff653bf9f22ec5/foxglove_schemas_protobuf-0.4.0.tar.gz", hash = "sha256:3fc9610d7ec6cc4b4029f746ab9699a4ad396bdd7551b99d2a37509d13ef3cc2", size = 31026, upload-time = "2026-07-29T20:46:47.162Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b1/ff/67f49b46e40dca6fb5e5077c3f665c07155f5a878cf349185957d075e999/foxglove_schemas_protobuf-0.4.0-py3-none-any.whl", hash = "sha256:e6a35df574f4c10ea455e8660abd09af028ea00d90e01c78f8696972e2decc95", size = 98809, upload-time = "2026-07-29T20:46:45.863Z" }, +] + [[package]] name = "fqdn" version = "1.5.1" @@ -1864,7 +1882,7 @@ name = "jinxed" version = "2.0.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "ansicon", marker = "sys_platform == 'win32'" }, + { name = "ansicon" }, ] sdist = { url = "https://files.pythonhosted.org/packages/cb/eb/8821ce6e7386e96355f2c6be83944925b4a0870572896fd33a4e61b8aa5a/jinxed-2.0.0.tar.gz", hash = "sha256:64b960b8f8d9966e1b2e7cb57ea2b1aa0a8d7e68045b96dc7ef58201e43a1209", size = 127118, upload-time = "2026-05-08T21:25:25.917Z" } wheels = [ @@ -2549,6 +2567,54 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/40/44/3ee09a5b60cb44c4f2fbc1c9015cfd6ff5afc08f991cab295d3024dcbf2d/lxml-6.1.0-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:7da13bb6fbadfafb474e0226a30570a3445cfd47c86296f2446dafbd77079ace", size = 3508860, upload-time = "2026-04-18T04:32:48.619Z" }, ] +[[package]] +name = "lz4" +version = "4.4.5" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/57/51/f1b86d93029f418033dddf9b9f79c8d2641e7454080478ee2aab5123173e/lz4-4.4.5.tar.gz", hash = "sha256:5f0b9e53c1e82e88c10d7c180069363980136b9d7a8306c4dca4f760d60c39f0", size = 172886, upload-time = "2025-11-03T13:02:36.061Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/93/5b/6edcd23319d9e28b1bedf32768c3d1fd56eed8223960a2c47dacd2cec2af/lz4-4.4.5-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:d6da84a26b3aa5da13a62e4b89ab36a396e9327de8cd48b436a3467077f8ccd4", size = 207391, upload-time = "2025-11-03T13:01:36.644Z" }, + { url = "https://files.pythonhosted.org/packages/34/36/5f9b772e85b3d5769367a79973b8030afad0d6b724444083bad09becd66f/lz4-4.4.5-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:61d0ee03e6c616f4a8b69987d03d514e8896c8b1b7cc7598ad029e5c6aedfd43", size = 207146, upload-time = "2025-11-03T13:01:37.928Z" }, + { url = "https://files.pythonhosted.org/packages/04/f4/f66da5647c0d72592081a37c8775feacc3d14d2625bbdaabd6307c274565/lz4-4.4.5-cp311-cp311-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:33dd86cea8375d8e5dd001e41f321d0a4b1eb7985f39be1b6a4f466cd480b8a7", size = 1292623, upload-time = "2025-11-03T13:01:39.341Z" }, + { url = "https://files.pythonhosted.org/packages/85/fc/5df0f17467cdda0cad464a9197a447027879197761b55faad7ca29c29a04/lz4-4.4.5-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:609a69c68e7cfcfa9d894dc06be13f2e00761485b62df4e2472f1b66f7b405fb", size = 1279982, upload-time = "2025-11-03T13:01:40.816Z" }, + { url = "https://files.pythonhosted.org/packages/25/3b/b55cb577aa148ed4e383e9700c36f70b651cd434e1c07568f0a86c9d5fbb/lz4-4.4.5-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:75419bb1a559af00250b8f1360d508444e80ed4b26d9d40ec5b09fe7875cb989", size = 1368674, upload-time = "2025-11-03T13:01:42.118Z" }, + { url = "https://files.pythonhosted.org/packages/fb/31/e97e8c74c59ea479598e5c55cbe0b1334f03ee74ca97726e872944ed42df/lz4-4.4.5-cp311-cp311-win32.whl", hash = "sha256:12233624f1bc2cebc414f9efb3113a03e89acce3ab6f72035577bc61b270d24d", size = 88168, upload-time = "2025-11-03T13:01:43.282Z" }, + { url = "https://files.pythonhosted.org/packages/18/47/715865a6c7071f417bef9b57c8644f29cb7a55b77742bd5d93a609274e7e/lz4-4.4.5-cp311-cp311-win_amd64.whl", hash = "sha256:8a842ead8ca7c0ee2f396ca5d878c4c40439a527ebad2b996b0444f0074ed004", size = 99491, upload-time = "2025-11-03T13:01:44.167Z" }, + { url = "https://files.pythonhosted.org/packages/14/e7/ac120c2ca8caec5c945e6356ada2aa5cfabd83a01e3170f264a5c42c8231/lz4-4.4.5-cp311-cp311-win_arm64.whl", hash = "sha256:83bc23ef65b6ae44f3287c38cbf82c269e2e96a26e560aa551735883388dcc4b", size = 91271, upload-time = "2025-11-03T13:01:45.016Z" }, + { url = "https://files.pythonhosted.org/packages/1b/ac/016e4f6de37d806f7cc8f13add0a46c9a7cfc41a5ddc2bc831d7954cf1ce/lz4-4.4.5-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:df5aa4cead2044bab83e0ebae56e0944cc7fcc1505c7787e9e1057d6d549897e", size = 207163, upload-time = "2025-11-03T13:01:45.895Z" }, + { url = "https://files.pythonhosted.org/packages/8d/df/0fadac6e5bd31b6f34a1a8dbd4db6a7606e70715387c27368586455b7fc9/lz4-4.4.5-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:6d0bf51e7745484d2092b3a51ae6eb58c3bd3ce0300cf2b2c14f76c536d5697a", size = 207150, upload-time = "2025-11-03T13:01:47.205Z" }, + { url = "https://files.pythonhosted.org/packages/b7/17/34e36cc49bb16ca73fb57fbd4c5eaa61760c6b64bce91fcb4e0f4a97f852/lz4-4.4.5-cp312-cp312-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:7b62f94b523c251cf32aa4ab555f14d39bd1a9df385b72443fd76d7c7fb051f5", size = 1292045, upload-time = "2025-11-03T13:01:48.667Z" }, + { url = "https://files.pythonhosted.org/packages/90/1c/b1d8e3741e9fc89ed3b5f7ef5f22586c07ed6bb04e8343c2e98f0fa7ff04/lz4-4.4.5-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2c3ea562c3af274264444819ae9b14dbbf1ab070aff214a05e97db6896c7597e", size = 1279546, upload-time = "2025-11-03T13:01:50.159Z" }, + { url = "https://files.pythonhosted.org/packages/55/d9/e3867222474f6c1b76e89f3bd914595af69f55bf2c1866e984c548afdc15/lz4-4.4.5-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:24092635f47538b392c4eaeff14c7270d2c8e806bf4be2a6446a378591c5e69e", size = 1368249, upload-time = "2025-11-03T13:01:51.273Z" }, + { url = "https://files.pythonhosted.org/packages/b2/e7/d667d337367686311c38b580d1ca3d5a23a6617e129f26becd4f5dc458df/lz4-4.4.5-cp312-cp312-win32.whl", hash = "sha256:214e37cfe270948ea7eb777229e211c601a3e0875541c1035ab408fbceaddf50", size = 88189, upload-time = "2025-11-03T13:01:52.605Z" }, + { url = "https://files.pythonhosted.org/packages/a5/0b/a54cd7406995ab097fceb907c7eb13a6ddd49e0b231e448f1a81a50af65c/lz4-4.4.5-cp312-cp312-win_amd64.whl", hash = "sha256:713a777de88a73425cf08eb11f742cd2c98628e79a8673d6a52e3c5f0c116f33", size = 99497, upload-time = "2025-11-03T13:01:53.477Z" }, + { url = "https://files.pythonhosted.org/packages/6a/7e/dc28a952e4bfa32ca16fa2eb026e7a6ce5d1411fcd5986cd08c74ec187b9/lz4-4.4.5-cp312-cp312-win_arm64.whl", hash = "sha256:a88cbb729cc333334ccfb52f070463c21560fca63afcf636a9f160a55fac3301", size = 91279, upload-time = "2025-11-03T13:01:54.419Z" }, + { url = "https://files.pythonhosted.org/packages/2f/46/08fd8ef19b782f301d56a9ccfd7dafec5fd4fc1a9f017cf22a1accb585d7/lz4-4.4.5-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:6bb05416444fafea170b07181bc70640975ecc2a8c92b3b658c554119519716c", size = 207171, upload-time = "2025-11-03T13:01:56.595Z" }, + { url = "https://files.pythonhosted.org/packages/8f/3f/ea3334e59de30871d773963997ecdba96c4584c5f8007fd83cfc8f1ee935/lz4-4.4.5-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:b424df1076e40d4e884cfcc4c77d815368b7fb9ebcd7e634f937725cd9a8a72a", size = 207163, upload-time = "2025-11-03T13:01:57.721Z" }, + { url = "https://files.pythonhosted.org/packages/41/7b/7b3a2a0feb998969f4793c650bb16eff5b06e80d1f7bff867feb332f2af2/lz4-4.4.5-cp313-cp313-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:216ca0c6c90719731c64f41cfbd6f27a736d7e50a10b70fad2a9c9b262ec923d", size = 1292136, upload-time = "2025-11-03T13:02:00.375Z" }, + { url = "https://files.pythonhosted.org/packages/89/d1/f1d259352227bb1c185288dd694121ea303e43404aa77560b879c90e7073/lz4-4.4.5-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:533298d208b58b651662dd972f52d807d48915176e5b032fb4f8c3b6f5fe535c", size = 1279639, upload-time = "2025-11-03T13:02:01.649Z" }, + { url = "https://files.pythonhosted.org/packages/d2/fb/ba9256c48266a09012ed1d9b0253b9aa4fe9cdff094f8febf5b26a4aa2a2/lz4-4.4.5-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:451039b609b9a88a934800b5fc6ee401c89ad9c175abf2f4d9f8b2e4ef1afc64", size = 1368257, upload-time = "2025-11-03T13:02:03.35Z" }, + { url = "https://files.pythonhosted.org/packages/a5/6d/dee32a9430c8b0e01bbb4537573cabd00555827f1a0a42d4e24ca803935c/lz4-4.4.5-cp313-cp313-win32.whl", hash = "sha256:a5f197ffa6fc0e93207b0af71b302e0a2f6f29982e5de0fbda61606dd3a55832", size = 88191, upload-time = "2025-11-03T13:02:04.406Z" }, + { url = "https://files.pythonhosted.org/packages/18/e0/f06028aea741bbecb2a7e9648f4643235279a770c7ffaf70bd4860c73661/lz4-4.4.5-cp313-cp313-win_amd64.whl", hash = "sha256:da68497f78953017deb20edff0dba95641cc86e7423dfadf7c0264e1ac60dc22", size = 99502, upload-time = "2025-11-03T13:02:05.886Z" }, + { url = "https://files.pythonhosted.org/packages/61/72/5bef44afb303e56078676b9f2486f13173a3c1e7f17eaac1793538174817/lz4-4.4.5-cp313-cp313-win_arm64.whl", hash = "sha256:c1cfa663468a189dab510ab231aad030970593f997746d7a324d40104db0d0a9", size = 91285, upload-time = "2025-11-03T13:02:06.77Z" }, + { url = "https://files.pythonhosted.org/packages/49/55/6a5c2952971af73f15ed4ebfdd69774b454bd0dc905b289082ca8664fba1/lz4-4.4.5-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:67531da3b62f49c939e09d56492baf397175ff39926d0bd5bd2d191ac2bff95f", size = 207348, upload-time = "2025-11-03T13:02:08.117Z" }, + { url = "https://files.pythonhosted.org/packages/4e/d7/fd62cbdbdccc35341e83aabdb3f6d5c19be2687d0a4eaf6457ddf53bba64/lz4-4.4.5-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:a1acbbba9edbcbb982bc2cac5e7108f0f553aebac1040fbec67a011a45afa1ba", size = 207340, upload-time = "2025-11-03T13:02:09.152Z" }, + { url = "https://files.pythonhosted.org/packages/77/69/225ffadaacb4b0e0eb5fd263541edd938f16cd21fe1eae3cd6d5b6a259dc/lz4-4.4.5-cp313-cp313t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:a482eecc0b7829c89b498fda883dbd50e98153a116de612ee7c111c8bcf82d1d", size = 1293398, upload-time = "2025-11-03T13:02:10.272Z" }, + { url = "https://files.pythonhosted.org/packages/c6/9e/2ce59ba4a21ea5dc43460cba6f34584e187328019abc0e66698f2b66c881/lz4-4.4.5-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e099ddfaa88f59dd8d36c8a3c66bd982b4984edf127eb18e30bb49bdba68ce67", size = 1281209, upload-time = "2025-11-03T13:02:12.091Z" }, + { url = "https://files.pythonhosted.org/packages/80/4f/4d946bd1624ec229b386a3bc8e7a85fa9a963d67d0a62043f0af0978d3da/lz4-4.4.5-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a2af2897333b421360fdcce895c6f6281dc3fab018d19d341cf64d043fc8d90d", size = 1369406, upload-time = "2025-11-03T13:02:13.683Z" }, + { url = "https://files.pythonhosted.org/packages/02/a2/d429ba4720a9064722698b4b754fb93e42e625f1318b8fe834086c7c783b/lz4-4.4.5-cp313-cp313t-win32.whl", hash = "sha256:66c5de72bf4988e1b284ebdd6524c4bead2c507a2d7f172201572bac6f593901", size = 88325, upload-time = "2025-11-03T13:02:14.743Z" }, + { url = "https://files.pythonhosted.org/packages/4b/85/7ba10c9b97c06af6c8f7032ec942ff127558863df52d866019ce9d2425cf/lz4-4.4.5-cp313-cp313t-win_amd64.whl", hash = "sha256:cdd4bdcbaf35056086d910d219106f6a04e1ab0daa40ec0eeef1626c27d0fddb", size = 99643, upload-time = "2025-11-03T13:02:15.978Z" }, + { url = "https://files.pythonhosted.org/packages/77/4d/a175459fb29f909e13e57c8f475181ad8085d8d7869bd8ad99033e3ee5fa/lz4-4.4.5-cp313-cp313t-win_arm64.whl", hash = "sha256:28ccaeb7c5222454cd5f60fcd152564205bcb801bd80e125949d2dfbadc76bbd", size = 91504, upload-time = "2025-11-03T13:02:17.313Z" }, + { url = "https://files.pythonhosted.org/packages/63/9c/70bdbdb9f54053a308b200b4678afd13efd0eafb6ddcbb7f00077213c2e5/lz4-4.4.5-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:c216b6d5275fc060c6280936bb3bb0e0be6126afb08abccde27eed23dead135f", size = 207586, upload-time = "2025-11-03T13:02:18.263Z" }, + { url = "https://files.pythonhosted.org/packages/b6/cb/bfead8f437741ce51e14b3c7d404e3a1f6b409c440bad9b8f3945d4c40a7/lz4-4.4.5-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:c8e71b14938082ebaf78144f3b3917ac715f72d14c076f384a4c062df96f9df6", size = 207161, upload-time = "2025-11-03T13:02:19.286Z" }, + { url = "https://files.pythonhosted.org/packages/e7/18/b192b2ce465dfbeabc4fc957ece7a1d34aded0d95a588862f1c8a86ac448/lz4-4.4.5-cp314-cp314-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:9b5e6abca8df9f9bdc5c3085f33ff32cdc86ed04c65e0355506d46a5ac19b6e9", size = 1292415, upload-time = "2025-11-03T13:02:20.829Z" }, + { url = "https://files.pythonhosted.org/packages/67/79/a4e91872ab60f5e89bfad3e996ea7dc74a30f27253faf95865771225ccba/lz4-4.4.5-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3b84a42da86e8ad8537aabef062e7f661f4a877d1c74d65606c49d835d36d668", size = 1279920, upload-time = "2025-11-03T13:02:22.013Z" }, + { url = "https://files.pythonhosted.org/packages/f1/01/d52c7b11eaa286d49dae619c0eec4aabc0bf3cda7a7467eb77c62c4471f3/lz4-4.4.5-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0bba042ec5a61fa77c7e380351a61cb768277801240249841defd2ff0a10742f", size = 1368661, upload-time = "2025-11-03T13:02:23.208Z" }, + { url = "https://files.pythonhosted.org/packages/f7/da/137ddeea14c2cb86864838277b2607d09f8253f152156a07f84e11768a28/lz4-4.4.5-cp314-cp314-win32.whl", hash = "sha256:bd85d118316b53ed73956435bee1997bd06cc66dd2fa74073e3b1322bd520a67", size = 90139, upload-time = "2025-11-03T13:02:24.301Z" }, + { url = "https://files.pythonhosted.org/packages/18/2c/8332080fd293f8337779a440b3a143f85e374311705d243439a3349b81ad/lz4-4.4.5-cp314-cp314-win_amd64.whl", hash = "sha256:92159782a4502858a21e0079d77cdcaade23e8a5d252ddf46b0652604300d7be", size = 101497, upload-time = "2025-11-03T13:02:25.187Z" }, + { url = "https://files.pythonhosted.org/packages/ca/28/2635a8141c9a4f4bc23f5135a92bbcf48d928d8ca094088c962df1879d64/lz4-4.4.5-cp314-cp314-win_arm64.whl", hash = "sha256:d994b87abaa7a88ceb7a37c90f547b8284ff9da694e6afcfaa8568d739faf3f7", size = 93812, upload-time = "2025-11-03T13:02:26.133Z" }, +] + [[package]] name = "markdown-it-py" version = "4.2.0" @@ -2711,6 +2777,32 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/41/09/5b161152e2d90f7b87f781c2e1267494aef9c32498df793f73ad0a0a494a/matplotlib_inline-0.2.2-py3-none-any.whl", hash = "sha256:3c821cf1c209f59fb2d2d64abbf5b23b67bcb2210d663f9918dd851c6da1fcf6", size = 9534, upload-time = "2026-05-08T17:33:32.055Z" }, ] +[[package]] +name = "mcap" +version = "1.5.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "lz4" }, + { name = "zstandard" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/26/16/2ac1cbeaa524453d616050db9fda14010eda507fd8130e0676879bb8faf9/mcap-1.5.0.tar.gz", hash = "sha256:9c385cc5e5a6bccff4aa0c6814b305faa2c3239ece46778661eaa6d809b1768a", size = 23110, upload-time = "2026-09-24T21:45:44.714Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/13/59/caa4d60a58f129659ea5403e6fb52808f87005dfcbaa03e84f0e07f0033e/mcap-1.5.0-py3-none-any.whl", hash = "sha256:44ba129d381abdca474fbf5bcee036db87d7075d9059b139bf2fee2975c8bd52", size = 21021, upload-time = "2026-09-24T21:45:43.51Z" }, +] + +[[package]] +name = "mcap-protobuf-support" +version = "0.5.4" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "mcap" }, + { name = "protobuf" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/cc/8e/98af824adce25082f1c7fb8e8aab48b1678a370f97633a55bfb06c40e3fd/mcap_protobuf_support-0.5.4.tar.gz", hash = "sha256:3af2de2c2dbda9d1dbb1a526d2ad0cf3604ab44947dd1c99a52a2f990b1eb308", size = 7059, upload-time = "2025-12-24T21:22:57.716Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/74/ee/cb4beace2bbaf3877848dbc555366c5a72ec1dabba5d620bf746143d91c4/mcap_protobuf_support-0.5.4-py3-none-any.whl", hash = "sha256:072f83b7b147b5faa25df21c175cbf6e53bf959f8a6aea534167055bc4526649", size = 7286, upload-time = "2025-12-24T21:22:56.393Z" }, +] + [[package]] name = "mdurl" version = "0.1.2" @@ -3309,7 +3401,7 @@ name = "nvidia-cudnn-cu12" version = "9.5.1.17" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "nvidia-cublas-cu12", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" }, + { name = "nvidia-cublas-cu12" }, ] wheels = [ { url = "https://files.pythonhosted.org/packages/2a/78/4535c9c7f859a64781e43c969a3a7e84c54634e319a996d43ef32ce46f83/nvidia_cudnn_cu12-9.5.1.17-py3-none-manylinux_2_28_x86_64.whl", hash = "sha256:30ac3869f6db17d170e0e556dd6cc5eee02647abc31ca856634d5a40f82c15b2", size = 570988386, upload-time = "2024-10-25T19:54:26.39Z" }, @@ -3320,7 +3412,7 @@ name = "nvidia-cufft-cu12" version = "11.3.0.4" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "nvidia-nvjitlink-cu12", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" }, + { name = "nvidia-nvjitlink-cu12" }, ] wheels = [ { url = "https://files.pythonhosted.org/packages/8f/16/73727675941ab8e6ffd86ca3a4b7b47065edcca7a997920b831f8147c99d/nvidia_cufft_cu12-11.3.0.4-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:ccba62eb9cef5559abd5e0d54ceed2d9934030f51163df018532142a8ec533e5", size = 200221632, upload-time = "2024-11-20T17:41:32.357Z" }, @@ -3349,9 +3441,9 @@ name = "nvidia-cusolver-cu12" version = "11.7.1.2" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "nvidia-cublas-cu12", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" }, - { name = "nvidia-cusparse-cu12", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" }, - { name = "nvidia-nvjitlink-cu12", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" }, + { name = "nvidia-cublas-cu12" }, + { name = "nvidia-cusparse-cu12" }, + { name = "nvidia-nvjitlink-cu12" }, ] wheels = [ { url = "https://files.pythonhosted.org/packages/f0/6e/c2cf12c9ff8b872e92b4a5740701e51ff17689c4d726fca91875b07f655d/nvidia_cusolver_cu12-11.7.1.2-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:e9e49843a7707e42022babb9bcfa33c29857a93b88020c4e4434656a655b698c", size = 158229790, upload-time = "2024-11-20T17:43:43.211Z" }, @@ -3363,7 +3455,7 @@ name = "nvidia-cusparse-cu12" version = "12.5.4.2" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "nvidia-nvjitlink-cu12", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" }, + { name = "nvidia-nvjitlink-cu12" }, ] wheels = [ { url = "https://files.pythonhosted.org/packages/06/1e/b8b7c2f4099a37b96af5c9bb158632ea9e5d9d27d7391d7eb8fc45236674/nvidia_cusparse_cu12-12.5.4.2-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:7556d9eca156e18184b94947ade0fba5bb47d69cec46bf8660fd2c71a4b48b73", size = 216561367, upload-time = "2024-11-20T17:44:54.824Z" }, @@ -3572,7 +3664,7 @@ name = "pexpect" version = "4.9.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "ptyprocess", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" }, + { name = "ptyprocess" }, ] sdist = { url = "https://files.pythonhosted.org/packages/42/92/cc564bf6381ff43ce1f4d06852fc19a2f11d180f23dc32d9588bee2f149d/pexpect-4.9.0.tar.gz", hash = "sha256:ee7d41123f3c9911050ea2c2dac107568dc43b2d3b0c7557a33212c398ead30f", size = 166450, upload-time = "2023-11-25T09:07:26.339Z" } wheels = [ @@ -3707,7 +3799,7 @@ resolution-markers = [ "python_full_version < '3.12' and sys_platform != 'emscripten' and sys_platform != 'win32'", ] dependencies = [ - { name = "numpy", marker = "python_full_version < '3.12'" }, + { name = "numpy" }, ] sdist = { url = "https://files.pythonhosted.org/packages/53/ae/677391c5e75858b5496dcdf4146952646c13a8b31c04090fa98c37a04198/positional_encodings-6.0.3.tar.gz", hash = "sha256:c144ad52e78f09d16effcbfabdf3e84050b56a37e342248476e18bbffd747b24", size = 6626, upload-time = "2024-01-04T18:17:44.133Z" } wheels = [ @@ -3716,7 +3808,7 @@ wheels = [ [package.optional-dependencies] pytorch = [ - { name = "torch", marker = "python_full_version < '3.12'" }, + { name = "torch" }, ] [[package]] @@ -3728,14 +3820,14 @@ resolution-markers = [ "python_full_version >= '3.14' and sys_platform == 'emscripten'", "python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32'", "python_full_version == '3.13.*' and sys_platform == 'win32'", - "python_full_version == '3.12.*' and sys_platform == 'win32'", "python_full_version == '3.13.*' and sys_platform == 'emscripten'", - "python_full_version == '3.12.*' and sys_platform == 'emscripten'", "python_full_version == '3.13.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", + "python_full_version == '3.12.*' and sys_platform == 'win32'", + "python_full_version == '3.12.*' and sys_platform == 'emscripten'", "python_full_version == '3.12.*' and sys_platform != 'emscripten' and sys_platform != 'win32'", ] dependencies = [ - { name = "numpy", marker = "python_full_version >= '3.12'" }, + { name = "numpy" }, ] sdist = { url = "https://files.pythonhosted.org/packages/6d/04/93a78a33088fe9115602883a39f8395cdb5561388a24e65f57c11920c638/positional_encodings-6.0.4.tar.gz", hash = "sha256:435db9596a73759e7caa8f677ebf6d2ee4d39187112244e4d8b6d7b62bc119ad", size = 6814, upload-time = "2024-10-23T07:33:48.273Z" } wheels = [ @@ -3744,7 +3836,7 @@ wheels = [ [package.optional-dependencies] pytorch = [ - { name = "torch", marker = "python_full_version >= '3.12'" }, + { name = "torch" }, ] [[package]] @@ -5641,7 +5733,7 @@ name = "triton" version = "3.3.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "setuptools", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" }, + { name = "setuptools" }, ] wheels = [ { url = "https://files.pythonhosted.org/packages/21/2f/3e56ea7b58f80ff68899b1dbe810ff257c9d177d288c6b0f55bf2fe4eb50/triton-3.3.1-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b31e3aa26f8cb3cc5bf4e187bf737cbacf17311e1112b781d4a059353dfd731b", size = 155689937, upload-time = "2025-05-29T23:39:44.182Z" }, @@ -6163,3 +6255,77 @@ sdist = { url = "https://files.pythonhosted.org/packages/30/21/093488dfc7cc8964d wheels = [ { url = "https://files.pythonhosted.org/packages/08/8a/0861bec20485572fbddf3dfba2910e38fe249796cb73ecdeb74e07eeb8d3/zipp-3.23.1-py3-none-any.whl", hash = "sha256:0b3596c50a5c700c9cb40ba8d86d9f2cc4807e9bedb06bcdf7fac85633e444dc", size = 10378, upload-time = "2026-04-13T23:21:45.386Z" }, ] + +[[package]] +name = "zstandard" +version = "0.25.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/fd/aa/3e0508d5a5dd96529cdc5a97011299056e14c6505b678fd58938792794b1/zstandard-0.25.0.tar.gz", hash = "sha256:7713e1179d162cf5c7906da876ec2ccb9c3a9dcbdffef0cc7f70c3667a205f0b", size = 711513, upload-time = "2025-09-14T22:15:54.002Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2a/83/c3ca27c363d104980f1c9cee1101cc8ba724ac8c28a033ede6aab89585b1/zstandard-0.25.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:933b65d7680ea337180733cf9e87293cc5500cc0eb3fc8769f4d3c88d724ec5c", size = 795254, upload-time = "2025-09-14T22:16:26.137Z" }, + { url = "https://files.pythonhosted.org/packages/ac/4d/e66465c5411a7cf4866aeadc7d108081d8ceba9bc7abe6b14aa21c671ec3/zstandard-0.25.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:a3f79487c687b1fc69f19e487cd949bf3aae653d181dfb5fde3bf6d18894706f", size = 640559, upload-time = "2025-09-14T22:16:27.973Z" }, + { url = "https://files.pythonhosted.org/packages/12/56/354fe655905f290d3b147b33fe946b0f27e791e4b50a5f004c802cb3eb7b/zstandard-0.25.0-cp311-cp311-manylinux2010_i686.manylinux2014_i686.manylinux_2_12_i686.manylinux_2_17_i686.whl", hash = "sha256:0bbc9a0c65ce0eea3c34a691e3c4b6889f5f3909ba4822ab385fab9057099431", size = 5348020, upload-time = "2025-09-14T22:16:29.523Z" }, + { url = "https://files.pythonhosted.org/packages/3b/13/2b7ed68bd85e69a2069bcc72141d378f22cae5a0f3b353a2c8f50ef30c1b/zstandard-0.25.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:01582723b3ccd6939ab7b3a78622c573799d5d8737b534b86d0e06ac18dbde4a", size = 5058126, upload-time = "2025-09-14T22:16:31.811Z" }, + { url = "https://files.pythonhosted.org/packages/c9/dd/fdaf0674f4b10d92cb120ccff58bbb6626bf8368f00ebfd2a41ba4a0dc99/zstandard-0.25.0-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:5f1ad7bf88535edcf30038f6919abe087f606f62c00a87d7e33e7fc57cb69fcc", size = 5405390, upload-time = "2025-09-14T22:16:33.486Z" }, + { url = "https://files.pythonhosted.org/packages/0f/67/354d1555575bc2490435f90d67ca4dd65238ff2f119f30f72d5cde09c2ad/zstandard-0.25.0-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:06acb75eebeedb77b69048031282737717a63e71e4ae3f77cc0c3b9508320df6", size = 5452914, upload-time = "2025-09-14T22:16:35.277Z" }, + { url = "https://files.pythonhosted.org/packages/bb/1f/e9cfd801a3f9190bf3e759c422bbfd2247db9d7f3d54a56ecde70137791a/zstandard-0.25.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:9300d02ea7c6506f00e627e287e0492a5eb0371ec1670ae852fefffa6164b072", size = 5559635, upload-time = "2025-09-14T22:16:37.141Z" }, + { url = "https://files.pythonhosted.org/packages/21/88/5ba550f797ca953a52d708c8e4f380959e7e3280af029e38fbf47b55916e/zstandard-0.25.0-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:bfd06b1c5584b657a2892a6014c2f4c20e0db0208c159148fa78c65f7e0b0277", size = 5048277, upload-time = "2025-09-14T22:16:38.807Z" }, + { url = "https://files.pythonhosted.org/packages/46/c0/ca3e533b4fa03112facbe7fbe7779cb1ebec215688e5df576fe5429172e0/zstandard-0.25.0-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:f373da2c1757bb7f1acaf09369cdc1d51d84131e50d5fa9863982fd626466313", size = 5574377, upload-time = "2025-09-14T22:16:40.523Z" }, + { url = "https://files.pythonhosted.org/packages/12/9b/3fb626390113f272abd0799fd677ea33d5fc3ec185e62e6be534493c4b60/zstandard-0.25.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:6c0e5a65158a7946e7a7affa6418878ef97ab66636f13353b8502d7ea03c8097", size = 4961493, upload-time = "2025-09-14T22:16:43.3Z" }, + { url = "https://files.pythonhosted.org/packages/cb/d3/23094a6b6a4b1343b27ae68249daa17ae0651fcfec9ed4de09d14b940285/zstandard-0.25.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:c8e167d5adf59476fa3e37bee730890e389410c354771a62e3c076c86f9f7778", size = 5269018, upload-time = "2025-09-14T22:16:45.292Z" }, + { url = "https://files.pythonhosted.org/packages/8c/a7/bb5a0c1c0f3f4b5e9d5b55198e39de91e04ba7c205cc46fcb0f95f0383c1/zstandard-0.25.0-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:98750a309eb2f020da61e727de7d7ba3c57c97cf6213f6f6277bb7fb42a8e065", size = 5443672, upload-time = "2025-09-14T22:16:47.076Z" }, + { url = "https://files.pythonhosted.org/packages/27/22/503347aa08d073993f25109c36c8d9f029c7d5949198050962cb568dfa5e/zstandard-0.25.0-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:22a086cff1b6ceca18a8dd6096ec631e430e93a8e70a9ca5efa7561a00f826fa", size = 5822753, upload-time = "2025-09-14T22:16:49.316Z" }, + { url = "https://files.pythonhosted.org/packages/e2/be/94267dc6ee64f0f8ba2b2ae7c7a2df934a816baaa7291db9e1aa77394c3c/zstandard-0.25.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:72d35d7aa0bba323965da807a462b0966c91608ef3a48ba761678cb20ce5d8b7", size = 5366047, upload-time = "2025-09-14T22:16:51.328Z" }, + { url = "https://files.pythonhosted.org/packages/7b/a3/732893eab0a3a7aecff8b99052fecf9f605cf0fb5fb6d0290e36beee47a4/zstandard-0.25.0-cp311-cp311-win32.whl", hash = "sha256:f5aeea11ded7320a84dcdd62a3d95b5186834224a9e55b92ccae35d21a8b63d4", size = 436484, upload-time = "2025-09-14T22:16:55.005Z" }, + { url = "https://files.pythonhosted.org/packages/43/a3/c6155f5c1cce691cb80dfd38627046e50af3ee9ddc5d0b45b9b063bfb8c9/zstandard-0.25.0-cp311-cp311-win_amd64.whl", hash = "sha256:daab68faadb847063d0c56f361a289c4f268706b598afbf9ad113cbe5c38b6b2", size = 506183, upload-time = "2025-09-14T22:16:52.753Z" }, + { url = "https://files.pythonhosted.org/packages/8c/3e/8945ab86a0820cc0e0cdbf38086a92868a9172020fdab8a03ac19662b0e5/zstandard-0.25.0-cp311-cp311-win_arm64.whl", hash = "sha256:22a06c5df3751bb7dc67406f5374734ccee8ed37fc5981bf1ad7041831fa1137", size = 462533, upload-time = "2025-09-14T22:16:53.878Z" }, + { url = "https://files.pythonhosted.org/packages/82/fc/f26eb6ef91ae723a03e16eddb198abcfce2bc5a42e224d44cc8b6765e57e/zstandard-0.25.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:7b3c3a3ab9daa3eed242d6ecceead93aebbb8f5f84318d82cee643e019c4b73b", size = 795738, upload-time = "2025-09-14T22:16:56.237Z" }, + { url = "https://files.pythonhosted.org/packages/aa/1c/d920d64b22f8dd028a8b90e2d756e431a5d86194caa78e3819c7bf53b4b3/zstandard-0.25.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:913cbd31a400febff93b564a23e17c3ed2d56c064006f54efec210d586171c00", size = 640436, upload-time = "2025-09-14T22:16:57.774Z" }, + { url = "https://files.pythonhosted.org/packages/53/6c/288c3f0bd9fcfe9ca41e2c2fbfd17b2097f6af57b62a81161941f09afa76/zstandard-0.25.0-cp312-cp312-manylinux2010_i686.manylinux2014_i686.manylinux_2_12_i686.manylinux_2_17_i686.whl", hash = "sha256:011d388c76b11a0c165374ce660ce2c8efa8e5d87f34996aa80f9c0816698b64", size = 5343019, upload-time = "2025-09-14T22:16:59.302Z" }, + { url = "https://files.pythonhosted.org/packages/1e/15/efef5a2f204a64bdb5571e6161d49f7ef0fffdbca953a615efbec045f60f/zstandard-0.25.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:6dffecc361d079bb48d7caef5d673c88c8988d3d33fb74ab95b7ee6da42652ea", size = 5063012, upload-time = "2025-09-14T22:17:01.156Z" }, + { url = "https://files.pythonhosted.org/packages/b7/37/a6ce629ffdb43959e92e87ebdaeebb5ac81c944b6a75c9c47e300f85abdf/zstandard-0.25.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:7149623bba7fdf7e7f24312953bcf73cae103db8cae49f8154dd1eadc8a29ecb", size = 5394148, upload-time = "2025-09-14T22:17:03.091Z" }, + { url = "https://files.pythonhosted.org/packages/e3/79/2bf870b3abeb5c070fe2d670a5a8d1057a8270f125ef7676d29ea900f496/zstandard-0.25.0-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:6a573a35693e03cf1d67799fd01b50ff578515a8aeadd4595d2a7fa9f3ec002a", size = 5451652, upload-time = "2025-09-14T22:17:04.979Z" }, + { url = "https://files.pythonhosted.org/packages/53/60/7be26e610767316c028a2cbedb9a3beabdbe33e2182c373f71a1c0b88f36/zstandard-0.25.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:5a56ba0db2d244117ed744dfa8f6f5b366e14148e00de44723413b2f3938a902", size = 5546993, upload-time = "2025-09-14T22:17:06.781Z" }, + { url = "https://files.pythonhosted.org/packages/85/c7/3483ad9ff0662623f3648479b0380d2de5510abf00990468c286c6b04017/zstandard-0.25.0-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:10ef2a79ab8e2974e2075fb984e5b9806c64134810fac21576f0668e7ea19f8f", size = 5046806, upload-time = "2025-09-14T22:17:08.415Z" }, + { url = "https://files.pythonhosted.org/packages/08/b3/206883dd25b8d1591a1caa44b54c2aad84badccf2f1de9e2d60a446f9a25/zstandard-0.25.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:aaf21ba8fb76d102b696781bddaa0954b782536446083ae3fdaa6f16b25a1c4b", size = 5576659, upload-time = "2025-09-14T22:17:10.164Z" }, + { url = "https://files.pythonhosted.org/packages/9d/31/76c0779101453e6c117b0ff22565865c54f48f8bd807df2b00c2c404b8e0/zstandard-0.25.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:1869da9571d5e94a85a5e8d57e4e8807b175c9e4a6294e3b66fa4efb074d90f6", size = 4953933, upload-time = "2025-09-14T22:17:11.857Z" }, + { url = "https://files.pythonhosted.org/packages/18/e1/97680c664a1bf9a247a280a053d98e251424af51f1b196c6d52f117c9720/zstandard-0.25.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:809c5bcb2c67cd0ed81e9229d227d4ca28f82d0f778fc5fea624a9def3963f91", size = 5268008, upload-time = "2025-09-14T22:17:13.627Z" }, + { url = "https://files.pythonhosted.org/packages/1e/73/316e4010de585ac798e154e88fd81bb16afc5c5cb1a72eeb16dd37e8024a/zstandard-0.25.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:f27662e4f7dbf9f9c12391cb37b4c4c3cb90ffbd3b1fb9284dadbbb8935fa708", size = 5433517, upload-time = "2025-09-14T22:17:16.103Z" }, + { url = "https://files.pythonhosted.org/packages/5b/60/dd0f8cfa8129c5a0ce3ea6b7f70be5b33d2618013a161e1ff26c2b39787c/zstandard-0.25.0-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:99c0c846e6e61718715a3c9437ccc625de26593fea60189567f0118dc9db7512", size = 5814292, upload-time = "2025-09-14T22:17:17.827Z" }, + { url = "https://files.pythonhosted.org/packages/fc/5f/75aafd4b9d11b5407b641b8e41a57864097663699f23e9ad4dbb91dc6bfe/zstandard-0.25.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:474d2596a2dbc241a556e965fb76002c1ce655445e4e3bf38e5477d413165ffa", size = 5360237, upload-time = "2025-09-14T22:17:19.954Z" }, + { url = "https://files.pythonhosted.org/packages/ff/8d/0309daffea4fcac7981021dbf21cdb2e3427a9e76bafbcdbdf5392ff99a4/zstandard-0.25.0-cp312-cp312-win32.whl", hash = "sha256:23ebc8f17a03133b4426bcc04aabd68f8236eb78c3760f12783385171b0fd8bd", size = 436922, upload-time = "2025-09-14T22:17:24.398Z" }, + { url = "https://files.pythonhosted.org/packages/79/3b/fa54d9015f945330510cb5d0b0501e8253c127cca7ebe8ba46a965df18c5/zstandard-0.25.0-cp312-cp312-win_amd64.whl", hash = "sha256:ffef5a74088f1e09947aecf91011136665152e0b4b359c42be3373897fb39b01", size = 506276, upload-time = "2025-09-14T22:17:21.429Z" }, + { url = "https://files.pythonhosted.org/packages/ea/6b/8b51697e5319b1f9ac71087b0af9a40d8a6288ff8025c36486e0c12abcc4/zstandard-0.25.0-cp312-cp312-win_arm64.whl", hash = "sha256:181eb40e0b6a29b3cd2849f825e0fa34397f649170673d385f3598ae17cca2e9", size = 462679, upload-time = "2025-09-14T22:17:23.147Z" }, + { url = "https://files.pythonhosted.org/packages/35/0b/8df9c4ad06af91d39e94fa96cc010a24ac4ef1378d3efab9223cc8593d40/zstandard-0.25.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:ec996f12524f88e151c339688c3897194821d7f03081ab35d31d1e12ec975e94", size = 795735, upload-time = "2025-09-14T22:17:26.042Z" }, + { url = "https://files.pythonhosted.org/packages/3f/06/9ae96a3e5dcfd119377ba33d4c42a7d89da1efabd5cb3e366b156c45ff4d/zstandard-0.25.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:a1a4ae2dec3993a32247995bdfe367fc3266da832d82f8438c8570f989753de1", size = 640440, upload-time = "2025-09-14T22:17:27.366Z" }, + { url = "https://files.pythonhosted.org/packages/d9/14/933d27204c2bd404229c69f445862454dcc101cd69ef8c6068f15aaec12c/zstandard-0.25.0-cp313-cp313-manylinux2010_i686.manylinux2014_i686.manylinux_2_12_i686.manylinux_2_17_i686.whl", hash = "sha256:e96594a5537722fdfb79951672a2a63aec5ebfb823e7560586f7484819f2a08f", size = 5343070, upload-time = "2025-09-14T22:17:28.896Z" }, + { url = "https://files.pythonhosted.org/packages/6d/db/ddb11011826ed7db9d0e485d13df79b58586bfdec56e5c84a928a9a78c1c/zstandard-0.25.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:bfc4e20784722098822e3eee42b8e576b379ed72cca4a7cb856ae733e62192ea", size = 5063001, upload-time = "2025-09-14T22:17:31.044Z" }, + { url = "https://files.pythonhosted.org/packages/db/00/87466ea3f99599d02a5238498b87bf84a6348290c19571051839ca943777/zstandard-0.25.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:457ed498fc58cdc12fc48f7950e02740d4f7ae9493dd4ab2168a47c93c31298e", size = 5394120, upload-time = "2025-09-14T22:17:32.711Z" }, + { url = "https://files.pythonhosted.org/packages/2b/95/fc5531d9c618a679a20ff6c29e2b3ef1d1f4ad66c5e161ae6ff847d102a9/zstandard-0.25.0-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:fd7a5004eb1980d3cefe26b2685bcb0b17989901a70a1040d1ac86f1d898c551", size = 5451230, upload-time = "2025-09-14T22:17:34.41Z" }, + { url = "https://files.pythonhosted.org/packages/63/4b/e3678b4e776db00f9f7b2fe58e547e8928ef32727d7a1ff01dea010f3f13/zstandard-0.25.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:8e735494da3db08694d26480f1493ad2cf86e99bdd53e8e9771b2752a5c0246a", size = 5547173, upload-time = "2025-09-14T22:17:36.084Z" }, + { url = "https://files.pythonhosted.org/packages/4e/d5/ba05ed95c6b8ec30bd468dfeab20589f2cf709b5c940483e31d991f2ca58/zstandard-0.25.0-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:3a39c94ad7866160a4a46d772e43311a743c316942037671beb264e395bdd611", size = 5046736, upload-time = "2025-09-14T22:17:37.891Z" }, + { url = "https://files.pythonhosted.org/packages/50/d5/870aa06b3a76c73eced65c044b92286a3c4e00554005ff51962deef28e28/zstandard-0.25.0-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:172de1f06947577d3a3005416977cce6168f2261284c02080e7ad0185faeced3", size = 5576368, upload-time = "2025-09-14T22:17:40.206Z" }, + { url = "https://files.pythonhosted.org/packages/5d/35/398dc2ffc89d304d59bc12f0fdd931b4ce455bddf7038a0a67733a25f550/zstandard-0.25.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:3c83b0188c852a47cd13ef3bf9209fb0a77fa5374958b8c53aaa699398c6bd7b", size = 4954022, upload-time = "2025-09-14T22:17:41.879Z" }, + { url = "https://files.pythonhosted.org/packages/9a/5c/36ba1e5507d56d2213202ec2b05e8541734af5f2ce378c5d1ceaf4d88dc4/zstandard-0.25.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:1673b7199bbe763365b81a4f3252b8e80f44c9e323fc42940dc8843bfeaf9851", size = 5267889, upload-time = "2025-09-14T22:17:43.577Z" }, + { url = "https://files.pythonhosted.org/packages/70/e8/2ec6b6fb7358b2ec0113ae202647ca7c0e9d15b61c005ae5225ad0995df5/zstandard-0.25.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:0be7622c37c183406f3dbf0cba104118eb16a4ea7359eeb5752f0794882fc250", size = 5433952, upload-time = "2025-09-14T22:17:45.271Z" }, + { url = "https://files.pythonhosted.org/packages/7b/01/b5f4d4dbc59ef193e870495c6f1275f5b2928e01ff5a81fecb22a06e22fb/zstandard-0.25.0-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:5f5e4c2a23ca271c218ac025bd7d635597048b366d6f31f420aaeb715239fc98", size = 5814054, upload-time = "2025-09-14T22:17:47.08Z" }, + { url = "https://files.pythonhosted.org/packages/b2/e5/fbd822d5c6f427cf158316d012c5a12f233473c2f9c5fe5ab1ae5d21f3d8/zstandard-0.25.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:4f187a0bb61b35119d1926aee039524d1f93aaf38a9916b8c4b78ac8514a0aaf", size = 5360113, upload-time = "2025-09-14T22:17:48.893Z" }, + { url = "https://files.pythonhosted.org/packages/8e/e0/69a553d2047f9a2c7347caa225bb3a63b6d7704ad74610cb7823baa08ed7/zstandard-0.25.0-cp313-cp313-win32.whl", hash = "sha256:7030defa83eef3e51ff26f0b7bfb229f0204b66fe18e04359ce3474ac33cbc09", size = 436936, upload-time = "2025-09-14T22:17:52.658Z" }, + { url = "https://files.pythonhosted.org/packages/d9/82/b9c06c870f3bd8767c201f1edbdf9e8dc34be5b0fbc5682c4f80fe948475/zstandard-0.25.0-cp313-cp313-win_amd64.whl", hash = "sha256:1f830a0dac88719af0ae43b8b2d6aef487d437036468ef3c2ea59c51f9d55fd5", size = 506232, upload-time = "2025-09-14T22:17:50.402Z" }, + { url = "https://files.pythonhosted.org/packages/d4/57/60c3c01243bb81d381c9916e2a6d9e149ab8627c0c7d7abb2d73384b3c0c/zstandard-0.25.0-cp313-cp313-win_arm64.whl", hash = "sha256:85304a43f4d513f5464ceb938aa02c1e78c2943b29f44a750b48b25ac999a049", size = 462671, upload-time = "2025-09-14T22:17:51.533Z" }, + { url = "https://files.pythonhosted.org/packages/3d/5c/f8923b595b55fe49e30612987ad8bf053aef555c14f05bb659dd5dbe3e8a/zstandard-0.25.0-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:e29f0cf06974c899b2c188ef7f783607dbef36da4c242eb6c82dcd8b512855e3", size = 795887, upload-time = "2025-09-14T22:17:54.198Z" }, + { url = "https://files.pythonhosted.org/packages/8d/09/d0a2a14fc3439c5f874042dca72a79c70a532090b7ba0003be73fee37ae2/zstandard-0.25.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:05df5136bc5a011f33cd25bc9f506e7426c0c9b3f9954f056831ce68f3b6689f", size = 640658, upload-time = "2025-09-14T22:17:55.423Z" }, + { url = "https://files.pythonhosted.org/packages/5d/7c/8b6b71b1ddd517f68ffb55e10834388d4f793c49c6b83effaaa05785b0b4/zstandard-0.25.0-cp314-cp314-manylinux2010_i686.manylinux_2_12_i686.manylinux_2_28_i686.whl", hash = "sha256:f604efd28f239cc21b3adb53eb061e2a205dc164be408e553b41ba2ffe0ca15c", size = 5379849, upload-time = "2025-09-14T22:17:57.372Z" }, + { url = "https://files.pythonhosted.org/packages/a4/86/a48e56320d0a17189ab7a42645387334fba2200e904ee47fc5a26c1fd8ca/zstandard-0.25.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:223415140608d0f0da010499eaa8ccdb9af210a543fac54bce15babbcfc78439", size = 5058095, upload-time = "2025-09-14T22:17:59.498Z" }, + { url = "https://files.pythonhosted.org/packages/f8/ad/eb659984ee2c0a779f9d06dbfe45e2dc39d99ff40a319895df2d3d9a48e5/zstandard-0.25.0-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:2e54296a283f3ab5a26fc9b8b5d4978ea0532f37b231644f367aa588930aa043", size = 5551751, upload-time = "2025-09-14T22:18:01.618Z" }, + { url = "https://files.pythonhosted.org/packages/61/b3/b637faea43677eb7bd42ab204dfb7053bd5c4582bfe6b1baefa80ac0c47b/zstandard-0.25.0-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:ca54090275939dc8ec5dea2d2afb400e0f83444b2fc24e07df7fdef677110859", size = 6364818, upload-time = "2025-09-14T22:18:03.769Z" }, + { url = "https://files.pythonhosted.org/packages/31/dc/cc50210e11e465c975462439a492516a73300ab8caa8f5e0902544fd748b/zstandard-0.25.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e09bb6252b6476d8d56100e8147b803befa9a12cea144bbe629dd508800d1ad0", size = 5560402, upload-time = "2025-09-14T22:18:05.954Z" }, + { url = "https://files.pythonhosted.org/packages/c9/ae/56523ae9c142f0c08efd5e868a6da613ae76614eca1305259c3bf6a0ed43/zstandard-0.25.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:a9ec8c642d1ec73287ae3e726792dd86c96f5681eb8df274a757bf62b750eae7", size = 4955108, upload-time = "2025-09-14T22:18:07.68Z" }, + { url = "https://files.pythonhosted.org/packages/98/cf/c899f2d6df0840d5e384cf4c4121458c72802e8bda19691f3b16619f51e9/zstandard-0.25.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:a4089a10e598eae6393756b036e0f419e8c1d60f44a831520f9af41c14216cf2", size = 5269248, upload-time = "2025-09-14T22:18:09.753Z" }, + { url = "https://files.pythonhosted.org/packages/1b/c0/59e912a531d91e1c192d3085fc0f6fb2852753c301a812d856d857ea03c6/zstandard-0.25.0-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:f67e8f1a324a900e75b5e28ffb152bcac9fbed1cc7b43f99cd90f395c4375344", size = 5430330, upload-time = "2025-09-14T22:18:11.966Z" }, + { url = "https://files.pythonhosted.org/packages/a0/1d/7e31db1240de2df22a58e2ea9a93fc6e38cc29353e660c0272b6735d6669/zstandard-0.25.0-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:9654dbc012d8b06fc3d19cc825af3f7bf8ae242226df5f83936cb39f5fdc846c", size = 5811123, upload-time = "2025-09-14T22:18:13.907Z" }, + { url = "https://files.pythonhosted.org/packages/f6/49/fac46df5ad353d50535e118d6983069df68ca5908d4d65b8c466150a4ff1/zstandard-0.25.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:4203ce3b31aec23012d3a4cf4a2ed64d12fea5269c49aed5e4c3611b938e4088", size = 5359591, upload-time = "2025-09-14T22:18:16.465Z" }, + { url = "https://files.pythonhosted.org/packages/c2/38/f249a2050ad1eea0bb364046153942e34abba95dd5520af199aed86fbb49/zstandard-0.25.0-cp314-cp314-win32.whl", hash = "sha256:da469dc041701583e34de852d8634703550348d5822e66a0c827d39b05365b12", size = 444513, upload-time = "2025-09-14T22:18:20.61Z" }, + { url = "https://files.pythonhosted.org/packages/3a/43/241f9615bcf8ba8903b3f0432da069e857fc4fd1783bd26183db53c4804b/zstandard-0.25.0-cp314-cp314-win_amd64.whl", hash = "sha256:c19bcdd826e95671065f8692b5a4aa95c52dc7a02a4c5a0cac46deb879a017a2", size = 516118, upload-time = "2025-09-14T22:18:17.849Z" }, + { url = "https://files.pythonhosted.org/packages/f0/ef/da163ce2450ed4febf6467d77ccb4cd52c4c30ab45624bad26ca0a27260c/zstandard-0.25.0-cp314-cp314-win_arm64.whl", hash = "sha256:d7541afd73985c630bafcd6338d2518ae96060075f9463d7dc14cfb33514383d", size = 476940, upload-time = "2025-09-14T22:18:19.088Z" }, +]