Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,10 +72,10 @@ not imply a separate runtime for that task or model.
| Shared Lightning training and validation hooks | [pl_utils/pl_model.py](egomimic/pl_utils/pl_model.py), [pl_utils/pl_data_utils.py](egomimic/pl_utils/pl_data_utils.py) |
| Episode loading, metadata, normalization, bounds and sampling | [rldb/zarr/zarr_dataset_multi.py](egomimic/rldb/zarr/zarr_dataset_multi.py) |
| Embodiment keymaps and coordinate frames | `egomimic/rldb/embodiment/{human,eva,yam}.py`; see [data guide](egomimic/rldb/AGENTS.md) |
| Generic bimanual ARC inputs used by E1 | [embodiment/bimanual_arc.py](egomimic/rldb/embodiment/bimanual_arc.py) |
| Generic bimanual time/ARC inputs | [embodiment/bimanual_arc.py](egomimic/rldb/embodiment/bimanual_arc.py) |
| Pose transforms and action encodings | [zarr/action_chunk_transforms.py](egomimic/rldb/zarr/action_chunk_transforms.py), [utils/pose_utils.py](egomimic/utils/pose_utils.py), [utils/action_encoding.py](egomimic/utils/action_encoding.py) |
| Shared HPT/PI robot metrics and videos | [eval/bimanual_cartesian_eval.py](egomimic/eval/bimanual_cartesian_eval.py), [eval/video.py](egomimic/eval/video.py); see [evaluation guide](egomimic/eval/AGENTS.md) |
| ARC and tempo evaluation | `egomimic/eval/{arc_bimanual_cartesian_eval,bimanual_tempo_eval,arc_metrics,e1_metrics}.py` |
| ARC evaluation | `egomimic/eval/{arc_bimanual_cartesian_eval,open_loop_sim,arc_metrics}.py`; [clock contract](docs/ARC_CLOCK_CONTRACT.md) |
| Dataset filters, task prompts, camera/frame choices and experiment parameters | `egomimic/hydra_configs/`; calibration matrices in `hydra_configs/calibration/` |
| Shared Eva/Yam collection, local graph rollout, Zarr replay and upload | [robot/AGENTS.md](egomimic/robot/AGENTS.md), [docs/YAM_RUNTIME.md](docs/YAM_RUNTIME.md) |
| Offline normalization export for any model | [scripts/data/precompute_norm_stats.py](scripts/data/precompute_norm_stats.py), or `trainHydra.py norm_stats_only=true` |
Expand Down
6 changes: 6 additions & 0 deletions docs/ARC_CHUNKING_MODES.md
Original file line number Diff line number Diff line change
Expand Up @@ -194,3 +194,9 @@ uses.
6. Validation videos (`video_overlay_mode: executed_chunk`) draw the executed
chunk and its ground truth at the replan frame and hold that pair while the
video advances through the chunk, then replan.
# Historical ARC modes

This document describes retired source contracts. For the supported Cartesian
API in this stack, use [ARC_CLOCK_CONTRACT.md](ARC_CLOCK_CONTRACT.md). Historical
checkpoints require their recorded revision; the mode names below are not
current selectable settings.
114 changes: 114 additions & 0 deletions docs/ARC_CLOCK_CONTRACT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
# Cartesian ARC: one codec, two clocks, two stream layouts

This contract replaces the historical Cartesian mean/wide/stacked/clock/E1
families. It builds on PR #263 and preserves the local first-row-delay
implementation. It does not change the separate planar or Libero codecs.

```yaml
_target_: egomimic.rldb.zarr.arc_length_tokenizer.TokenizeBimanualArcLengthCartesian
clock_mode: velocity # velocity | duration
streams: full_separation # full_separation | translation_rotation
resampled_vector_length: 100
min_distance_unit: 0.42 # meters, per arm
rotation_distance_unit: 1.7453292519943295 # radians, per arm (100 degrees)
dt: 0.03333333333333333
hold_window_frames: 400
gripper_stream_mode: events # events | distance; separate streams only
gripper_distance_unit: null # required positive for distance mode
```

## Shape and clock order

Geometry always occupies the first 14 columns:
`[left xyz, left YPR, left gripper, right xyz, right YPR, right gripper]`.

| streams | Shape | Remaining columns |
| --- | --- | --- |
| full_separation | M × 20 | T-left, T-right, R-left, R-right, G-left, G-right |
| translation_rotation | M × 18 | T-left, T-right, R-left, R-right |

Each arm and stream spends its own spatial budget. Supports from different
streams in the same row are not necessarily simultaneous.

In translation_rotation, grippers are sampled at their arm's translation
supports and interpolated with its translation clock. There is no independent
gripper termination racer or gripper distance budget. A fully stationary
translation stream retains its initial gripper target; use full_separation to
represent independent gripper movement.

## Timing rows

Row zero of every timing column represents that stream's initial delay.
Rows i > 0 represent the incoming interval from waypoint i-1 to waypoint i.
There is no repeated final padding clock.

- velocity: row zero is seconds; later rows are the distance between emitted
supports divided by the source interval's seconds. Units are m/s, rad/s,
and gripper units/s.
- duration: every row is `log(seconds + 0.001)`, including the initial delay.
The codec removes the log exactly once. `duration` does not mean absolute
timestamps or raw seconds.

Translation distances are Euclidean, rotation distances are SO(3) geodesic,
and gripper distances are absolute value differences. No velocity smoothing,
physical speed clamp, or decoder-duration floor is introduced.

An initial wait retains constant geometry before the first movement. Rotation
and distance-gripper target times use first arrival, so a final stationary
plateau does not stretch the last moving interval.

Full-window holds retain constant waypoints. Velocity interval rates are zero.
Duration intervals span the configured hold_window_frames × dt. Constant
streams do not terminate a chunk. Internal dwells are not explicitly allocated
duplicate supports; zero speed alone cannot encode a zero-distance dwell time.

Separate event grippers are held commands, not E1's fitted aperture ramps.
Command-change supports and endpoint padding retain the prior local semantics.

## Evaluation and rollout

Both use the same codec and `control_steps()`:

1. Keep an exact integer waypoint prefix, e.g. 30 of M100.
2. Decode every stream using its own start delay and incoming intervals.
3. Replan when the first moving stream reaches its prefix endpoint.
4. Ignore fully constant streams in this race.

Validation is an open-loop segment simulation against recorded observations.
Ground truth and videos cover the executed control steps. It does not apply a
second fixed-tick E1 regime. Distance-prefix token truncation is unsupported.

The robot adapter pads its fixed output tensor after the executed prefix and
reports the actual replan boundary via `execution_steps()`. No hardware is
started by constructing a decoder.

Malformed/nonfinite tokens fail. A moving stream with unusable timing cannot
win the termination race; an explicit caller horizon can bound a stalled clock.

## Migration and checkpoint safety

The old keyword options velocity_mode, velocity_layout and velocity_norm are
not accepted by the Cartesian codec. TokenizeBimanualArcLengthE1 and its
projection subclass are removed, along with their selectable ARC recipes and
codec-specific diagnostics. Baseline time-chunk retiming and unrelated model,
dataset, planar and Libero implementations remain separate.

Old serialized configs/checkpoints must use their original recorded source.
Do not reinterpret them by token width, rename their normalizers, or resume
training after changing this representation. Generate fresh normalization
statistics and a fresh run. The sparse row-zero delay may require existing
quantile widening (`norm_stats.widen_degenerate_quantiles: true`).

The first-delay six-stream refactor was compared numerically against its
preceding local implementation across 32 synthetic clock/gripper/hold cases,
including token, detokenizer, and termination parity. New regression tests cover
all four clock/stream combinations and validation/rollout agreement.

Verification for this migration: 536 focused CPU tests passed, including
normalization-cache rejection, baseline graph forward/backward, source adapters,
and planar ARC regressions. All 379 shipped Hydra YAML files composed.
The 32 six-stream comparisons used predecessor
`f364923a3e675106882823da17f840a9505ab972`, with token tolerance 1e-10,
decoded-trajectory tolerance 1e-9, and identical execution-step counts.
These checks do not establish GPU throughput, full pretrained-checkpoint
reload compatibility, or hardware safety. No cluster or robot process was run.
4 changes: 4 additions & 0 deletions docs/STATTEMPO.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
# Aidan stationery source and recipes

> Historical source record: this cleanup stack removes the E1 ARC codec and
> its recipes. Use [ARC_CLOCK_CONTRACT.md](ARC_CLOCK_CONTRACT.md) for new ARC
> work. Existing checkpoints require their original pinned source.

New ARC/stattempo source work starts from `aidan/stattempo-consolidated-main-20261011`
in `GaTech-RL2/EgoVerse-graph`. This is the main-based consolidation of
`aidan/stattempo-streams-ref-20261011` at `51cd2bf8bc52380757130b209ae1f686cbe42b6f`.
Expand Down
3 changes: 2 additions & 1 deletion egomimic/eval/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@ The root AGENTS.md applies here. Configure evaluators in
DTW, Fréchet, reverse-KL and coverage calculations.
- `arc_bimanual_cartesian_eval.py`, `arc_metrics.py`: ARC-specific decoding and
trajectory alignment layered on the shared Cartesian evaluator.
- `bimanual_tempo_eval.py`, `e1_metrics.py`: tempo/duration scoring.
- `open_loop_sim.py`, `distance_budget_dtw.py`: shared first-moving-stream
simulation and trajectory scoring with the canonical Cartesian ARC codec.
- `planar_action_eval.py`, `synthetic_trajectory_eval.py`: nonrobot evaluation.
- `checkpoint_loading.py`: strict graph checkpoint restoration and EMA selection.
- `token_diagnostics.py`, `latent_archive.py`: provider-owned token capture,
Expand Down
Loading
Loading