From 3d01b752a742a7ee4f0652aebe41e9d0a6fbcb35 Mon Sep 17 00:00:00 2001 From: AnikethCheluva Date: Fri, 23 Oct 2026 18:05:14 -0400 Subject: [PATCH] [Docs 2/2] map the feature-sized rebuild and current configuration contracts --- AGENTS.md | 26 +- README.md | 35 +- docs/ARC_CHUNKING_MODES.md | 14 + docs/ARC_CLOCK_CONTRACT.md | 186 ++++++++++ docs/ARC_OAT.md | 8 +- docs/ARC_REBUILD.md | 145 ++++++++ docs/ARC_STACK.md | 7 + docs/ARC_TSNE.md | 121 +++++++ docs/CANONICAL_CLUSTER_STACK.md | 69 ++++ docs/CONFIGURATION.md | 139 ++++++++ docs/CURRENT_FAMILY_PROFILES.md | 2 +- docs/DISTANCE_BUDGET_DTW.md | 60 ++++ docs/LIBERO_ARC_OAT_DP.md | 2 +- docs/LIBERO_ARC_OAT_HYBRID.md | 4 +- docs/LIBERO_ARC_REPLAY.md | 2 +- docs/PI05_GRAPH.md | 12 +- docs/STATTEMPO.md | 40 +++ docs/YAM_RUNTIME.md | 318 ++++++++++++++---- docs/action-flow-usocket-candidates.md | 6 +- docs/cluster-pipeline.md | 11 +- docs/config_cleanup_migration.json | 56 +++ docs/experiment_config_migration.json | 153 +++++++++ .../action-flow-merge-readiness.md | 5 +- docs/integration/CONFIG_DEDUPLICATION.md | 16 +- docs/integration/DATA_TOOLS.md | 12 +- docs/integration/MIGRATION.md | 25 +- docs/integration/RECIPE_PARITY.md | 2 +- docs/integration/THREE_SOURCE_MERGE.md | 2 +- egomimic/eval/AGENTS.md | 8 +- egomimic/rldb/AGENTS.md | 23 +- egomimic/robot/AGENTS.md | 31 +- training.md | 50 ++- 32 files changed, 1449 insertions(+), 141 deletions(-) create mode 100644 docs/ARC_CHUNKING_MODES.md create mode 100644 docs/ARC_CLOCK_CONTRACT.md create mode 100644 docs/ARC_REBUILD.md create mode 100644 docs/ARC_TSNE.md create mode 100644 docs/CANONICAL_CLUSTER_STACK.md create mode 100644 docs/CONFIGURATION.md create mode 100644 docs/DISTANCE_BUDGET_DTW.md create mode 100644 docs/STATTEMPO.md create mode 100644 docs/config_cleanup_migration.json create mode 100644 docs/experiment_config_migration.json diff --git a/AGENTS.md b/AGENTS.md index d3f50947c8..5d962066ef 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,7 +1,7 @@ # Repo Agent Rules -See [the verification policy](docs/VERIFICATION_POLICY.md): keep tests and -verification fixtures outside this repository; never commit them. +See [the verification policy](docs/VERIFICATION_POLICY.md): do not commit tests, +fixtures or local verification scripts. ## Shell / Command Execution To run commands in the interactive shell, source `emimic/bin/activate` when it @@ -39,6 +39,14 @@ whole scratch/project root. Candidate inventory alone does not authorize deletio Publish source with Graphite `gt submit` using the user's authorized GitHub account, preserving branch parents. Do not merge without an explicit user request. +## Aidan stationery source + +For current ARC work, use the feature-sized rebuild rooted on main. +See [docs/ARC_REBUILD.md](docs/ARC_REBUILD.md); #263–#279 remain reference history. +Read [docs/STATTEMPO.md](docs/STATTEMPO.md) and the experiment README for the +retained task recipes and source-preservation rules. Older branches are +historical run sources. Preserve existing jobs' recorded commits. + ## Current PushT recipes Read the current selection section in `docs/cluster-pipeline.md` before choosing @@ -68,10 +76,10 @@ not imply a separate runtime for that task or model. | Shared Lightning training and validation hooks | [pl_utils/pl_model.py](egomimic/pl_utils/pl_model.py), [pl_utils/pl_data_utils.py](egomimic/pl_utils/pl_data_utils.py) | | Episode loading, metadata, normalization, bounds and sampling | [rldb/zarr/zarr_dataset_multi.py](egomimic/rldb/zarr/zarr_dataset_multi.py) | | Embodiment keymaps and coordinate frames | `egomimic/rldb/embodiment/{human,eva,yam}.py`; see [data guide](egomimic/rldb/AGENTS.md) | -| Generic bimanual ARC inputs used by E1 | [embodiment/bimanual_arc.py](egomimic/rldb/embodiment/bimanual_arc.py) | +| Generic bimanual time/ARC inputs | [embodiment/bimanual_arc.py](egomimic/rldb/embodiment/bimanual_arc.py) | | Pose transforms and action encodings | [zarr/action_chunk_transforms.py](egomimic/rldb/zarr/action_chunk_transforms.py), [utils/pose_utils.py](egomimic/utils/pose_utils.py), [utils/action_encoding.py](egomimic/utils/action_encoding.py) | | Shared HPT/PI robot metrics and videos | [eval/bimanual_cartesian_eval.py](egomimic/eval/bimanual_cartesian_eval.py), [eval/video.py](egomimic/eval/video.py); see [evaluation guide](egomimic/eval/AGENTS.md) | -| ARC and tempo evaluation | `egomimic/eval/{arc_bimanual_cartesian_eval,bimanual_tempo_eval,arc_metrics,e1_metrics}.py` | +| ARC evaluation | `egomimic/eval/{arc_bimanual_cartesian_eval,open_loop_sim,arc_metrics}.py`; [clock contract](docs/ARC_CLOCK_CONTRACT.md) | | Dataset filters, task prompts, camera/frame choices and experiment parameters | `egomimic/hydra_configs/`; calibration matrices in `hydra_configs/calibration/` | | Shared Eva/Yam collection, local graph rollout, Zarr replay and upload | [robot/AGENTS.md](egomimic/robot/AGENTS.md), [docs/YAM_RUNTIME.md](docs/YAM_RUNTIME.md) | | Offline normalization export for any model | [scripts/data/precompute_norm_stats.py](scripts/data/precompute_norm_stats.py), or `trainHydra.py norm_stats_only=true` | @@ -100,14 +108,12 @@ rg -n 'class BimanualCartesianEval|class PI05Stage|class MultiDataset' egomimic - Evaluators expose `data_requirements()` and `trainer_overrides()`. Complete episode requests must reject partial loaders or validation-loop limits before model construction. See [the integration contract](docs/GENERIC_PIPELINE_CONTRACT.md). -- `egomimic/campaigns/pi05/`, `e1_fold.py`, and `pi05_graph_eval.py` retain old +- `egomimic/campaigns/pi05/` and `pi05_graph_eval.py` retain old import names. Their implementations now live at the canonical paths above. Use current YAML examples when migrating older serialized configuration APIs. -- Do not commit tests, test fixtures, or local verification scripts. Keep any - regression checks outside the repository and report their results in the PR. - Verify normalization, pose conventions and checkpoint loading with local CPU - checks before a cluster run; hardware and pretrained-model validation remain - separate. Historical test sources remain available in Git history. +- Keep regression checks and fixtures outside the repository; never commit tests. + Verify geometry, normalization and checkpoint behavior locally and report results. + Hardware and pretrained-model validation remain separate. ## Preserve work and verify the target diff --git a/README.md b/README.md index 2abe9728ac..2cd4fab0c3 100644 --- a/README.md +++ b/README.md @@ -3,9 +3,15 @@ This repository contains EgoVerse data processing plus a dependency-aware Pipeline stack for training and evaluation. +For the current review stack and retained/removed features, see the +[feature-sized ARC rebuild](docs/ARC_REBUILD.md). Tests and local verification +scripts are not shipped; see the [verification policy](docs/VERIFICATION_POLICY.md). + For cluster training and maintenance, start with [Pipeline and cluster operations](docs/cluster-pipeline.md). It distinguishes current source, historical run checkouts, and shared data/runtime dependencies. +For current task recipes and the data/model/evaluator configuration layout, +start with the [configuration guide](docs/CONFIGURATION.md). For the integration branch, start with the [migration guide](docs/integration/MIGRATION.md) for the locked environment, retained HPT/PI recipes, data tools, and checkpoint rules. [Integration status](docs/integration/PLAN.md) records the draft stack and @@ -117,6 +123,11 @@ To programatically view the SQL table of all episodes + metadata see [``sql_tuto #### Interactive Dataset Browser +For data-config → modern multistream ARC tokenization → paired t-SNE, use the +[offline ARC representation explorer](docs/ARC_TSNE.md). It exports self-contained +HTML with non-ARC time baselines, optional clock removal, linked anchor selection, +source trajectories, and configuration provenance. No checkpoint is required. + `latent_inspector.py` also ships a local web app for browsing a **folder of per-episode zarrs** — scrub any episode, overlay the recorded actions (cartesian trajectory / orientation axes / MANO keypoints), and toggle language annotations. Frames are rendered server-side using each episode's `zarr.json` camera intrinsics, so it works for any embodiment (the overlay is drawn by that episode's embodiment class, e.g. `Human`/`Eva`; human poses are projected in the head frame). ```bash @@ -141,14 +152,22 @@ python egomimic/scripts/data_download/sync_s3.py \ ``` ### Training -Select a complete Pipeline experiment; the root config deliberately has no -implicit model, data, or evaluator: -``` bash -python egomimic/trainHydra.py --config-name=train_zarr_cartesian \ - model= data= \ - evaluator= -``` -For full instructions on training see [``training.md``](./training.md) +Select a complete task recipe. Each public real experiment explicitly links its +model, data, evaluator, trainer, callbacks and logger. Inspect it without +starting training: + +```bash +python egomimic/trainHydra.py \ + +experiment=arc_real/stationary_velocity --cfg job --resolve +``` + +The [experiment menu](egomimic/hydra_configs/experiment/README.md) lists +`baseline`, `arc_vel` and `arc_dur` for each real task. A cluster launch adds +`-m hydra/launcher=` after resource/data preflight; +machine-specific paths and credentials must be configured. Advanced direct +`model=... data=... evaluator=...` composition remains supported; the root +config deliberately does not select those components implicitly. +See [training.md](./training.md) for the graph and launch workflow. ### Converting your own data See [``embodiment_tutorial.ipynb``](./egomimic/scripts/tutorials/embodiment_tutorial.ipynb) as reference to write a conversion script for your own data. diff --git a/docs/ARC_CHUNKING_MODES.md b/docs/ARC_CHUNKING_MODES.md new file mode 100644 index 0000000000..18da225b83 --- /dev/null +++ b/docs/ARC_CHUNKING_MODES.md @@ -0,0 +1,14 @@ +# Cartesian ARC stream selection + +The former race, joint-distance and multistream selectors have been removed. +Cartesian ARC always uses independent stream supports and clocks, with +first-moving-stream replanning. Choose only the clock representation and stream +layout described in [ARC_CLOCK_CONTRACT.md](ARC_CLOCK_CONTRACT.md). + +The supported layouts are full separation, translation/rotation, +translation-only and rotation-only. Projection is maintained on an optional +separate review branch, not in this stack's default source. + +Historical checkpoints require their recorded Git source and configuration. +The historical mode documentation remains available in Git history; its timing, +normalization and evaluation contracts are not interchangeable with arrival-v2. diff --git a/docs/ARC_CLOCK_CONTRACT.md b/docs/ARC_CLOCK_CONTRACT.md new file mode 100644 index 0000000000..6d77ef00d3 --- /dev/null +++ b/docs/ARC_CLOCK_CONTRACT.md @@ -0,0 +1,186 @@ +# Cartesian ARC: explicit-current-pose arrival clocks + +This is the current Cartesian contract. It is a new representation relative to +the former first-delay token: row zero now describes the current measured pose +to the first future target. No row has a special start-delay meaning. +Separate planar and Libero codecs are unchanged. + +Select a complete experiment from the +[experiment menu](../egomimic/hydra_configs/experiment/README.md). The example +below describes the codec API, not a universal experiment's D/R/M settings: + +```yaml +_target_: egomimic.rldb.zarr.arc_length_tokenizer.TokenizeBimanualArcLengthCartesian +initial_action_key: arc.current_action +clock_mode: velocity # velocity | duration +streams: full_separation # full_separation | translation_rotation | translation | rotation +resampled_vector_length: 100 +min_distance_unit: 0.42 # meters, per arm +rotation_distance_unit: 1.7453292519943295 # radians, per arm (100 degrees) +dt: 0.03333333333333333 +gripper_stream_mode: events # events | distance; separate streams only +gripper_distance_unit: null # required positive for distance mode +``` + +## Current state and future source + +The transform requires two distinct inputs: + +- `arc.current_action`: the finite 14D measured current pose at time zero, + transformed into exactly the same coordinate frame, YPR convention and + gripper convention as the action geometry. +- `actions_cartesian`: future rows beginning at source index `idx+1`. + Raw row i arrives at `(i+1) * dt`; the current observation row is not + duplicated into this window. + +The current pose is supplied separately to tokenization and every decoding +entrypoint. It must never be guessed from token row zero or replaced by a zero +pose except in an explicitly constructed synthetic example whose current pose +really is zero. Timing uses the configured control-grid dt, not irregular +measured timestamps. + +## Geometry, supports and clock order + +Geometry occupies the first 14 columns: +`[left xyz, left YPR, left gripper, right xyz, right YPR, right gripper]`. + +| streams | Shape | Timing columns | +| --- | --- | --- | +| full_separation | M × 20 | T-left, T-right, R-left, R-right, G-left, G-right | +| translation_rotation | M × 18 | T-left, T-right, R-left, R-right | +| translation | M × 16 | T-left, T-right | +| rotation | M × 16 | R-left, R-right | + +Translation-only samples XYZ, YPR and gripper on the translation supports. +Rotation-only samples all three on the rotation supports. In +translation_rotation, the gripper follows translation. Full separation gives +each component its own supports and clock. A token row therefore need not +represent simultaneous source times across separate streams. + +D and R bound cumulative travel **within the supplied future trajectory**. +The measured-current-to-first-future-target segment is included in incoming +rates and decoding, but is not retroactively included in those support budgets. +Anchor-inclusive emitted travel can exceed the source budget by that incoming +segment. Translation uses Euclidean distances; rotation uses SO(3) geodesic +distances; gripper distance uses absolute command differences. + +Exactly zero-progress geometry uses supports distributed over the actual +source indices. This retains the real available time span without inventing +movement or extending the trajectory to a configured hold duration. + +## The same timing meaning in every row + +Let q[-1] be the measured current pose, t[-1]=0, and q[i] the emitted future +waypoint with source arrival time t[i]. + +### Velocity + +Every row, including row zero, stores the incoming segment speed: + +```text +v[i] = distance(q[i-1], q[i]) / (t[i] - t[i-1]) +``` + +Thus row zero is measured-current-to-first-target speed, not seconds and not +a start delay. The numerator is computed between emitted supports, matching +the segment reconstructed by the decoder. Units are m/s, rad/s or gripper +units/s. Tiny zero intervals use numerical guards. Rates are not smoothed or +physically speed-clamped; decode uses the rate magnitude. + +Normalization provenance identifies this timing layout as `incoming_rate_v2`. + +### Duration + +Every row stores its raw absolute arrival time from the current observation: + +```text +clock[i] = t[i] +``` + +There is no logarithm, exponentiation, or special row-zero delay. For future +targets at 0.1, 0.2 and 0.3 seconds, the token stores `[0.1, 0.2, 0.3]`. +The decoder derives incoming intervals with `diff([0, corrected_times])`. + +Predicted times are projected to a non-decreasing absolute clock by isotonic +regression. Translation has a 5 m/s maximum-speed bound that includes the +current-pose-to-row-zero segment: + +```text +fastest[i] = sum(incoming_distance[0:i+1]) / 5 +corrected_times = fastest + max(isotonic(predicted_times - fastest), 0) +``` + +Rotation and gripper clocks receive monotone projection without a translation +speed bound. No arbitrary minimum chunk duration is added. +Normalization provenance uses `absolute_arrival_seconds_v2`. + +## Pauses and grippers + +There is no motion-start-delay extractor, synthetic 400-frame hold clock, +or dedicated internal-pause support allocation. + +Duration can preserve a stationary span when it is represented in the emitted +support times. Fully static duration tokens retain the actual source span; +they are not extended to 400 frames. Velocity is zero on a zero-distance +interval and cannot encode how long that dwell lasted. This includes terminal +gripper dwells and all-static chunks. + +For separate event grippers, supports remain initial future command, command +changes, source endpoint and repeated endpoint padding. If there are more +events than M supports, they are downsampled. Reconstruction is step/hold +command behavior, not fitted aperture ramps. Distance grippers use their +independent cumulative-travel budget. + +Single-stream velocity has the same limitation: if the selected component +does not move but another component does, that selected component's zero rate +cannot encode the other component's timing. Use a separated clock layout or +duration when that timing must be represented. + +## Decode and execution boundary + +Both evaluation and rollout provide the measured current pose explicitly: + +```python +codec.control_steps(token_prefix, initial_action=current_pose, max_steps=horizon) +codec.detokenize(token_prefix, horizon, initial_action=current_pose) +``` + +The codec starts at the measured pose at t=0 and emits future controls at +`dt, 2*dt, ..., horizon*dt`. XYZ uses linear interpolation, rotations use +SO(3) interpolation, and event grippers use held commands. + +For first-stream execution: + +1. Keep an exact integer prefix of future waypoint rows. +2. Decode all incoming intervals, including current-to-row-zero. +3. Replan when the first moving stream finishes that prefix. +4. Exclude constant streams from the race while another stream moves. + +A fully static velocity token contains no duration. `control_steps` therefore +requires the caller's positive `max_steps` and returns that explicit bound. +For fully static duration tokens it uses the earliest final absolute stream +arrival. A moving PW stream with unusable rates cannot win the termination +race; the explicit caller horizon can bound it. Nonfinite tokens or anchors +fail rather than being repaired silently. + +`clock_token_seconds` requires the same explicit anchor when converting +duration columns. Sixteen-column helpers also require explicit `streams`; +shape alone cannot distinguish translation-only from rotation-only. + +## Migration and checkpoint safety + +The former mean/wide/stacked/log-clock/E1 APIs are not supported. +Projection is outside the main contract. `hold_window_frames`, +`projection_offset_m`, `velocity_mode`, `velocity_layout` and +`velocity_norm` are not codec constructor options. + +First-delay checkpoints and normalization statistics are incompatible with +these arrival clocks despite matching tensor shapes. Preserve their recorded +source for historical reloads; do not infer the layout from token width or +resume an old run after this change. New runs require fresh normalization +statistics and the new timing-layout provenance. + +Checks and fixtures remain outside the application repository under the +[verification policy](VERIFICATION_POLICY.md). Local numerical checks do not +establish GPU throughput, pretrained-checkpoint compatibility or hardware +safety. This source update does not start a cluster or robot process. diff --git a/docs/ARC_OAT.md b/docs/ARC_OAT.md index bb4c3d0e2a..6500a5df4b 100644 --- a/docs/ARC_OAT.md +++ b/docs/ARC_OAT.md @@ -450,18 +450,18 @@ examples use the measured LIBERO-10 selections in the [dated replay report](results/libero_arc_timed_replay_20260921.md): ```sh -python -m egomimic.trainHydra +experiment=oat/libero_oattok \ +python -m egomimic.trainHydra +experiment=arc_sim/oat/libero_oattok \ benchmark.suite=libero_10 benchmark.dataset=/datasets/arc_oat/libero_10.zarr \ hydra.run.dir=/runs/tokenizer -python -m egomimic.trainHydra +experiment=oat/libero_oatpolicy \ +python -m egomimic.trainHydra +experiment=arc_sim/oat/libero_oatpolicy \ benchmark.suite=libero_10 benchmark.dataset=/datasets/arc_oat/libero_10.zarr \ benchmark.tokenizer_checkpoint=/runs/tokenizer/checkpoints/last.ckpt \ hydra.run.dir=/runs/oat -python -m egomimic.trainHydra +experiment=oat/libero_arc_dur_policy \ +python -m egomimic.trainHydra +experiment=arc_sim/oat/libero_arc_dur_policy \ benchmark.suite=libero_10 benchmark.dataset=/datasets/arc_oat/libero_10.zarr \ benchmark.arc_waypoints=32 benchmark.arc_max_translation=0.8 \ benchmark.arc_max_rotation_degrees=192 hydra.run.dir=/runs/arc_dur -python -m egomimic.trainHydra +experiment=oat/libero_arc_stk_policy \ +python -m egomimic.trainHydra +experiment=arc_sim/oat/libero_arc_stk_policy \ benchmark.suite=libero_10 benchmark.dataset=/datasets/arc_oat/libero_10.zarr \ benchmark.arc_waypoints=32 benchmark.arc_max_translation=1.6 \ benchmark.arc_max_rotation_degrees=128 hydra.run.dir=/runs/arc_stk diff --git a/docs/ARC_REBUILD.md b/docs/ARC_REBUILD.md new file mode 100644 index 0000000000..14ef09d81e --- /dev/null +++ b/docs/ARC_REBUILD.md @@ -0,0 +1,145 @@ +# ARC arrival-clock rebuild + +Main review stack: [Graphite #311](https://app.graphite.com/github/pr/GaTech-RL2/EgoVerse-graph/311). +Optional projection extension: [#317](https://github.com/GaTech-RL2/EgoVerse-graph/pull/317); it is not an ancestor of the main stack tip. + +Base main remains `a525d73897bec64397048e7c8897fb886555567b`. Actual main, historical #263–#279, and existing training runs are untouched. This is a source review stack, not a cluster deployment or evidence of robot performance. + +## Review order + +The same PR identities are retained. Numeric branch prefixes are historical; +follow these group labels and PR parents. Evaluators now have their own **Eval** +section. Projection is no longer part of Cartesian core creation on this stack. + +### Preparation + +| Level / PR | Change | +| --- | --- | +| 1 / [#284](https://github.com/GaTech-RL2/EgoVerse-graph/pull/284) | remove committed tests and retain configuration audits | +| 2 / [#312](https://github.com/GaTech-RL2/EgoVerse-graph/pull/312) | Withdraw inherited Cartesian ARC and preserve baseline-only integrations | + +### ARC + +| Level / PR | Change | +| --- | --- | +| 3 / [#290](https://github.com/GaTech-RL2/EgoVerse-graph/pull/290) | share rotation travel and first-crossing interpolation primitives | +| 4 / [#297](https://github.com/GaTech-RL2/EgoVerse-graph/pull/297) | Create translation ARC with measured-proprio incoming waypoint rates | +| 5 / [#313](https://github.com/GaTech-RL2/EgoVerse-graph/pull/313) | Add independent rotation clocks and rotation-only supports | +| 6 / [#314](https://github.com/GaTech-RL2/EgoVerse-graph/pull/314) | Add independent event-based gripper clocks | +| 7 / [#315](https://github.com/GaTech-RL2/EgoVerse-graph/pull/315) | Add distance-resampled gripper clocks | +| 8 / [#316](https://github.com/GaTech-RL2/EgoVerse-graph/pull/316) | Add absolute arrival seconds and anchor-aware monotone decoding | +| 9 / [#318](https://github.com/GaTech-RL2/EgoVerse-graph/pull/318) | Add first-moving-stream termination without synthetic hold timers | + +### Pipeline + +| Level / PR | Change | +| --- | --- | +| 10 / [#281](https://github.com/GaTech-RL2/EgoVerse-graph/pull/281) | preserve native action windows before interpolation | +| 11 / [#287](https://github.com/GaTech-RL2/EgoVerse-graph/pull/287) | align cameras proprioception and actions on one control clock | +| 12 / [#288](https://github.com/GaTech-RL2/EgoVerse-graph/pull/288) | preserve measured ARC anchors and metric-frame observation metadata | +| 13 / [#289](https://github.com/GaTech-RL2/EgoVerse-graph/pull/289) | bind post-tokenization normalization to exact arrival contracts | +| 14 / [#291](https://github.com/GaTech-RL2/EgoVerse-graph/pull/291) | resolve independent distance-bounded next-frame source windows | +| 15 / [#292](https://github.com/GaTech-RL2/EgoVerse-graph/pull/292) | pass explicit native decoder context and execution prefixes | +| 16 / [#282](https://github.com/GaTech-RL2/EgoVerse-graph/pull/282) | accept existing token sequences without an extra axis | +| 17 / [#283](https://github.com/GaTech-RL2/EgoVerse-graph/pull/283) | add optional clean-sample bounds during integration | +| 18 / [#285](https://github.com/GaTech-RL2/EgoVerse-graph/pull/285) | make ResNet ImageNet normalization explicit and opt-in | +| 19 / [#286](https://github.com/GaTech-RL2/EgoVerse-graph/pull/286) | add ABC-DiT through shared graph observation and flow stages | +| 20 / [#293](https://github.com/GaTech-RL2/EgoVerse-graph/pull/293) | preserve W&B identity across automatic Lightning resumes | +| 21 / [#301](https://github.com/GaTech-RL2/EgoVerse-graph/pull/301) | replace campaign-named image stem with shared ResNet-MLP implementation | + +### Eval + +| Level / PR | Change | +| --- | --- | +| 22 / [#298](https://github.com/GaTech-RL2/EgoVerse-graph/pull/298) | score world-frame arrival-clock full-episode DTW | +| 23 / [#299](https://github.com/GaTech-RL2/EgoVerse-graph/pull/299) | evaluate next-frame open-loop chunks and videos from measured anchors | + +### Robot + +| Level / PR | Change | +| --- | --- | +| 24 / [#300](https://github.com/GaTech-RL2/EgoVerse-graph/pull/300) | decode measured-proprio incoming ARC for robot execution | +| 25 / [#294](https://github.com/GaTech-RL2/EgoVerse-graph/pull/294) | add bounded rollout IK recovery and selectable endpoint solvers | +| 26 / [#295](https://github.com/GaTech-RL2/EgoVerse-graph/pull/295) | record demonstration-compatible rollout episodes and tempo summaries | +| 27 / [#296](https://github.com/GaTech-RL2/EgoVerse-graph/pull/296) | add standalone camera alignment diagnostics | +| 28 / [#306](https://github.com/GaTech-RL2/EgoVerse-graph/pull/306) | connect bound controls recording and reset guides to the rollout dashboard | + +### Config + +| Level / PR | Change | +| --- | --- | +| 29 / [#302](https://github.com/GaTech-RL2/EgoVerse-graph/pull/302) | factor reusable model bases and HPT PI component groups | +| 30 / [#303](https://github.com/GaTech-RL2/EgoVerse-graph/pull/303) | organize simulation recipes under arc_sim and update launcher selectors | +| 31 / [#304](https://github.com/GaTech-RL2/EgoVerse-graph/pull/304) | integrate arrival-v2 source offsets and explicit experiment contracts | +| 32 / [#305](https://github.com/GaTech-RL2/EgoVerse-graph/pull/305) | remove remaining E1 campaign runtime and temporary withdrawal support | +| 33 / [#309](https://github.com/GaTech-RL2/EgoVerse-graph/pull/309) | update PI launch examples and integration matrix to canonical config paths | + +### Tools + +| Level / PR | Change | +| --- | --- | +| 34 / [#307](https://github.com/GaTech-RL2/EgoVerse-graph/pull/307) | add checkpoint sweeps and explicit validation-only logging | +| 35 / [#308](https://github.com/GaTech-RL2/EgoVerse-graph/pull/308) | add source-window measurements and paired ARC representation inspection | +| 36 / [#310](https://github.com/GaTech-RL2/EgoVerse-graph/pull/310) | package ARC inspection assets and lock optional TracIK dependency | + +### Docs + +| Level / PR | Change | +| --- | --- | +| 37 / [#280](https://github.com/GaTech-RL2/EgoVerse-graph/pull/280) | use the authorized GitHub account for Graphite publication | +| 38 / [#311](https://github.com/GaTech-RL2/EgoVerse-graph/pull/311) | map the feature-sized rebuild and current configuration contracts | + +## Behavioral changes in this revision + +- Cartesian ARC always uses independent streams. There are no selectable race, + cumulative/joint-distance or multistream algorithms. Cumulative geometric + length remains a necessary internal calculation, not a mode. +- The layouts are full separation (M×20), translation/rotation (M×18), + translation-only and rotation-only (M×16). Projection is optional elsewhere. +- Native ARC source rows start at the next recorded frame, not the observation + frame. Baseline indexing is unchanged. Native ARC stride must be one. +- Current measured pose is passed explicitly in the action frame. Row zero is + an incoming speed for PW, or an absolute arrival time for duration, just like + the later incoming intervals. Decoder queries are dt, 2dt, …, not time zero. +- There is no leading-delay extraction, synthetic 400-frame hold target, dwell + event allocation or duration floor. PW cannot encode the duration of exactly + zero-distance intervals; all-static PW uses the caller's explicit horizon. +- Normalization still runs after tokenization. Exact arrival-v2 contracts reject + old or missing normalization bindings; shapes alone do not establish compatibility. +- Eval and robot adapters supply the same measured native anchor. Context is + model-declared; no class-name or token-width inference supplies a starting pose. + +These are intentional algorithmic changes, not a numerical-parity rebuild of +#279. The earlier rebuild at `2cec3b844046c967f33d326298735d0a482076ee` +preserved that old behavior; its sources and verification remain in Git history. +New training requires fresh compatible statistics and experiment identities. + +## Scope and integration + +Preparation withdraws only Cartesian/bimanual ARC; independent planar/simulation +codecs remain. Six incremental codec snapshots build incoming-rate translation, +rotation layouts, event grippers, distance grippers, absolute clocks, and +first-stream termination. Unsupported options are absent at each stage. + +The configuration migration remains the activation boundary for complete +experiment menus and renamed runtime entry points. Intermediate helper layers +are reviewable components, not production launch candidates. New methods, +configs, normalization and rollout context must be deployed together. + +Event grippers still cover changes over the supplied source window, downsample +when supports exceed M and repeat the endpoint to fill M. No new event-count +budget is invented. Their zero-distance PW dwell-loss limitation remains; see +[clock contract](ARC_CLOCK_CONTRACT.md) for details and source-distance caps. + +## Canonical paths and checks + +- Cartesian codec: `egomimic/rldb/zarr/cartesian_arc.py` (stable import from `arc_length_tokenizer.py`). +- Episode evaluation: `egomimic/eval/open_loop_stream.py` (stable `open_loop_sim.py`). +- Robot decoding: `egomimic/robot/stream_decoder.py` (stable `arc_decoder.py`). +- [Configuration](CONFIGURATION.md), [clock contract](ARC_CLOCK_CONTRACT.md), + [DTW contract](DISTANCE_BUDGET_DTW.md), [no committed tests](VERIFICATION_POLICY.md). + +Verification scripts/fixtures remain outside the application repository. CPU +oracle, frame/context, normalization, source-offset, config and staged-codec +checks do not establish hardware safety or real-robot success. No training job +or physical robot is started by this rebuild. diff --git a/docs/ARC_STACK.md b/docs/ARC_STACK.md index 0cecdc7117..a555f407b5 100644 --- a/docs/ARC_STACK.md +++ b/docs/ARC_STACK.md @@ -1,5 +1,12 @@ # ARC consolidation stack +For the current feature-sized rebuild, see [ARC_REBUILD.md](ARC_REBUILD.md). + +> Historical reference, not a current API or launch menu. Removed E1/legacy +> codecs, configs and launchers described below require their recorded source. +> Current usage: [clock contract](ARC_CLOCK_CONTRACT.md) and +> [experiment menu](../egomimic/hydra_configs/experiment/README.md). + This is a dated architecture snapshot. It does not select the current PushT recipe or live launcher; start with `docs/cluster-pipeline.md`. diff --git a/docs/ARC_TSNE.md b/docs/ARC_TSNE.md new file mode 100644 index 0000000000..c3791fe2e9 --- /dev/null +++ b/docs/ARC_TSNE.md @@ -0,0 +1,121 @@ +# ARC representation explorer + +Generate an offline HTML t-SNE report directly from a trusted data config. No +checkpoint, training job, W&B session, web server, CDN, or GPU is required. +These are **native action/token feature projections, not learned model latents**. + +From the repository root, in the project environment: + +```sh +export EGOVERSE_ABC_DATASET_DIR=/path/to/local/episodes +python -m egomimic.scripts.data_visualization.arc_tsne \ + --data-config egomimic/hydra_configs/visualization/arc_tsne_yam.yaml \ + --max-episodes 12 --samples-per-episode 64 \ + --clock both --output output/arc-tsne +``` + +Open `output/arc-tsne/index.html` directly. The HTML embeds its complete report; +copying that one file is sufficient. `report.json` contains coordinates, anchor +metadata and provenance; `features.npz` contains the exact pre-scaling feature +matrices in the same row order. Existing output directories are never overwritten. + +## Configs and sampling + +Accepts a standalone data YAML with `train_datasets` / `valid_datasets`, or a +full Hydra config with those fields under `data`. Hydra defaults are composed +relative to the supplied config directory; use a top-level config when its +defaults reference other groups. External interpolations (e.g. `abc.*`) must +come from that full config or repeatable `--override` arguments. Only the +selected resolver/filter is instantiated: no model or data module setup. + +- `--dataset yam_bimanual`: required when the selected split has multiple datasets. +- `--split train|valid`: preserves configured validation fraction and split seed. + The supplied RL2-oriented example uses **5% validation**; existing configs are + never silently changed. Episode-count/hash pins are checked when configured. +- `--representations both|arc|baseline`: baseline-only also accepts ordinary + Cartesian configs. ARC requires the current training transform + `TokenizeBimanualArcLengthCartesian`; legacy E1 codecs are deliberately rejected. +- `--clock include|exclude|both`: removes **all timing rows/channels before** + scaling, PCA, and t-SNE. Handles mean stacked `(M+1,14)`, granular stacked + `(2M,14)`, and wide `(M,28)` layouts. Explicit clock removal is not a guarantee + of time invariance: pose/gripper geometry and endpoint coverage can retain + timing correlations. +- `--baseline-rows 100`: a separate fixed-time read of the same anchor, using + the configured pre-tokenization geometric transforms and repeat-last padding + at episode tails. Action key horizons are changed on a shallow leaf copy, not + the ARC leaf. It is not ARC detokenization. The comparison + holds anchors and geometric transforms fixed, not the temporal coverage: + ARC is distance/rotation capped, baseline is time capped. `source_rows` exposes + the ARC source-window length in the report. Configs with explicit interpolation + before tokenization retain that interpolation; use the supplied YAM recipe + for native-cadence comparisons. +- Sampling is deterministic, uniform without replacement over episodes and + then frames (defaults: 12 episodes × 64 anchors). Short episodes contribute + fewer points. No point is silently substituted after a loader fallback. + Invalid/nonfinite/sentinel samples cause an error instead of selective dropping. +- Multistream/rotation/D/M/dt come from the actual training tokenizer. The example + explicitly selects multistream, D=0.4m, R=24°, M=100, per-waypoint wide clocks. + Existing configs retain their mode rather than being silently migrated. + +Resolver discovery may still inspect the complete local inventory to preserve +the split; sampling limits apply to tokenization, not inventory discovery. +There is no recursive scan or parallel loader. Non-local resolvers are rejected +unless `--allow-remote-resolution` is explicitly passed. That option invokes +the configured resolver and **can query SQL or download episodes**, so use a +local config for lightweight inspection. A partial local cache produces a split +of that local inventory, not necessarily the original full training split; +use matching pinned inventories for exact training-split parity. Do not relabel +a control-rate resolver as a plain local resolver: doing so changes cadence. + +Only use trusted configs: Hydra targets and filter lambdas execute Python. +Reports include episode/task identifiers, paths, and selected resolved config; +review these before sharing outside your team. + +## Projection and HTML + +Each representation has its own fit: constant features removed, optional +per-feature standardization (`--scaling standard|none`), PCA to at most 50 +dimensions, then seeded 2D t-SNE. Perplexity is capped to `(N-1)/3` for small +samples; actual values, dimensions and KL divergence are recorded. Native +units without scaling can overweight some feature types. Euler angles retain +the training representation's wrap discontinuities. All-constant input fails +instead of showing invented structure. Computation uses one numerical thread. + +Linked selection highlights the same anchor across panels. Filter by episode, +color by episode/task/embodiment, select via canvas or keyboard-accessible point +controls/table, inspect source XY trajectories, or download coordinates. Filters +do not refit t-SNE. Source trajectories retain the configured coordinate frame; +they are not camera overlays. Images/video and checkpoint-learned latents remain +the responsibility of the existing Dash latent inspector. + +Do not compare axes or absolute distances between independent t-SNE fits. The +viewer is for exploratory neighborhoods, not a quantitative representation score. + +## Tests + +### Bounded SQL-inventory exports + +`arc_tsne_inventory` reads a configured SQL inventory in a read-only transaction, +applies the 5% split to the full matching inventory, then opens only sampled +episodes. The RL2 organizing recipes select exact `lab=rl2`, +`task=organize_stationary`, and human/YAM bimanual embodiment. They explicitly +preserve native source cadence before modern multistream tokenization and omit +all camera keys. Missing sampled episodes cause an error unless `--action-cache` +opts into sparse staging: only pose/action arrays and Zarr metadata are fetched, +one object at a time, with a default 64 MiB total download limit. Shared caches +are never overwritten; no missing episode is replaced with a different sample. + +```sh +python -m egomimic.scripts.data_visualization.arc_tsne_inventory \ + --data-config egomimic/hydra_configs/visualization/rl2_organize_human_bimanual.yaml \ + --max-episodes 12 --samples-per-episode 64 --output /path/to/new/report +``` + +Use `rl2_organize_yam_bimanual.yaml` for YAM. Both export baseline, ARC-clock and +ARC-no-clock panels. Reports retain the full split IDs, selected IDs, transfer +counts, and exact tokenizer configuration. Human gripper channels are zero-padded +because those recordings have no robot gripper signal. + +```sh +python -m pytest tests/test_arc_tsne.py tests/test_arc_chunking_modes.py -q +``` diff --git a/docs/CANONICAL_CLUSTER_STACK.md b/docs/CANONICAL_CLUSTER_STACK.md new file mode 100644 index 0000000000..2d9269957a --- /dev/null +++ b/docs/CANONICAL_CLUSTER_STACK.md @@ -0,0 +1,69 @@ +# Canonical cluster application stack + +## September 25 feature tip + +The canonical layers have been reconciled with `main` at +`161e3a0c40182ba003434d3adaeaa6cbd12107d8` using history-preserving merges. +Each canonical parent remains an ancestor of its child; published commits were +not rebased or force-pushed. The cluster overlays are siblings immediately above +`canonical/arc-chunking-modes`, with application trees identical to that tip. + +The stack below now continues above `canonical/run-resume` with: + +10. `canonical/distance-budget-global-dtw`: episode-distance rollout budgets and + bidirectional, GT-frame-balanced DTW. +11. `canonical/visual-bc-launch-readiness`: visual-only robot/human recipes and + corrected BC targets. +12. `canonical/open-loop-full-frame-video-wandb`: per-frame validation video, + fresh-run logging, and organized retained experiments. +13. `canonical/arc-chunking-modes`: race, multistream, and joint-distance + translation; separate hybrid R clocks; matching evaluation and cache + provenance. See `ARC_CHUNKING_MODES.md`. + +Pace, Sky2, ICE, and Lambda each retain a cluster-specific configuration overlay +above the shared application tip. Updating refs does not change active checkouts +or running jobs. Lambda's original launcher addition remains in the historical +application ancestry; this feature does not rewrite that published history. + +## Original canonical layers + +The shared application stack is rooted on GitHub `main`. Each branch represents +one user-visible feature; follow-up fixes stay on that feature branch. + +Bottom to top: + +1. `canonical/arc-data-runtime`: native 100-frame baseline windows, distance-aware + ARC source windows, and timed reconstruction. +2. `canonical/hpt-clock-heads`: separate clock heads and shared-stem mixtures. +3. `canonical/open-loop-evaluation`: executed-prefix metrics, per-frame capped + GT/prediction videos, joint-distance semantics, and trajectory comparisons. +4. `canonical/checkpoint-validation`: offline checkpoint sweeps and their logs. +5. `canonical/robot-bc-recipes`: RL2 stationery and ABC towels BC recipes. +6. `canonical/hybrid-arc`: independent translation/rotation caps and clocks with + per-waypoint velocity. +7. `canonical/human-bc-recipes`: human-bimanual MECKA folding-clothes recipes. +8. `canonical/visual-abc-multitask`: no-language ABC campaigns, 30 Hz sampling, + joint-distance calibration, and normalization (historically 20%; retained + recipes now default to 10%). +9. `canonical/run-resume`: stable W&B identities for resumed training, while + offline checkpoint evaluation retains a separate W&B identity. + +`cluster/pace`, `cluster/sky2`, `cluster/ice`, and `cluster/lambda` are sibling configuration +overlays above the same application tip. Do not put shared model, data, or +evaluator implementation changes in those overlays. + +The original cluster commits are preserved by the +`snapshots/cluster-sync-20260922/{pace,sky2,ice}` tags. Existing PR branches are +not deleted or force-pushed by this consolidation. + +Syncing a branch ref does not deploy code into an active job's checkout. Running +jobs remain pinned to their recorded source commits; promotion requires a clean +canonical checkout and the usual launch/cache/validation gates. ICE's existing +untracked files must not be overwritten. + +The `hpt180`/`hpt300` names are historical profile identifiers, not claims of +exact total parameter counts. The robot visual profiles instantiate roughly +232M/249M parameters, respectively; the bimanual human profile also has its own +stem/domain parameters. Architecture and checkpoint shapes are preserved by +this cleanup. See the experiment-directory README for the retained profile +dimensions and how to measure a composed model. diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md new file mode 100644 index 0000000000..a0e7cc4d6e --- /dev/null +++ b/docs/CONFIGURATION.md @@ -0,0 +1,139 @@ +# Configuration guide + +Start with a task experiment, then follow its Hydra `defaults` links. Folder +names organize recipes; Python `_target_` declarations identify implementations. +The generic pipeline does not choose behavior from a model or task name. + +```text +egomimic/hydra_configs/ +├── experiment/ +│ ├── arc_real/ +│ │ ├── stationary_{time,duration,velocity}.yaml +│ │ ├── towels_{time,duration,velocity}.yaml +│ │ └── earbuds_{time,duration,velocity}.yaml +│ ├── abc_arc/robot_bc/ retained ABC-lab recipes, including multitask +│ └── arc_sim/ PushT, LIBERO, OAT and synthetic recipes +├── data/ episode selection, transforms and loaders +├── model/ +│ ├── base/ shared model definitions and settings +│ ├── hpt/ HPT recipe leaves +│ ├── pi/ PI recipe leaves +│ └── … distinct non-HPT/PI model families +├── model_contract/ shared deployment and data contracts +├── evaluator/ +│ ├── eval_open_loop_sim.yaml +│ ├── eval_cartesian_*.yaml behavior/frame-specific metrics and videos +│ ├── _base/ inherited evaluator defaults +│ └── viz/ reusable renderers +└── hydra/launcher/ cluster allocation, separate from experiment +``` + +## Select and inspect one experiment + +The three real tasks each expose `_time`, `_velocity` and `_duration`: +nine standard entry points, preserving the current task pools and protocols. +Every leaf directly links model, data, +evaluator, trainer, callbacks and logger. The model owns the optimizer and +scheduler. Real recipes do not inherit other experiments: each contains its +task/run overrides and imports reusable definitions from the component groups. + +Activate the project environment, configure required data-cache/weight paths, +then inspect without running training: + +```bash +python egomimic/trainHydra.py \ + +experiment=arc_real/stationary_velocity --cfg job --resolve +``` + +The [experiment menu](../egomimic/hydra_configs/experiment/README.md) records +task pools and numeric defaults. The top-level experiment directories `abc/` +and `abc_arc/human_bc/`, and `arc_real/arcdur_abstime/`, are removed. Their old +commands are historical, not aliases for the retained task recipes. The +removed absolute-time recipe menu does not remove duration mode: +`arc_real/_duration` uses absolute waypoint seconds. + +For an authorized cluster launch, use that same experiment with `-m` and an +explicit approved `hydra/launcher=...` selection. Verify the resolved resource +request, effective global batch, pinned data and normalization before launch. +See [training.md](../training.md) and [cluster operations](cluster-pipeline.md). + +## What belongs in each layer + +| Layer | Owns | Does not imply | +| --- | --- | --- | +| `experiment/arc_real/_` | Complete component selection, task values and run schedule | Another task-specific Python runtime | +| `data/` | SQL filters or pinned episode lists, splits, transforms, loader behavior | The model architecture | +| `model/hpt/`, `model/pi/` | Model graph and family-specific recipe overrides | A model-specific evaluator implementation | +| `model/base/` | Identical inherited graphs/settings factored out of leaves | An experiment with data and evaluation selected | +| `model_contract/` | Reusable inference/deployment and preprocessing declarations | Permission to reinterpret a checkpoint | +| `evaluator/` | Metrics, frame reversion, execution protocol and artifacts | Training-time tokenization choices inferred from a model name | + +`model/base/` contains shared HPT flow, cotrain, keypoint and Qwen definitions, +PI defaults, and YAM HPT/diffusion settings. Distinct architectures and +contracts remain distinct; only equivalent definitions should be shared. +The Python networks stay in `egomimic/models/`, and configured graph stages +stay in `egomimic/pipeline/`. Moving YAML does not move or resize a network. + +### Direct component links; no shared experiment parents + +`experiment/_shared/` and the real experiment-parent chain are removed. Every +one of the nine `arc_real/` recipes selects its six components directly. Model +dimensions live in `model/base/real_*_settings`, training defaults in +`trainer/real_*`, and the checkpoint schedule in +`callbacks/real_yam`. Task identity, dataset selection, caps and run +overrides remain visible in the experiment itself. + +Canonical clock dimensions/provenance live in +`model_contract/arc_clock_settings`. YAM ARC experiments explicitly +select `model/hpt/yam_300m_arc` and the matching `data/abc_arc/*_arc` +adapter; the data adapter retains the original pinned pool and binds the +model-declared transforms through `data/_base/yam_arc`. + +Component inheritance still deduplicates model graphs and dataset lists. +There is no earbuds-to-towels-to-stationery experiment inheritance, and no +extra experiment file is needed to launch any real recipe. Simulation recipes +retain their existing parent structure under the renamed `arc_sim/` folder. + +## Evaluator selection + +| Selection | Purpose | +| --- | --- | +| `eval_open_loop_sim` | Recorded-episode execution simulation for real baseline and ARC recipes; metrics, DTW and executed-chunk videos | +| `eval_bimanual_cartesian` | Time-indexed chunk errors and overlays, without sequential execution/replanning | +| `eval_cartesian_chunk_metrics` | Metrics-only chunk comparison using the configured YAM batch budget; no renderer | +| `eval_cartesian_camera`, `eval_cartesian_wrist` | Cartesian pose metrics with the chosen frame reversion | +| `eval_cartesian_*_prompt` | Corresponding metrics with sampled-prompt overlays; `6d` variants use continuous-6D rotations | + +For example, the old `eval_hpt` selector becomes `eval_cartesian_camera`, and +the old root `eval_pi` becomes `eval_cartesian_camera_prompt`. A model name is +not sufficient to choose the right evaluator: frame, rotation encoding, +ground-truth key and execution protocol must match the data/model contract. +Renderer-only defaults live in `evaluator/viz/`; common evaluator definitions +live in `evaluator/_base/`. + +`eval_open_loop_sim.yaml` groups settings by execution, canonical codec, +complete-episode scoring and artifacts. Real baselines predict 100 actions, +execute the first 25 at 30 Hz, then replan against the next recorded +observation. ARC keeps a waypoint prefix and ends when the first moving stream +finishes. This is open-loop replay of recorded observations, not closed-loop +physics or robot evaluation. Ordered complete episodes and matching video/GT +coverage are part of its contract, not optional batch-sampling details. + +The [Cartesian clock contract](ARC_CLOCK_CONTRACT.md) defines velocity versus +absolute-seconds duration, four stream layouts and measured-proprio-to-waypoint arrival semantics. +Planar and LIBERO codecs have their own contracts; the removed Cartesian +legacy options must not be reintroduced into the open-loop YAML. + +## Provenance and migration + +Use the new selectors for new compositions. Keep the original source commit, +resolved config, normalization and checkpoint for an existing run. Do not +resume a different action representation under the same run identity or infer +decoder compatibility from M×20/M×18/M×16 shape alone. YAML organization does +not migrate trained weights. See [checkpoint rules](integration/CHECKPOINTS.md) +and [current real-task recipes](STATTEMPO.md). + +The exact old-to-new model, evaluator and data selector map is in +[`config_cleanup_migration.json`](config_cleanup_migration.json). Removed +experiment menus have no automatic replacement: choosing another task recipe +could change its data pool or training protocol. diff --git a/docs/CURRENT_FAMILY_PROFILES.md b/docs/CURRENT_FAMILY_PROFILES.md index b56772423d..12a7a1d0ba 100644 --- a/docs/CURRENT_FAMILY_PROFILES.md +++ b/docs/CURRENT_FAMILY_PROFILES.md @@ -61,7 +61,7 @@ selected emitted configs remain separate immutable task artifacts. ## PushShapes: explicit multiplier versus historical physical speed -AF selects `pusht/action_flow_cotrain_uc_multiplier_interpolation`; its typed +AF selects `arc_sim/pusht/action_flow_cotrain_uc_multiplier_interpolation`; its typed route requires fresh initialization, scalar raw dimensionless multiplier, no physical-speed reference, Euler50, batches32+32 and fixed EMA0.9978. The latest explicit PushT preference uses normal native per-source image processing, not diff --git a/docs/DISTANCE_BUDGET_DTW.md b/docs/DISTANCE_BUDGET_DTW.md new file mode 100644 index 0000000000..5dc698c902 --- /dev/null +++ b/docs/DISTANCE_BUDGET_DTW.md @@ -0,0 +1,60 @@ +# Recorded-episode open-loop DTW + +The current metric version is `arc_arrival_global_dtw_v7`. Older scores are +not directly comparable after changing target origin from frame t to t+1. + +## Shared execution for scores and videos + +For ARC, retain exactly M × execute_fraction waypoints (an integer, at least +two). Decode every stream from measured current proprio using its own clock. +Replan when the first moving stream exhausts that prefix, bounded only by the +remaining episode or an explicitly declared caller horizon. No selectable +race/joint-distance mode and no decoder-imposed hold duration remain. + +The recorded observation at t predicts targets t+1 onward. At each replan, +store its native same-frame `arc.current_action` and its EEF-to-world metric +matrices. The former is the decoder's initial state; the latter restores +predictions and GT to one world coordinate system. Neither is inferred from +predicted waypoint zero. Missing ARC initial states fail explicitly. + +The baseline retains same-index targets and its configured fixed control-step +prefix. This is offline recorded-observation segment evaluation—not a physical +closed-loop simulation. At each boundary the policy sees a new recorded +observation, not the physical consequence of its previous prediction. + +The paired MSE and DTW paths walk the same predicted ARC chunk boundaries. +Default executed-chunk video overlays hold the actual decoded chunk while the +recorded video advances, then replace it at the next replan. The optional +`arc_video_trajectory_cap_mode: distance` crops diagnostic overlays only; the +default `execution_horizon` shows the full executed prefix. + +## One global alignment + +Concatenate predictions and GT in world XYZ, then compute one endpoint-anchored, +monotone DTW over all six position coordinates. The path permits horizontal, +vertical and diagonal moves. Reporting averages matches within each GT +timestamp first, then averages GT timestamps. Dataset aggregation weights by +GT frame count; an episode-macro score is also logged. + +`Valid/open_loop_sim/Distance_DTW/` reports XYZ_MSE, Episode_XYZ_MSE, +Segments, GT_Frames, Predicted_Samples, GT_Coverage, Prediction_Coverage and +Duration_Ratio. Under this shared episode-time walk, the total rollout sample +count is capped at episode coverage, so Duration_Ratio is normally one; it is +not an independent policy-speed score. Inspect per-segment control steps and +paired time-indexed MSE to understand timing errors. Good DTW alone is not +proof of correct speed or successful robot behavior. + +The nominal source-prefix distance remains D × (K−1)/(M−1). Source supports +span the future path; current-proprio-to-first-target travel is an additional +incoming segment, not included in that source D budget. This number is a +diagnostic, not a guarantee about predicted physical travel. + +## Resources and compatibility + +Exact DTW uses O(N×M) work and backpointers. `dtw_max_cells` guards the matrix +size. Recorded episode ends bound rollout steps; caller-supplied horizons also +bound all-static PW tokens, which contain no duration information. + +Checkpoint sweeps require matching source, model, normalization and metric +contracts. Legacy tensors or caches cannot be loaded as arrival-v2 merely +because their shapes match. Keep old checkpoints on their recorded source. diff --git a/docs/LIBERO_ARC_OAT_DP.md b/docs/LIBERO_ARC_OAT_DP.md index cc154cbbfa..593b74aeed 100644 --- a/docs/LIBERO_ARC_OAT_DP.md +++ b/docs/LIBERO_ARC_OAT_DP.md @@ -47,7 +47,7 @@ only the camera/state encoder receives the observation LR. Use `--arc-backbone oat_dp` with a standalone `--arc-profile` and one ARC mode in `scripts/benchmarks/launch_libero_osmo.py`. The new experiment is -`oat/libero_arc_oat_dp_policy`; the existing default remains the U-Net. +`arc_sim/oat/libero_arc_oat_dp_policy`; the existing default remains the U-Net. A fresh policy may reuse `--arc-replay-runs-file` without `--resume-from-run`. The worker still checks the complete replay confirmation, chosen R/D/M, diff --git a/docs/LIBERO_ARC_OAT_HYBRID.md b/docs/LIBERO_ARC_OAT_HYBRID.md index 5ac4aeb035..0058513a6b 100644 --- a/docs/LIBERO_ARC_OAT_HYBRID.md +++ b/docs/LIBERO_ARC_OAT_HYBRID.md @@ -42,8 +42,8 @@ episodes, two observations, a 32-action horizon and 16 executed actions per replan. These are separate experiment runs; previous ARC/OAT weights are not used to initialize the models. -The YAML recipes are `oat/libero_arc_oattok` and -`oat/libero_arc_oatpolicy`. They compose the existing `LiberoArcStage`, +The YAML recipes are `arc_sim/oat/libero_arc_oattok` and +`arc_sim/oat/libero_arc_oatpolicy`. They compose the existing `LiberoArcStage`, `OATTokenizerStage`/`OATPolicyStage`, and ARC decoder in `PipelineAlgo`. The shared trainer, decoded replay loader, normalizer and evaluator are used. Validation reports dense controller-action errors after decoding, while the diff --git a/docs/LIBERO_ARC_REPLAY.md b/docs/LIBERO_ARC_REPLAY.md index ba294aeb09..f652470ac9 100644 --- a/docs/LIBERO_ARC_REPLAY.md +++ b/docs/LIBERO_ARC_REPLAY.md @@ -60,7 +60,7 @@ selection 2–9, and final testing 35–49. Each mode freezes its own R/D/M choi These specs use 16 workers, recycling after four episodes, and 128 GiB memory. Use `--replay-spec libero_arc_replay_stk` or `libero_arc_replay_dur` when rendering an OSMO workflow. Their corresponding policy recipes are -`oat/libero_arc_stk_policy` and `oat/libero_arc_dur_policy`. +`arc_sim/oat/libero_arc_stk_policy` and `arc_sim/oat/libero_arc_dur_policy`. The earlier LIBERO codec/results are explicitly **`joint_dur`**, a shared-clock M×11 baseline. They cannot be reused as STK or independent-clock DUR evidence. diff --git a/docs/PI05_GRAPH.md b/docs/PI05_GRAPH.md index 52d2057b2a..bff4e543cd 100644 --- a/docs/PI05_GRAPH.md +++ b/docs/PI05_GRAPH.md @@ -38,8 +38,14 @@ accept `WT`, `PY`, `PI05_WEIGHTS`, and their existing norm/output/resume options Their lab filesystem defaults and Slurm partitions still need to match the cluster. Checkpoints are retained. No launcher was submitted during this port. -Data, model and evaluator groups are prefixed with `pi05/`, for example -`data=pi05/mecka_abc_pi_6d`. The older `aria_bimanual` configuration slot maps +Data groups retain `pi05/`, for example `data=pi05/mecka_abc_pi_6d`. Model +recipes now live under `model/pi/` with their shared implementation defaults +under `model/base/`; for example `model=pi/bc_mecka_abc_6d`. +Evaluators are named by behavior and frame, such as +`evaluator=eval_cartesian_wrist_6d_prompt`, not by model family. See the +[configuration guide](CONFIGURATION.md). This config reorganization does not +move the Python backend in `models/pi05/` or its compatibility imports. +The older `aria_bimanual` configuration slot maps to `human_bimanual`; the SQL filters and episode exclusions retain their source values. Some historical filters intentionally describe older episode-table snapshots and may require a matching snapshot to reproduce their experiments. @@ -91,7 +97,7 @@ dataset access and robot actuation remain cluster validation work. ## Attention and latent analysis -Root `pi0.5_*` recipes declare `PI05AttentionProvider` under +The retained `model/pi/*_legacy` recipes declare `PI05AttentionProvider` under `model.diagnostic_provider`. Select `data=cotrain_pi_latent evaluator=eval_latent` with a cotraining model to capture attention keys through the normal graph. Set `data.selection.mode=pairs` for the source-pinned pairs, or `custom` with diff --git a/docs/STATTEMPO.md b/docs/STATTEMPO.md new file mode 100644 index 0000000000..c08b354070 --- /dev/null +++ b/docs/STATTEMPO.md @@ -0,0 +1,40 @@ +# Current real-task recipes + +The current ARC cleanup stack is based on PR #263. Its public experiment +entry points live under `egomimic/hydra_configs/experiment/arc_real/`: + +- `stationary_{time,velocity,duration}` — 417 training / 24 validation episodes +- `towels_{time,velocity,duration}` — 462 training / 17 validation episodes +- `earbuds_{time,velocity,duration}` — 339 training / 15 validation episodes + +ABC-lab multitask recipes remain under `experiment/abc_arc/robot_bc/multitask/`, +separate from the nine rl2 real-task recipes introduced in PR #277. + +Each YAML selects a complete model/data/evaluator/training/logging configuration. +See the [experiment menu](../egomimic/hydra_configs/experiment/README.md) for +defaults, pinned dataset pools, variants and launch selection, and the +[configuration guide](CONFIGURATION.md) for the shared-component layout. + +The canonical [clock contract](ARC_CLOCK_CONTRACT.md) has incoming waypoint +velocity or absolute waypoint seconds, including measured current proprio to waypoint zero. The +`full_separation`, `translation_rotation`, `translation` and `rotation` +layouts use six, four, two and two clocks respectively. `_duration` selects the +absolute-seconds duration representation; the separate `arc_real/arcdur_abstime` +experiment menu is removed, not required to enable it. Retired codecs are not +selectable. + +All nine use HPT300, 240k optimizer steps and 25% open-loop evaluation. ARC +defaults to M200, a 100-frame source window, one translation clock per arm and +D=0.22 m for stationery/earbuds or D=0.33 m for towels. The rollout contract +executes 50% of the ARC waypoints; these distinct evaluation/deployment prefixes +are preserved from PR #277. + +YAM task recipes retain their HPT300 architecture and episode +membership. They link directly to `model/hpt/yam_300m` (baseline) or +`model/hpt/yam_300m_arc` (ARC), their matching data adapter, and the +open-loop evaluator. Reusable settings live in `model/base/`, `trainer/` and +`model_contract/`; no real recipe imports another experiment or `_shared/`. + +Older experiments, removed launchers and checkpoint layouts remain recoverable +from their original Git commits. Preserve every running job's recorded source, +config and normalizer. Do not switch a checkout used by active jobs. diff --git a/docs/YAM_RUNTIME.md b/docs/YAM_RUNTIME.md index 60243cbe6f..eaac664877 100644 --- a/docs/YAM_RUNTIME.md +++ b/docs/YAM_RUNTIME.md @@ -1,5 +1,9 @@ # Shared Eva and Yam robot runtime +Historical test commands below refer to their recorded source revisions. +The current checkout keeps checks outside the repository; see the +[verification policy](VERIFICATION_POLICY.md). + Collection and rollout live in `egomimic/robot/`, outside embodiment directories. `interface.py` defines their common contract; the existing Eva `ARXInterface` and the new `yam/interface.py::YamInterface` implement it. Eva remains supported, @@ -274,6 +278,32 @@ X, home, quit and exceptions. These files remain available for inspection. ## Local graph rollout +Check the complete checkpoint load on CPU before opening station devices: + +```bash +python -m egomimic.robot.rollout \ + --config egomimic/hydra_configs/robot/yam_rl2_towels_hpt300_rollout.yaml \ + --check-model +python -m egomimic.robot.rollout \ + --config egomimic/hydra_configs/robot/yam_rl2_earbuds_hpt300_rollout.yaml \ + --check-model +``` + +`--check-model` forces CPU in memory, restores the bound graph and strict +checkpoint weights, prints a JSON result, and exits before creating any robot, +CAN, camera, or dashboard. It leaves the YAML unchanged. `--check-config` +validates settings only; it does not test checkpoint compatibility. + +The RL2 HPT300 profiles select the 180k-step towel and earbud bundles from +2026-10-12; the default `yam_rl2_hptflow_rollout.yaml` selects towels. Their +schema-2 inference sidecars require the matching bound-inference runtime. +The older tokenshapes/IK runtime fails constructing `FlowDenoiserStage` with +`unexpected keyword argument 'clip_sample_range'`. Use the matching runtime; +removing the keyword alone leaves further graph/schema incompatibilities. +Keep each bundle's config, data context, normalizer and inference sidecar +together. These profiles retain the existing IK controls: MuJoCo by default, +optional TRAC-IK, and pose relaxation disabled by default. + ```bash python -m egomimic.robot.rollout --config egomimic/hydra_configs/robot/yam_rollout.yaml python -m egomimic.robot.rollout --config egomimic/hydra_configs/robot/eva_rollout.yaml @@ -292,6 +322,15 @@ bundle. Rollout discovers either that generic name or the immutable the exact resolved `model.pipeline`, inference declaration and data context; station camera maps, calibration, prompt, and safety limits stay in the rollout YAML. +Do not put `inference_graph`, inferred family profiles or sampler defaults in +the station policy. The loader rejects these obsolete settings rather than +silently ignoring them. The checkpoint picker prefers `data-context.json` +(including run-prefixed names); a historical `norm_stats.json` filename does +not exempt its contents from the full-context and checkpoint-binding checks. +The RL2 template requires explicit bundle paths through +`YAM_HPTFLOW_TRAINING_CONFIG`, `YAM_HPTFLOW_CHECKPOINT`, and +`YAM_HPTFLOW_NORMALIZER` (the full data-context file, despite the legacy variable +name). There is no fallback to an old unbound checkpoint. To verify and re-export a sidecar for an already-bound checkpoint: ```bash @@ -314,15 +353,15 @@ inference mode is `graph`. The rollout loop has one model-independent boundary: it passes the current camera/proprio observation to `policy.predict()` and receives a canonical Cartesian action trajectory. The selected model's `inference-config.yaml` owns -the rest of inference: optional observation history, checkpoint-stage matching, +the rest of inference: optional observation history, stable stage identifiers, sampler settings, native output shape, ARC decoding, and the final output contract. A model with history keeps and resets that history inside the policy; the rollout loop does not branch on model family. Profiles fail closed unless -exactly one checkpoint-declared stage/variant match is found. This keeps Flow, +exactly one model-declared deployment profile resolves to its stable stage ID. This keeps Flow, ARC-velocity, ARC-duration, and diffusion checkpoints behind the same robot interface while preventing a Flow sampler override from being applied to a -diffusion stage. `policy.inference_graph` remains only as a legacy fallback for -checkpoint bundles created before the model-owned artifact. +diffusion stage. Old inferred profiles are not a deployment fallback; retain the +original runtime or perform a verified migration for unbound bundles. Each matched profile may expose a bounded `overrides` mapping. Those declarations are the sole source of runtime controls shown by the dashboard: the UI does not @@ -347,66 +386,222 @@ when those frames actually coincide. `action_frame: eef_frame` anchors every predicted pose in a chunk to the measured EEF pose at inference time. `model_frame` instead uses the configured calibration. Predictions are unnormalized once before decoding/frame reversion. A generated -ARC artifact contains a matching `inference_graph.profiles` entry, for example: +ARC artifact contains the schema-2 model declaration and immutable binding +hashes; it is generated, not hand-written. Select the matching +`experiment/arc_real//arc_vel` or `arc_dur` recipe. Its model contract declares +the decoder, clock mode, stream separation, waypoint cap and runtime controls. +Changing a YAML cannot migrate weights trained with another representation. + +The current codec supports incoming per-waypoint velocity or absolute waypoint +seconds. Stream layouts are `full_separation` (M×20), `translation_rotation` +(M×18), `translation` (M×16) and `projection` (M×16). In the last two, rotation +and gripper use their arm's translation clock; projection measures progress +at an offset point on the end effector. Row zero encodes the leading delay. +See the [clock contract](ARC_CLOCK_CONTRACT.md) and +[configuration guide](CONFIGURATION.md). Retired codecs require their original +source revision; they are not runtime alternatives in this stack. + +A `decoder_attribute` control binds only to an explicitly configured decoder. +Generic inference consumes the decoder's optional `execution_steps()` +capability; otherwise `replan_every` remains authoritative. Decoding precedes +IK, and both arms must pass joint-step guards before either command is sent. +Camera loss pauses commands and discards the plan. Quit with q/Escape or Ctrl-C. + +### One codec: the training tokenizer + +`FirstStreamArcDecoder` executes multistream tokens from `arc_length_tokenizer` +until the first moving stream reaches its configured waypoint prefix cap. +`BimanualIntervalArcDecoder` decodes the whole token. Both use the training +codec's own clocks; no checkpoint-era tokenizer copies are carried. + +Use the exported model contract's `first_stream`, `execute_percent` and +sampling budget rather than imposing a station-wide default. The frozen time +contract (`model_contract/yam_wrists.yaml`) declares `TimeChunkRetimer` +and replay/hold tempo controls; 100% preserves the original chunk. Replay tempo +and termination use decode-only helpers and do not alter training targets. + +### Rollout capture and active time + +The dashboard's **Record episode (d)** control captures the same top-level HDF5 +observations/actions/cameras as GELLO collection. It also records policy plans, +per-row timestamps and inference settings under `rollout/`. Save a take as +success, failure or unlabeled; discard removes only that take. **Recorded +episodes** lists finalized takes. MP4 capture remains available independently. + +The background writer uses a bounded queue. Slow disk, a full queue or low free +space ends capture as incomplete without adding robot commands or blocking the +control loop. Startup/save metadata still requires small filesystem operations; +continuous image writes and finalization run on the writer thread. The RL2 +profile sets a 20 GB free-space floor. Plan records distinguish the selected +executable prefix from actual commands; per-row commands are the execution +evidence. Restart saves an active take as unlabeled; camera/model changes and +abnormal exit preserve it as incomplete. Never overwrite existing data. + +**Rollout time** is active session time since Continue: explicit pause and +camera/model-load holds freeze it; browser reconnect preserves it; Restart +resets to zero. This is not a per-action motion-time measurement and does not +subtract inference latency or safety decision wait time. + +After a failed checkpoint selection, execution remains locked until a valid +model loads. The old loaded model is never resumed implicitly. + +### Reset guides + +The **Reset guide** menu above the live cameras on both dashboards offers +**Off** (default) and saved placement templates. The RL2 earbuds template shows +adjacent left-earbud, case and right-earbud regions projected onto the table in +`front_img_1`. It is a visual annotation, not surveyed calibration or an +automatic reset: recheck placement if the camera or table moves. Guides are SVG +browser annotations only; raw camera pixels, recorded videos and episodes, +policy inputs and robot commands are unchanged. They hide when disconnected, +awaiting frames or on an incompatible image aspect ratio, track the image's +letterboxing, and survive a websocket reconnect until the page is refreshed. + +Template geometry lives in `egomimic/hydra_configs/robot/reset_overlay_templates.yaml`. +To add or change a template run `python -m egomimic.scripts.build_reset_guides` +and commit both generated dashboard catalogs and the rollout renderer; +`--check` detects a stale catalog or renderer drift. The renderer is +task-agnostic (its canonical block lives in `teleop_dashboard_static/app.js`; +the builder synchronizes rollout's copy); keep template coordinates out of +JavaScript. Refresh the browser to load these static-only changes; no +collection or robot session restart is needed. + +### Rollout gripper force limit + +Rollouts start with `robot.gripper_force_limit: 20.0` N (the YAM driver's maximum +is 50 N; GELLO collection stays at 50). When the robot can change it live +(`YamInterface.set_gripper_force_limit`), the rollout dashboard shows one +**Gripper force limit** slider per arm (1-50 N). A change is applied by the +rollout thread on its next control tick and the dashboard shows the limit the +driver reports; a driver that refuses keeps its limit and says so in the status. + +### Optional rollout IK pose relaxation + +`ik_relaxation.enabled` defaults to `false`. On YAM Cartesian rollouts the +**Allow small IK pose errors** dashboard checkbox changes it on the control +thread, discards the queued chunk and requests a fresh plan. It is independent +of checkpoint inference settings and persists across model changes and Restart +within the process; a process restart restores the YAML value. The checkbox +shows the applied state and the configured bounds. + +```yaml +ik_relaxation: + enabled: false + position_tolerance_m: 0.005 + rotation_tolerance_deg: 3.0 +``` + +Strict endpoint IK is always tried first. Only after it fails does a bounded +retry use tolerance-scaled position and orientation errors. A candidate must +satisfy **both** FK error bounds relative to the original target; a low summed +cost alone is insufficient. The normal paired joint-step guard still applies. +The fallback does not change the target gripper opening, the pinned Quest +solver, or joint replay. Configuration is validated without devices, including +while disabled: position tolerance must be in (0, 0.02] meters and rotation in +(0, 15] degrees. These are absolute envelopes, not increments to strict limits. +The defaults are experimental and require physical validation on the task. +This relaxes all orientation axes equally; it does not plan around an +unreachable fold apex or guarantee recovery from a singularity. + +### Differential IK recovery + +`yam_rl2_hptflow_rollout.yaml` and the two HPT300 profiles enable +`ik_recovery`; other profiles default to disabled. Normal successful endpoint +IK is unchanged. When either arm's strict solve fails, both arms use local, +posture-regularized differential steps toward the same queued Cartesian row. +The pinned Quest `streaming_ik.py` and teleop settings are unchanged. ```yaml -kind: egomimic.graph-inference -schema_version: 1 -status: ready -model_pipeline_sha256: -inference_contract_sha256: -inference_graph_sha256: -inference_graph: - input: - history_length: 1 - output: - representation: cartesian - shape: [100, 14] - profiles: - flow_arcdur: - match: - stage_target: egomimic.pipeline.stages_flow.FlowDenoiserStage - variant: arcdur - native_shape: [100, 16] - overrides: - inference_steps: - label: Euler integration steps - description: Number of Flow solver steps used for each prediction. - type: integer - min: 1 - max: 100 - step: 1 - default: 10 - target: - kind: stage_attribute - attribute_path: num_inference_steps - replan_every: - label: Repredict every - description: Execute this many actions before requesting a fresh prediction. - type: integer - min: 1 - max: 100 - step: 1 - default: 30 - target: - kind: policy_attribute - attribute_path: replan_every - adapter: - decoder: - _target_: egomimic.robot.arc_decoder.BimanualArcDecoder - token_layout: e1_dur - min_distance_unit: 0.4 - resampled_vector_length: 100 - dt: 0.03333333333333333 - action_horizon: 100 +ik_recovery: + enabled: true + max_joint_velocity: 1.0 + position_scale_m: 0.05 + rotation_scale_rad: 0.5 + damping: 0.01 + posture_weight: 0.05 + max_ticks_per_target: 30 ``` -Use the codec and numbers from training. Supported layouts are `lab`, `e1_dur`, -`e1_logdur` and `e1_profile`. Decoding precedes IK. The inference policy returns -the profile-selected executable prefix; the rollout loop has no model-specific -replanning parameter. Both arms' commands must pass the joint step limit before -either command is sent. Camera loss pauses commands and discards the old plan. -Quit with q/Escape or Ctrl-C. +Each solve is a six-joint bounded least-squares problem with posture continuity, +extra regularization on joints 2-4, scaled Cartesian residuals and stronger +damping near singularity. Its bounds intersect the model joint limits with +one tick of motion around BOTH measured joints and the last sent command. +The rate is the smaller of the recovery rate and the rollout rate, with the +actual configured rollout period. A bounded nonlinear backtracking check +requires Cartesian error reduction; solver failure, incompatible bounds or +no progress commands neither arm and discards the plan. + +The row remains queued during partial progress. Both grippers retain their +last commanded opening until both commanded endpoints reach the strict IK +tolerances, then the original row's gripper targets apply. No later row can +execute during recovery. Pause, restart, camera loss and new-model events +still clear the queue. After `max_ticks_per_target` commanded recovery steps, +the plan is discarded and the policy replans; this is not a session-wide +stall timeout. A permanently infeasible target may still require operator +intervention. Recovery changes timing and may affect inter-hand geometry; +it does not enforce a cloth constraint or provide collision avoidance. +Checkpoint plan metadata records the recovery settings. Set +`ik_recovery.enabled: false` to restore the endpoint-only rollout behavior. + +### Optional TRAC-IK endpoint solver + +The dashboard **Endpoint IK** selector and the YAML choose the endpoint solver: + +```yaml +ik_solver: + backend: mujoco # default; select pytracik to opt in + timeout_s: 0.005 # per arm, (0, 0.02] seconds + solver_type: Distance # also Speed, Manip1, Manip2 +``` + +Install the optional dependency in the station environment with +`python -m pip install pytracik==0.0.3` (or the `yam-tracik` project extra). +`--check-config` verifies that it imports when selected in YAML. A missing +library or failed chain check is reported explicitly; a failed dashboard switch +keeps the previous backend. Settings persist across Restart and checkpoint +changes in that process; a new process uses YAML again. + +TRAC-IK replaces strict endpoint solving. The adapter generates a serial URDF +from the **loaded** MuJoCo model's joints, limits and TCP, then checks TRAC-IK FK +against MuJoCo on 13 configurations before selection. Every returned endpoint is +checked for finite joints, joint limits, and the same independent Cartesian +acceptance bounds (normally 2 mm and 0.02 rad). Both arms must still pass the +rollout's joint-step guard. MuJoCo remains responsible for FK, optional bounded +pose relaxation and bounded differential recovery. Backend changes run on the +rollout thread, prepare both arms before switching, discard queued waypoints, +and resume from a fresh observation. The selected solver settings are recorded +with each plan. + +A reproducible CPU comparison, with no robot-driver imports: + +```bash +python -m egomimic.robot.yam.benchmark_ik \ + --xml-path /path/to/the/station/yam_linear_4310_d405.xml \ + --count 100 --repeats 3 --seed 20261011 \ + --solver-type Distance --timeout-s 0.005 --max-step 0.4 \ + --output /tmp/yam-ik-comparison.json +``` + +Station CPU result (2026-10-11, pytracik 0.0.3, MuJoCo 3.4.0; valid endpoints / +endpoints within the 0.4-rad command step, out of 300 per group): TRAC-IK +solved 300 / 300 high-TCP targets against MuJoCo's 273 / 273 and rescued 72 +failed reachable attempts while missing three on its time budget; typical +reachable-case medians were 5.5 ms per arm for TRAC-IK versus 0.6-1.8 ms for +MuJoCo. MuJoCo stays the default; the comparison is synthetic and does not +establish physical fold success. + +### Station monitors and checkers + +- `egomimic/robot/camera_align/` watches the front D405 for camera or desk + motion against a saved reference (SIFT roll / tilt / pan, ArUco poses) and + serves a loopback page on port 8090. `run.sh` opens the camera directly + (`--source realsense`, so no rollout dashboard is needed; stop it before a + rollout or `collect_gello` starts), `SOURCE=stream ./run.sh` reads the rollout + dashboard's websocket instead. See its README. +- `python -m egomimic.robot.hdf5_tempo ` buckets recorded GELLO + episodes by tempo and skips an unusable file instead of aborting the scan. +- `python -m egomimic.robot.rollout --config --check-model` strictly + restores the selected bundle on CPU and exits before any device opens. ## Zarr replay and upload @@ -438,9 +633,6 @@ directory. `--list` performs no upload and needs no AWS credentials. ## Validation scope -The regression sources referenced below are historical and no longer shipped; -see [the verification policy](VERIFICATION_POLICY.md). Keep new checks local. - `tests/test_robot_runtime.py` checks collection, format compatibility, world-frame clutch control, stale tracking/cameras, previews, Eva/Yam interfaces, cleanup and Zarr replay. `tests/test_robot_graph_policy.py` checks strict checkpoint loading, diff --git a/docs/action-flow-usocket-candidates.md b/docs/action-flow-usocket-candidates.md index 202a6d22af..90d8b9e352 100644 --- a/docs/action-flow-usocket-candidates.md +++ b/docs/action-flow-usocket-candidates.md @@ -11,7 +11,7 @@ reconstruction-only phase. The generic pipeline runner is unchanged. ### FM-only clean stop-gradient (method 2) -Config: `pusht/action_flow_bc_usocket_latent_fm_sg_recon1_s42`. +Config: `arc_sim/pusht/action_flow_bc_usocket_latent_fm_sg_recon1_s42`. - Reconstruction: `A → E → g → MSE(A)`; updates E and g, weight1. - Latent FM: `A → E → stop-gradient → latent bridge → v(z,t,O) → latent MSE`; @@ -28,7 +28,7 @@ Inference: `Gaussian latent → reverse Euler16(v,O) → g → action`. ### Learned Gaussian-bridge likelihood (method 3) -Config: `pusht/action_flow_bc_usocket_bridge_likelihood_s42`. +Config: `arc_sim/pusht/action_flow_bc_usocket_bridge_likelihood_s42`. Private reference mean `mu(A,t)=(1-t)E_raw(A,t)` ends exactly at zero. The reference stage, Gaussian noising stage, conditional reverse-mean transformer, @@ -64,7 +64,7 @@ errors and fixed-bank stochastic EnergyScore@32 remain required. ### Exact graph section (method 4, restricted diagnostic) -Config: `pusht/action_flow_bc_usocket_graph_section_s42`. +Config: `arc_sim/pusht/action_flow_bc_usocket_graph_section_s42`. `E(A)=(A,f(A))`; split each latent token into x4,h4 and decode `g(x,h)=x + [R(x,h)-R(x,f(x))]`. The subtraction is grouped before adding x. diff --git a/docs/cluster-pipeline.md b/docs/cluster-pipeline.md index 39b8fa40f7..5942487be9 100644 --- a/docs/cluster-pipeline.md +++ b/docs/cluster-pipeline.md @@ -21,14 +21,21 @@ Refresh remote state and verify ancestry before choosing a new source. Older documents describing these PRs as unmerged are historical. Existing checkpoint source commits remain authoritative for strict reload and resume. +## Aidan stationery selection + +New ARC/stattempo sessions use the clock-contract cleanup stack above PR #263. +[The stationery guide](STATTEMPO.md) records the retained real-task recipes and +source-preservation rules. Existing checkpoint source pins remain unchanged. +Resolve the selected stack commit before new work. + ## Current PushT selection The paired raw-multiplier recipes are: | Family | Experiment selector | | --- | --- | -| Action Flow | `pusht/action_flow_cotrain_uc_multiplier_interpolation` | -| Matched Standard DP | `pusht/planar_uc_manual4919_dp_261m_af_obs_multiplier` | +| Action Flow | `arc_sim/pusht/action_flow_cotrain_uc_multiplier_interpolation` | +| Matched Standard DP | `arc_sim/pusht/planar_uc_manual4919_dp_261m_af_obs_multiplier` | Use normal per-source image processing, retaining 32 U-Socket plus 32 ChainGripper samples per optimizer update. ChainGripper manual4919 contains diff --git a/docs/config_cleanup_migration.json b/docs/config_cleanup_migration.json new file mode 100644 index 0000000000..6dd8d523b1 --- /dev/null +++ b/docs/config_cleanup_migration.json @@ -0,0 +1,56 @@ +{ + "note": "Selectors only. Preserve original source/configs for existing checkpoints; removed experiments have no automatic replacement.", + "renamed": { + "data/cotrain_pi_pickplace_qwen": "data/cotrain_pickplace_qwen", + "evaluator/eval_hpt": "evaluator/eval_cartesian_camera", + "evaluator/eval_hpt_wrist": "evaluator/eval_cartesian_wrist", + "evaluator/eval_pi": "evaluator/eval_cartesian_camera_prompt", + "evaluator/pi05/eval_pi": "evaluator/eval_cartesian_wrist_prompt", + "evaluator/pi05/eval_pi_camframe_6d": "evaluator/eval_cartesian_camera_6d_prompt", + "evaluator/pi05/eval_pi_wristframe_6d": "evaluator/eval_cartesian_wrist_6d_prompt", + "evaluator/pi05/eval_pi_wristframe_6d_grip": "evaluator/eval_cartesian_wrist_6d_gripper_prompt", + "evaluator/viz/cartesian_wrist": "evaluator/viz/cartesian", + "evaluator/viz/cartesian_wrist_lang": "evaluator/viz/cartesian_annotations", + "evaluator/viz/cotrain_lang": "evaluator/viz/cartesian_annotations", + "evaluator/viz/pi_cartesian": "evaluator/viz/cartesian", + "evaluator/viz/pi_cartesian_lang": "evaluator/viz/cartesian_prompt", + "evaluator/viz/pi_cartesian_lang_wrist": "evaluator/viz/cartesian_prompt", + "model/abc/yam_bimanual_hpt": "model/hpt/yam_bimanual", + "model/abc_arc/hpt_bimanual_visual": "model/hpt/bimanual_visual", + "model/abc_arc/hpt_yam_visual": "model/hpt/yam_visual", + "model/egobridge": "model/hpt/egobridge", + "model/hpt_bc_flow_aria": "model/hpt/bc_flow_human", + "model/hpt_bc_flow_eva": "model/hpt/bc_flow_eva", + "model/hpt_bc_flow_human": "model/hpt/bc_flow_human", + "model/hpt_bc_flow_mecka": "model/hpt/bc_flow_mecka", + "model/hpt_bc_flow_scale": "model/hpt/bc_flow_scale", + "model/hpt_bc_keypoints_base": "model/base/hpt_keypoints", + "model/hpt_bc_keypoints_wrist": "model/hpt/bc_keypoints_wrist", + "model/hpt_bc_pickplace_qwen_pertoken": "model/hpt/bc_pickplace_qwen_pertoken", + "model/hpt_bc_pickplace_qwen_pooled": "model/hpt/bc_pickplace_qwen_pooled", + "model/hpt_cotrain_enc_dec_base": "model/base/hpt_cotrain_enc_dec", + "model/hpt_cotrain_flow_seperate_head": "model/hpt/cotrain_flow_seperate_head", + "model/hpt_cotrain_flow_shared_head": "model/hpt/cotrain_flow_shared_head", + "model/hpt_cotrain_mecka_flow_shared_head": "model/hpt/cotrain_mecka_flow_shared_head", + "model/hpt_cotrain_scale_flow_shared_head": "model/hpt/cotrain_mecka_flow_shared_head", + "model/pi0.5_base": "model/base/pi05", + "model/pi0.5_bc_aria": "model/pi/bc_aria_legacy", + "model/pi0.5_bc_eva": "model/pi/bc_eva_legacy", + "model/pi0.5_bc_mecka": "model/pi/bc_mecka_legacy", + "model/pi0.5_bc_scale": "model/pi/bc_mecka_legacy", + "model/pi0.5_cotrain_eva_aria": "model/pi/cotrain_eva_aria_legacy", + "model/pi0.5_cotrain_mecka_scale": "model/pi/bc_mecka_legacy", + "model/pi05/pi0.5_bc_abc_eva_6d": "model/pi/bc_abc_eva_6d", + "model/pi05/pi0.5_bc_eva": "model/pi/bc_eva", + "model/pi05/pi0.5_bc_mecka_6d": "model/pi/bc_mecka_6d", + "model/pi05/pi0.5_bc_mecka_abc_6d": "model/pi/bc_mecka_abc_6d", + "model/pi05/pi0.5_cotrain_eva_aria_6d": "model/pi/cotrain_eva_aria_6d", + "model/pi05/pi0.5_ft_abc_eva_6d": "model/pi/bc_abc_eva_6d", + "model/pi05/pi0.5_ft_aria_fold_grip": "model/pi/ft_aria_fold_grip" + }, + "removed_experiment_menus": [ + "abc", + "abc_arc/human_bc", + "arc_real/arcdur_abstime" + ] +} diff --git a/docs/experiment_config_migration.json b/docs/experiment_config_migration.json new file mode 100644 index 0000000000..e8a4411ee8 --- /dev/null +++ b/docs/experiment_config_migration.json @@ -0,0 +1,153 @@ +{ + "abc_arc/cotrain/organize_clock_duration_full_separation": "arc_real/stationary_duration", + "abc_arc/cotrain/organize_clock_duration_translation_rotation": "arc_real/stationary_duration", + "abc_arc/cotrain/organize_clock_velocity_full_separation": "arc_real/stationary_velocity", + "abc_arc/cotrain/organize_clock_velocity_translation_rotation": "arc_real/stationary_velocity", + "abc_arc/cotrain/organize_rl2_elmo_hpt180_cotrain_baseline": null, + "abc_arc/cotrain/organize_rl2_elmo_hpt180_cotrain_multistream": null, + "abc_arc/cotrain/organize_rl2_elmo_hpt300_cotrain_baseline": "arc_real/stationary_time", + "abc_arc/cotrain/organize_rl2_elmo_hpt300_cotrain_multistream": "arc_real/stationary_velocity", + "abc_arc/cotrain/organize_rl2_elmo_hpt300_cotrain_multistream_logclock": "arc_real/stationary_duration", + "abc_arc/cotrain/organize_rl2_elmo_hpt300_cotrain_multistream_perwaypoint": "arc_real/stationary_velocity", + "abc_arc/robot_bc/abc_multitask4_abcdit_baseline_visual_openloop": "abc_arc/robot_bc/multitask/variants/abc_multitask4_abcdit_baseline_visual_openloop", + "abc_arc/robot_bc/abc_multitask4_abcdit_hybrid_visual_openloop": "abc_arc/robot_bc/multitask/variants/abc_multitask4_abcdit_hybrid_visual_openloop", + "abc_arc/robot_bc/abc_multitask4_hpt300_baseline_visual_openloop": "abc_arc/robot_bc/multitask/baseline", + "abc_arc/robot_bc/abc_multitask4_hpt300_hybrid_visual_openloop": "abc_arc/robot_bc/multitask/arc_vel", + "abc_arc/robot_bc/abc_rl2_towels_abcdit_baseline_visual_openloop": null, + "abc_arc/robot_bc/abc_rl2_towels_abcdit_multistream_logclock_visual_openloop": null, + "abc_arc/robot_bc/abc_rl2_towels_abcdit_multistream_visual_openloop": null, + "abc_arc/robot_bc/abc_rl2_towels_hpt180_multistream_visual_openloop": null, + "abc_arc/robot_bc/abc_towels_hpt180_baseline_visual_openloop": "abc_arc/robot_bc/abc_towels_hpt180_baseline_visual_openloop", + "abc_arc/robot_bc/abc_towels_hpt180_hybrid_visual_openloop": "abc_arc/robot_bc/abc_towels_hpt180_hybrid_visual_openloop", + "abc_arc/robot_bc/abc_visual_hpt300_base": null, + "abc_arc/robot_bc/stationery_rl2_hpt300_visual_baseline_openloop": null, + "abc_arc/robot_bc/stationery_rl2_hpt300_visual_hybrid_openloop": null, + "fold_and_stack_towels/arc_dur": "arc_real/towels_duration", + "fold_and_stack_towels/arc_vel": "arc_real/towels_velocity", + "fold_and_stack_towels/baseline": "arc_real/towels_time", + "fold_and_stack_towels/variants/abc_rl2_towels_abcdit_baseline_visual_openloop": null, + "fold_and_stack_towels/variants/abc_rl2_towels_abcdit_multistream_logclock_visual_openloop": null, + "fold_and_stack_towels/variants/abc_rl2_towels_abcdit_multistream_visual_openloop": null, + "fold_and_stack_towels/variants/abc_rl2_towels_hpt180_multistream_visual_openloop": null, + "fold_and_stack_towels/variants/abc_towels_hpt180_baseline_visual_openloop": "abc_arc/robot_bc/abc_towels_hpt180_baseline_visual_openloop", + "fold_and_stack_towels/variants/abc_towels_hpt180_hybrid_visual_openloop": "abc_arc/robot_bc/abc_towels_hpt180_hybrid_visual_openloop", + "fold_and_stack_towels/variants/new290plus74_baseline": null, + "libero/action_flow_libero10_oat_dp_matched_s42": "arc_sim/libero/action_flow_libero10_oat_dp_matched_s42", + "libero/action_flow_libero_goal_oat_dp_matched_s42": "arc_sim/libero/action_flow_libero_goal_oat_dp_matched_s42", + "libero/action_flow_libero_object_oat_dp_matched_s42": "arc_sim/libero/action_flow_libero_object_oat_dp_matched_s42", + "libero/action_flow_libero_spatial_oat_dp_matched_s42": "arc_sim/libero/action_flow_libero_spatial_oat_dp_matched_s42", + "libero/action_flow_libero_spatial_sharpa_gripper_keypoints_s42": "arc_sim/libero/action_flow_libero_spatial_sharpa_gripper_keypoints_s42", + "libero/action_flow_libero_spatial_sharpa_mot_s42": "arc_sim/libero/action_flow_libero_spatial_sharpa_mot_s42", + "libero/dp_libero10_oat_batch16_keep_steps_s42": "arc_sim/libero/dp_libero10_oat_batch16_keep_steps_s42", + "libero/dp_libero_goal_oat_batch16_keep_steps_s42": "arc_sim/libero/dp_libero_goal_oat_batch16_keep_steps_s42", + "libero/dp_libero_object_oat_batch16_keep_steps_s42": "arc_sim/libero/dp_libero_object_oat_batch16_keep_steps_s42", + "libero/dp_libero_spatial_oat_batch16_keep_steps_s42": "arc_sim/libero/dp_libero_spatial_oat_batch16_keep_steps_s42", + "libero_historical/action_flow_libero10_avln_80k_s42": "arc_sim/libero_historical/action_flow_libero10_avln_80k_s42", + "libero_historical/action_flow_libero10_h240_euler50_dithalf_80k_av0_s42": "arc_sim/libero_historical/action_flow_libero10_h240_euler50_dithalf_80k_av0_s42", + "libero_historical/action_flow_libero10_h240_euler50_dithalf_80k_s42": "arc_sim/libero_historical/action_flow_libero10_h240_euler50_dithalf_80k_s42", + "libero_historical/action_flow_libero_goal_h240_euler50_dithalf_80k_av0_s42": "arc_sim/libero_historical/action_flow_libero_goal_h240_euler50_dithalf_80k_av0_s42", + "libero_historical/action_flow_libero_goal_h240_euler50_dithalf_80k_s42": "arc_sim/libero_historical/action_flow_libero_goal_h240_euler50_dithalf_80k_s42", + "libero_historical/action_flow_libero_object_h240_euler50_dithalf_80k_av0_s42": "arc_sim/libero_historical/action_flow_libero_object_h240_euler50_dithalf_80k_av0_s42", + "libero_historical/action_flow_libero_object_h240_euler50_dithalf_80k_s42": "arc_sim/libero_historical/action_flow_libero_object_h240_euler50_dithalf_80k_s42", + "libero_historical/action_flow_libero_spatial_h240_euler50_dithalf_80k_av0_s42": "arc_sim/libero_historical/action_flow_libero_spatial_h240_euler50_dithalf_80k_av0_s42", + "libero_historical/action_flow_libero_spatial_h240_euler50_dithalf_80k_s42": "arc_sim/libero_historical/action_flow_libero_spatial_h240_euler50_dithalf_80k_s42", + "libero_historical/dp_libero10_oat_dp_matched_s42": "arc_sim/libero_historical/dp_libero10_oat_dp_matched_s42", + "libero_historical/dp_libero_goal_oat_dp_matched_s42": "arc_sim/libero_historical/dp_libero_goal_oat_dp_matched_s42", + "libero_historical/dp_libero_object_oat_dp_matched_s42": "arc_sim/libero_historical/dp_libero_object_oat_dp_matched_s42", + "libero_historical/dp_libero_spatial_oat_dp_matched_s42": "arc_sim/libero_historical/dp_libero_spatial_oat_dp_matched_s42", + "multitask/arc_dur": "abc_arc/robot_bc/multitask/arc_dur", + "multitask/arc_vel": "abc_arc/robot_bc/multitask/arc_vel", + "multitask/baseline": "abc_arc/robot_bc/multitask/baseline", + "multitask/variants/abc_multitask4_abcdit_baseline_visual_openloop": "abc_arc/robot_bc/multitask/variants/abc_multitask4_abcdit_baseline_visual_openloop", + "multitask/variants/abc_multitask4_abcdit_hybrid_visual_openloop": "abc_arc/robot_bc/multitask/variants/abc_multitask4_abcdit_hybrid_visual_openloop", + "oat/libero_arc_dur_1_r192_d1p6_m24": "arc_sim/oat/libero_arc_dur_1_r192_d1p6_m24", + "oat/libero_arc_dur_2_r384_d0p8_m32": "arc_sim/oat/libero_arc_dur_2_r384_d0p8_m32", + "oat/libero_arc_dur_policy": "arc_sim/oat/libero_arc_dur_policy", + "oat/libero_arc_global_chebyshev": "arc_sim/oat/libero_arc_global_chebyshev", + "oat/libero_arc_global_common": "arc_sim/oat/libero_arc_global_common", + "oat/libero_arc_global_fourier": "arc_sim/oat/libero_arc_global_fourier", + "oat/libero_arc_global_pilot": "arc_sim/oat/libero_arc_global_pilot", + "oat/libero_arc_global_uniform": "arc_sim/oat/libero_arc_global_uniform", + "oat/libero_arc_oat_base": "arc_sim/oat/libero_arc_oat_base", + "oat/libero_arc_oat_dp_policy": "arc_sim/oat/libero_arc_oat_dp_policy", + "oat/libero_arc_oatpolicy": "arc_sim/oat/libero_arc_oatpolicy", + "oat/libero_arc_oattok": "arc_sim/oat/libero_arc_oattok", + "oat/libero_arc_policy": "arc_sim/oat/libero_arc_policy", + "oat/libero_arc_shared_r384_d1p6_m36": "arc_sim/oat/libero_arc_shared_r384_d1p6_m36", + "oat/libero_arc_stk_1_r192_d1p6_m36": "arc_sim/oat/libero_arc_stk_1_r192_d1p6_m36", + "oat/libero_arc_stk_2_r192_d0p8_m32": "arc_sim/oat/libero_arc_stk_2_r192_d0p8_m32", + "oat/libero_arc_stk_policy": "arc_sim/oat/libero_arc_stk_policy", + "oat/libero_base": "arc_sim/oat/libero_base", + "oat/libero_oatpolicy": "arc_sim/oat/libero_oatpolicy", + "oat/libero_oattok": "arc_sim/oat/libero_oattok", + "organize_stationary/_base/arc": "arc_real/stationary_velocity", + "organize_stationary/arc_dur": "arc_real/stationary_duration", + "organize_stationary/arc_vel": "arc_real/stationary_velocity", + "organize_stationary/baseline": "arc_real/stationary_time", + "organize_stationary/variants/organize_rl2_elmo_hpt180_cotrain_baseline": null, + "organize_stationary/variants/organize_rl2_elmo_hpt180_cotrain_multistream": null, + "organize_stationary/variants/stationery_rl2_hpt300_visual_baseline_openloop": null, + "organize_stationary/variants/stationery_rl2_hpt300_visual_hybrid_openloop": null, + "pusht/action_flow_bc_usocket_bridge_likelihood_s42": "arc_sim/pusht/action_flow_bc_usocket_bridge_likelihood_s42", + "pusht/action_flow_bc_usocket_graph_section_s42": "arc_sim/pusht/action_flow_bc_usocket_graph_section_s42", + "pusht/action_flow_bc_usocket_latent_fm_sg_recon1_200m_adamw_lr1e5_s42": "arc_sim/pusht/action_flow_bc_usocket_latent_fm_sg_recon1_200m_adamw_lr1e5_s42", + "pusht/action_flow_bc_usocket_latent_fm_sg_recon1_200m_muon_lr1e5_s42": "arc_sim/pusht/action_flow_bc_usocket_latent_fm_sg_recon1_200m_muon_lr1e5_s42", + "pusht/action_flow_bc_usocket_latent_fm_sg_recon1_s42": "arc_sim/pusht/action_flow_bc_usocket_latent_fm_sg_recon1_s42", + "pusht/action_flow_bc_usocket_recon100_s42": "arc_sim/pusht/action_flow_bc_usocket_recon100_s42", + "pusht/action_flow_bc_usocket_recon10_s42": "arc_sim/pusht/action_flow_bc_usocket_recon10_s42", + "pusht/action_flow_bc_usocket_recon10_warmup10k_s42": "arc_sim/pusht/action_flow_bc_usocket_recon10_warmup10k_s42", + "pusht/action_flow_bc_usocket_recon1_s42": "arc_sim/pusht/action_flow_bc_usocket_recon1_s42", + "pusht/action_flow_chain_latent_fm_sg_unite_h384_sum14_cfg4_val8_s42": "arc_sim/pusht/action_flow_chain_latent_fm_sg_unite_h384_sum14_cfg4_val8_s42", + "pusht/action_flow_cotrain_uc_multiplier_interpolation": "arc_sim/pusht/action_flow_cotrain_uc_multiplier_interpolation", + "pusht/action_flow_usocket_candidate_common": "arc_sim/pusht/action_flow_usocket_candidate_common", + "pusht/action_flow_usocket_latent_fm_sg_unite_denoiser90m_sum14_cfg4_finalval1_s42": "arc_sim/pusht/action_flow_usocket_latent_fm_sg_unite_denoiser90m_sum14_cfg4_finalval1_s42", + "pusht/action_latent_vfm_usocket_val01_h16": "arc_sim/pusht/action_latent_vfm_usocket_val01_h16", + "pusht/parents/action_flow_cotrain_uc_manual4919": "arc_sim/pusht/parents/action_flow_cotrain_uc_manual4919", + "pusht/parents/action_flow_usocket_h384": "arc_sim/pusht/parents/action_flow_usocket_h384", + "pusht/parents/action_flow_usocket_h384_sum14": "arc_sim/pusht/parents/action_flow_usocket_h384_sum14", + "pusht/planar_uc_manual4919_dp_261m_af_obs_multiplier": "arc_sim/pusht/planar_uc_manual4919_dp_261m_af_obs_multiplier", + "pusht/planar_usocket_chain_manual4919_standard_dp_retimed_masked": "arc_sim/pusht/planar_usocket_chain_manual4919_standard_dp_retimed_masked", + "pusht/planar_v2_base": "arc_sim/pusht/planar_v2_base", + "pusht/planar_v2_chain_arc_bc": "arc_sim/pusht/planar_v2_chain_arc_bc", + "pusht/planar_v2_chain_base": "arc_sim/pusht/planar_v2_chain_base", + "pusht/planar_v2_chain_direct_bc": "arc_sim/pusht/planar_v2_chain_direct_bc", + "pusht/planar_v2_chain_dp_paper": "arc_sim/pusht/planar_v2_chain_dp_paper", + "pusht/planar_v2_chain_dp_standard": "arc_sim/pusht/planar_v2_chain_dp_standard", + "pusht/planar_v2_cotrain_clean_dp_standard": "arc_sim/pusht/planar_v2_cotrain_clean_dp_standard", + "pusht/planar_v2_usocket_arc_bc": "arc_sim/pusht/planar_v2_usocket_arc_bc", + "pusht/planar_v2_usocket_arc_graph_tok": "arc_sim/pusht/planar_v2_usocket_arc_graph_tok", + "pusht/planar_v2_usocket_arc_hybrid_D40_M16_R24deg_bc": "arc_sim/pusht/planar_v2_usocket_arc_hybrid_D40_M16_R24deg_bc", + "pusht/planar_v2_usocket_arc_paper_curvature_D40_M16_R24deg": "arc_sim/pusht/planar_v2_usocket_arc_paper_curvature_D40_M16_R24deg", + "pusht/planar_v2_usocket_arc_paper_uniform_D40_M16_R24deg": "arc_sim/pusht/planar_v2_usocket_arc_paper_uniform_D40_M16_R24deg", + "pusht/planar_v2_usocket_base": "arc_sim/pusht/planar_v2_usocket_base", + "pusht/planar_v2_usocket_direct_bc": "arc_sim/pusht/planar_v2_usocket_direct_bc", + "pusht/planar_v2_usocket_dp_paper": "arc_sim/pusht/planar_v2_usocket_dp_paper", + "pusht/planar_v2_usocket_dp_standard": "arc_sim/pusht/planar_v2_usocket_dp_standard", + "pusht/unite_cotrain_usocket_chain_val01_h16": "arc_sim/pusht/unite_cotrain_usocket_chain_val01_h16", + "pusht/unite_usocket_register_sweep_val01_h16": "arc_sim/pusht/unite_usocket_register_sweep_val01_h16", + "synthetic/direct_fm3d_gaussian_paraboloid": "arc_sim/synthetic/direct_fm3d_gaussian_paraboloid", + "synthetic/direct_fm3d_gaussian_torus": "arc_sim/synthetic/direct_fm3d_gaussian_torus", + "synthetic/unite_gaussian_torus": "arc_sim/synthetic/unite_gaussian_torus", + "synthetic/unite_gaussian_torus_latent2_codec10k": "arc_sim/synthetic/unite_gaussian_torus_latent2_codec10k", + "synthetic/unite_gaussian_torus_latent4": "arc_sim/synthetic/unite_gaussian_torus_latent4", + "synthetic/unite_gaussian_torus_latent4_codec10k": "arc_sim/synthetic/unite_gaussian_torus_latent4_codec10k", + "synthetic/vfm_gaussian_torus": "arc_sim/synthetic/vfm_gaussian_torus", + "synthetic/vfm_gaussian_torus_latent2_codec10k": "arc_sim/synthetic/vfm_gaussian_torus_latent2_codec10k", + "synthetic/vfm_gaussian_torus_latent4": "arc_sim/synthetic/vfm_gaussian_torus_latent4", + "synthetic/vfm_gaussian_torus_latent4_codec10k": "arc_sim/synthetic/vfm_gaussian_torus_latent4_codec10k", + "wireless_earbuds/arc_dur": "arc_real/earbuds_duration", + "wireless_earbuds/arc_vel": "arc_real/earbuds_velocity", + "wireless_earbuds/baseline": "arc_real/earbuds_time", + "yam_arc_grid/_base/earbuds_updated_hpt300": "arc_real/earbuds_time", + "yam_arc_grid/_base/stattempo": null, + "yam_arc_grid/_base/stattempo_openloop": null, + "yam_arc_grid/_base/towels_hpt300": "arc_real/towels_time", + "yam_arc_grid/bc_rl2_stattempo_slowpace366_arcdur_abstime_m200_d033": "arc_real/stationary_duration", + "yam_arc_grid/bc_rl2_stattempo_sp366e51_arcdur_abstime_m200_d022": "arc_real/stationary_duration", + "yam_arc_grid/earbuds_plus49_arcdur_abstime_m200_d022_hpt300": "arc_real/earbuds_duration", + "yam_arc_grid/earbuds_updated_arcdur_abstime_m200_d033_hpt300": "arc_real/earbuds_duration", + "yam_arc_grid/earbuds_updated_time_hpt300": "arc_real/earbuds_time", + "yam_arc_grid/towels_new290plus74_time_hpt300": null, + "yam_arc_grid/towels_since0924plus74_arcdur_m200_d033_hpt300": "arc_real/towels_duration", + "yam_arc_grid/towels_since0924plus74_time_hpt300": "arc_real/towels_time" +} diff --git a/docs/experiments/action-flow-merge-readiness.md b/docs/experiments/action-flow-merge-readiness.md index 332b85ed35..439333bb70 100644 --- a/docs/experiments/action-flow-merge-readiness.md +++ b/docs/experiments/action-flow-merge-readiness.md @@ -44,8 +44,9 @@ within an arbitrary grace period. The old failed ICE smoke remains failed; run a readiness-aware scheduled smoke before relying on preemption recovery for a new cluster run. No training jobs or merge into main were requested here. -GitHub access uses Skynet's authenticated `ElmoPA` identity through `sky2`, the -working fallback after `sky1` stalled. Publication includes the previously +For that historical publication, GitHub access used the `ElmoPA` identity through +`sky2` after `sky1` stalled; this is provenance, not an account requirement for +future submissions. Publication includes the previously unpublished gradient audit, sphere/cube experiment, stop-gradient ablation, and requeue commits. Existing experimental checkpoints remain bound to their original source revisions. diff --git a/docs/integration/CONFIG_DEDUPLICATION.md b/docs/integration/CONFIG_DEDUPLICATION.md index 32f1f29047..00b6e44c41 100644 --- a/docs/integration/CONFIG_DEDUPLICATION.md +++ b/docs/integration/CONFIG_DEDUPLICATION.md @@ -1,5 +1,11 @@ # Model configuration consolidation: first local layer +> Historical October 1 audit, not the current config inventory. Its alias +> names, counts and review status describe that source snapshot. Current +> family folders, shared bases and task entry points are documented in +> [CONFIGURATION.md](../CONFIGURATION.md). Preserve this report as evidence; +> do not use its old selectors for new runs. + This is a partial implementation of the October 1 consolidation handoff. The complete A/#197, B/#159 and C/#160 integration has **not** been assembled, reviewed, pushed or merged. Main and the published source branches are unchanged. @@ -28,11 +34,11 @@ available, while 1,356 duplicated YAML lines are removed. | Compatibility name (`model=`) | Canonical definition (`model=`) | | --- | --- | -| `hpt_bc_flow_aria` | `hpt_bc_flow_human` | -| `hpt_cotrain_scale_flow_shared_head` | `hpt_cotrain_mecka_flow_shared_head` | -| `pi0.5_bc_scale` | `pi0.5_bc_mecka` | -| `pi0.5_cotrain_mecka_scale` | `pi0.5_bc_mecka` | -| `pi05/pi0.5_ft_abc_eva_6d` | `pi05/pi0.5_bc_abc_eva_6d` | +| `hpt/bc_flow_human` | `hpt/bc_flow_human` | +| `hpt/cotrain_mecka_flow_shared_head` | `hpt/cotrain_mecka_flow_shared_head` | +| `pi/bc_mecka_legacy` | `pi/bc_mecka_legacy` | +| `pi/bc_mecka_legacy` | `pi/bc_mecka_legacy` | +| `pi/bc_abc_eva_6d` | `pi/bc_abc_eva_6d` | Live launch and normalization scripts use the canonical names. Dataset selection stays explicit; choosing a shared model does not change the selected diff --git a/docs/integration/DATA_TOOLS.md b/docs/integration/DATA_TOOLS.md index fb084e40c9..ff7bf30b05 100644 --- a/docs/integration/DATA_TOOLS.md +++ b/docs/integration/DATA_TOOLS.md @@ -1,5 +1,9 @@ # Data and visualization tools +Historical test commands below refer to their recorded source revisions. +The current checkout keeps checks outside the repository; see the +[verification policy](../VERIFICATION_POLICY.md). + Owner: graph integration. These tools use recorded data and do not load a model, download pretrained weights, fit normalization statistics, or alter episodes. @@ -35,8 +39,9 @@ The old scripts depended on absent `rldb.utils` and `egomimicUtils` imports at the audit baseline; copying them would not preserve a runnable workflow. For wrist-frame targets, configure the renderer's `transform_list` explicitly. -Prediction-renderer selections such as `pi_cartesian_lang_wrist` remain usable -with `evaluator=eval_hpt_wrist`; keypoints use `eval_keypoints_wrist`. These +Prediction-renderer selections such as `evaluator/viz/cartesian_prompt` remain +usable with `evaluator=eval_cartesian_wrist`; keypoints use +`eval_keypoints_wrist`. These renderer selections expect the evaluator to perform coordinate reversion, as in the original runtime. They do not independently apply it a second time. @@ -81,9 +86,6 @@ require SQL, cloud packages or a policy backend. ## Evidence and remaining gates -The following regression evidence is historical; test sources are no longer -shipped. See [the verification policy](../VERIFICATION_POLICY.md). - `tests/test_data_tool_parity.py` converts an actual HDF5 fixture, checks unchanged source bytes and overwrite rejection, imports with cloud/model modules blocked, encodes a complete annotated episode, validates video frame count/FPS and saved diff --git a/docs/integration/MIGRATION.md b/docs/integration/MIGRATION.md index 5871735b5e..0f9c18f8c7 100644 --- a/docs/integration/MIGRATION.md +++ b/docs/integration/MIGRATION.md @@ -24,22 +24,32 @@ verification and separate Yam hardware environment. ## Select a complete recipe -The general `train_zarr_cartesian` entry point requires explicit model, data and -evaluator choices. For example, after configuring credentials and dataset paths: +Prefer a complete task entry point; its defaults explicitly select model, data, +evaluator, trainer, callbacks and logging. Inspect without launching: + +```bash +python egomimic/trainHydra.py \ + +experiment=arc_real/stationary_velocity --cfg job --resolve +``` + +See [the configuration guide](../CONFIGURATION.md) for the arc_real/arc_sim menus, +family-organized model configs and behavior-named evaluators. The general +`train_zarr_cartesian` entry point also supports explicit component selection. +For example, after configuring credentials and dataset paths: ```bash python -m egomimic.trainHydra --config-name=train_zarr_cartesian \ - model=hpt_bc_flow_eva data=eva evaluator=eval_hpt + model=hpt/bc_flow_eva data=eva evaluator=eval_cartesian_camera ``` The PI convenience entry point retains its concrete ABC/6D/wrist-frame default -and requires an explicit pretrained weight path. Root retained recipes can be +and requires an explicit pretrained weight path. Retained legacy recipes can be selected on the general entry point too: ```bash export EGOVERSE_PI05_WEIGHTS=/absolute/path/to/pi05_base_pytorch python -m egomimic.trainHydra --config-name=train_zarr_cartesian \ - model=pi0.5_bc_eva data=eva_pi evaluator=eval_pi + model=pi/bc_eva_legacy data=eva_pi evaluator=eval_cartesian_camera_prompt ``` PI weights/tokenizers must be available at the selected model's configured @@ -52,8 +62,9 @@ it contradicts that contract. The nested PI recipes expose `model.coordinate_frame=camframe|eef_frame`; it must match the data selection and the evaluator's frame reversion. The model declaration also chooses the deployment frame, so a station cannot silently reinterpret wrist-relative -actions as camera-frame actions. E1's `e1.policy_domain` is explicit experiment -metadata and binds its source identity and data contract. +actions as camera-frame actions. Contracts bind the selected model and data +semantics without inferring them from folder names. Removed E1 recipes must +use their original source; they are not current launch choices. The keypoint wrist entry point shares the retained keypoint architecture but selects a wrist-frame model contract. Nested Yam/ARC models declare padded diff --git a/docs/integration/RECIPE_PARITY.md b/docs/integration/RECIPE_PARITY.md index c0e6e12fca..f8d86654ac 100644 --- a/docs/integration/RECIPE_PARITY.md +++ b/docs/integration/RECIPE_PARITY.md @@ -26,7 +26,7 @@ Two stale source settings are corrected explicitly: `actions_joints`, while their shipped keymaps/transforms only produced Cartesian state/actions. The retained Cartesian recipes bind those actual keys. This is not a joint-action checkpoint converter. -- `hpt_cotrain_enc_dec_base` declared eight decoder context tokens against +- `base/hpt_cotrain_enc_dec` declared eight decoder context tokens against a 64-token trunk. Its deterministic decoder now declares 64 context tokens; its smooth-L1 loss retains beta 0.05. It rejects mismatched target shapes. diff --git a/docs/integration/THREE_SOURCE_MERGE.md b/docs/integration/THREE_SOURCE_MERGE.md index 534b954081..a0bf8a1b49 100644 --- a/docs/integration/THREE_SOURCE_MERGE.md +++ b/docs/integration/THREE_SOURCE_MERGE.md @@ -73,7 +73,7 @@ parity retest passes seven cases. The next full run reached a terminal result: ongoing repairs and is not a final-head gate. Its failures remain recorded. This runner does not change cluster execution or establish exact-runtime or historical-run parity. -Publication through authenticated Skynet Graphite and the final GitHub merge are +Publication through Graphite and the final GitHub merge are still incomplete. No final integration, deployment, training or GPU parity claim is made by this progress record. diff --git a/egomimic/eval/AGENTS.md b/egomimic/eval/AGENTS.md index 1f8417d2f0..ad0f2d80bd 100644 --- a/egomimic/eval/AGENTS.md +++ b/egomimic/eval/AGENTS.md @@ -14,7 +14,9 @@ The root AGENTS.md applies here. Configure evaluators in DTW, Fréchet, reverse-KL and coverage calculations. - `arc_bimanual_cartesian_eval.py`, `arc_metrics.py`: ARC-specific decoding and trajectory alignment layered on the shared Cartesian evaluator. -- `bimanual_tempo_eval.py`, `e1_metrics.py`: tempo/duration scoring. +- `open_loop_stream.py` (stable exports in `open_loop_sim.py`), + `distance_budget_dtw.py`: shared first-moving-stream + simulation and trajectory scoring with the canonical Cartesian ARC codec. - `planar_action_eval.py`, `synthetic_trajectory_eval.py`: nonrobot evaluation. - `checkpoint_loading.py`: strict graph checkpoint restoration and EMA selection. - `token_diagnostics.py`, `latent_archive.py`: provider-owned token capture, @@ -29,5 +31,5 @@ are YAML settings, including the `train_viz` group. Frame reversion must match the data configuration. Rotation encodings and gripper padding affect both actions and proprio; test both sides of the round -trip. Keep regression checks outside the repository and report local results; -do not commit tests or their fixtures. +trip. Keep regression checks and fixtures outside the repository; never commit +tests. Report local verification results in the PR. diff --git a/egomimic/rldb/AGENTS.md b/egomimic/rldb/AGENTS.md index 647fa4d732..3b4a088336 100644 --- a/egomimic/rldb/AGENTS.md +++ b/egomimic/rldb/AGENTS.md @@ -12,12 +12,23 @@ The root AGENTS.md applies here. Dataset/experiment selection belongs in conventions. `action_mode`, `coord_frame`, `rotation_mode`, camera mappings, local-frame corrections and calibration inputs come from YAML. - `embodiment/bimanual_arc.py`: raw-window horizons and time/ARC target choices - used by the E1 recipes. `e1_fold.py` is an old import name only. + used by the current Cartesian task recipes. - `zarr/action_chunk_transforms.py`: shared pose, rotation, gripper, interpolation and key-routing transforms. Add general transform behavior here. -- `zarr/arc_length_tokenizer.py`, `zarr/e1_arc_tokenizer.py`: the two ARC timing - representations. Check their layouts before interpreting the last channels. -- `zarr/e1_resolvers.py`, `zarr/e1_anchor_sampler.py`: metadata overrides and +- `zarr/cartesian_arc.py` (stable exports in `arc_length_tokenizer.py`): Cartesian + ARC with `clock_mode` velocity or duration (absolute arrival seconds, + anchor-inclusive speed-bounded isotonic decode). `streams` supports + full_separation (M×20), translation_rotation (M×18), translation or rotation + (M×16, one clock per arm). Row zero is current measured pose to first future + target, never a start delay. The finite same-frame 14D `arc.current_action` + anchor is required separately; source rows begin at idx+1 / time dt, and + decoded controls are sampled at dt, 2dt, … . Every PW row is an incoming + speed. No synthetic 400-frame holds or projection mode are supported. + A zero-progress resampling fallback spans actual source times; fully static + PW needs the caller's explicit horizon because zero rates contain no time. + D/R budgets cover the future source path, excluding the incoming anchor + segment. See `docs/ARC_CLOCK_CONTRACT.md`. +- `zarr/embodiment_resolvers.py`, `zarr/progress_sampler.py`: metadata overrides and configurable anchor sampling used by the tempo recipes. - `filters.py`: dataset query builder. Filter values belong in the selected YAML. - `zarr/selection.py`: pinned random/paired/custom diagnostic selections and @@ -33,5 +44,5 @@ embodiment IDs when loading a cache. Live rollouts require the full `normalizer_state`, not just numeric bounds. Do not change stored episode metadata to rename an embodiment. Canonicalize the -metadata at the loader boundary. Verify geometric and ARC/PI changes with local -CPU checks outside the repository; do not commit tests or their fixtures. +metadata at the loader boundary. Keep regression checks and their fixtures outside +the repository; never commit tests. Record local verification results in PRs. diff --git a/egomimic/robot/AGENTS.md b/egomimic/robot/AGENTS.md index ac91d63802..124ba06cf8 100644 --- a/egomimic/robot/AGENTS.md +++ b/egomimic/robot/AGENTS.md @@ -45,6 +45,12 @@ for Eva/Yam setup, Quest app build requirements, data format and operator comman activation after the open-then-squeeze gate, and validate-both-before-commanding guard. Do not add a GELLO repo dependency. - `rollout.py`: shared rollout loop. Inference uses only the local graph path. + Preserve main's failed-model-selection lockout; never resume the old model + implicitly. HDF5 episode capture must enqueue existing observations/commands + without adding a motion path or blocking a control tick on disk writes. +- `rollout_dashboard.py`: the server-owned monotonic timer starts at Continue, + excludes explicit pauses and camera/model holds, survives browser reconnects, + and resets on Restart. Keep numeric/boolean/enum controls model-owned. - `virtual_gello_teleop.py`: MuJoCo-only followers with generated or passive real GELLO inputs. Reuse the shared leader reader/controller and the supplied combined calibration/home; absolute checks must preserve full-turn errors. @@ -52,11 +58,30 @@ for Eva/Yam setup, Quest app build requirements, data format and operator comman or Quest to this diagnostic. Synthetic headless checks open no devices. - `graph_policy.py`: strict PipelineAlgo checkpoint loading, full normalization state, camera/proprio mapping and explicit Cartesian action-frame conversion. + Station YAML must not contain legacy inline `inference_graph` profiles or + inferred sampler defaults. Keep model semantics in the checkpoint-bound + model declaration and prefer full `data-context.json` bundle sidecars. + Documentation-only inference fragments belong under `docs/examples`, not + the selectable Hydra model tree; a fragment is not a complete model recipe. - `replay_policy.py`: read-only Zarr joint replay, including split `rl2_yam.episode.v1` actions; require completed stores and bound reads by `committed_samples`/`total_frames` and EOF. -- `arc_decoder.py`: decode native ARC predictions before frame conversion and IK. - E1 timing channels follow all 14 pose channels; they are not interleaved. +- `stream_decoder.py` (stable exports in `arc_decoder.py`): decode native ARC predictions before frame conversion and IK. + Two, four or six timing channels follow all 14 pose channels; they are not interleaved. + Decoder settings belong to explicit model YAML `decoder_attribute` controls; + never append controls based on a model class or inferred token width. The + decoders use the live `egomimic.rldb.zarr` tokenizers only; no checkpoint-era + codec copies are carried. Anything the codec needs for deployment + (``detokenize(times=)``, `stream_clocks`, `stream_ends`) is decode-only; never + change tokenize for the decoder. Generic inference consumes an optional + `execution_steps()` capability, not ARC-specific statistics. Keep immutable + DataContext/checkpoint binding and schema-2 artifacts; do not restore the + legacy inferred exporter/JSON loader. +- `ik_options.py`, `yam/rollout_ik.py`, `yam/tracik.py`: opt-in pose + relaxation, bounded differential recovery and the TRAC-IK endpoint backend. + MuJoCo stays the default; every candidate still passes the joint-step guard. +- `camera_align/`, `hdf5_tempo.py`: station monitors with no robot writes; the + alignment monitor holds the D405 when started with `--source realsense`. - `oculus_reader/`: the pinned RAIL/Yam APK, matching source, and Python reader. The bundled app emits tracking-origin/world poses under the established `wE9ryARX` tag; the reader is world-only and rejects a head-frame mode. Keep the @@ -70,7 +95,7 @@ Use the same graph, data normalization and evaluation components for HPT and PI. Do not introduce an embodiment-specific inference server or upstream runtime launcher. Do not deprecate Eva when adding another robot implementation. -Keep CPU verification outside the repository; do not commit tests. `--help`, GELLO +Keep CPU verification outside the repository; never commit tests. `--help`, GELLO `--check-config`/`--list-ports`, and uploader `--list` open no devices. Collection/rollout commands open hardware; operate a physical station only when the user requests that operation. Preserve source branches, user worktrees and diff --git a/training.md b/training.md index 35998ff880..c2288e3beb 100644 --- a/training.md +++ b/training.md @@ -13,7 +13,21 @@ Its model, data, and evaluator defaults are intentionally null: every run must select a complete experiment instead of inheriting a historical policy by accident. -A direct composition has this shape: +Prefer a task-first experiment. Each public real recipe links all components +needed by the training entry point: + +```bash +python egomimic/trainHydra.py \ + +experiment=arc_real/stationary_velocity --cfg job --resolve +``` + +This prints the configuration without training. Substitute `stationary_duration` +or `stationary_time`, or select a task from the +[experiment menu](egomimic/hydra_configs/experiment/README.md). +See [CONFIGURATION.md](docs/CONFIGURATION.md) for shared parents, model families +and behavior-named evaluators. + +For advanced component selection, a direct composition has this shape: ```bash python egomimic/trainHydra.py \ @@ -62,6 +76,20 @@ not inferred, with `train_only` or its mirror `inference_only`. ## Action tokenization +### Cartesian bimanual ARC + +Real ARC experiments select `abc.arc_clock_mode=velocity|duration` and +`abc.arc_streams=full_separation|translation_rotation|translation|projection`. +Velocity stores start delay followed by incoming per-waypoint rates; duration +stores absolute waypoint seconds, including start delay in row zero. Shapes +are M×20, M×18 or M×16 according to stream separation. The tokenizer and +first-moving-stream decoder share this contract. See +[ARC_CLOCK_CONTRACT.md](docs/ARC_CLOCK_CONTRACT.md) for geometry, timing, +holds and checkpoint compatibility. + +### Planar ARC + +The following is the separate planar codec, not the Cartesian clock API above. Arc-length tokenization can run either in the loader's `transform_list` or as graph nodes. The node form makes the boundary visible and lintable: @@ -99,8 +127,8 @@ episode's 30 Hz frames and the tokenizer, and `arc_dt` is the true capture period. A loader-side arc path that resamples before tokenizing would understate arc length and make that constant wrong. -Compare `experiment/pusht/planar_v2_usocket_arc_bc` (tokenized in the loader) -with `experiment/pusht/planar_v2_usocket_arc_graph_tok` (tokenized in the +Compare `experiment/arc_sim/pusht/planar_v2_usocket_arc_bc` (tokenized in the loader) +with `experiment/arc_sim/pusht/planar_v2_usocket_arc_graph_tok` (tokenized in the graph). Both are valid; only the second shows the tokenizer in `gt`-style graph output and in `tools/config_graph.py`. @@ -123,14 +151,16 @@ They support annotation and embedding utilities and do not select a policy. ## Slurm -After a configuration composes and passes its required smoke test, add `-m` to -use the configured Submitit launcher: +After a configuration composes and passes its required smoke test, select the +reviewed Submitit launcher explicitly and add `-m`, even for one run: ```bash -python egomimic/trainHydra.py --config-name=train_zarr_cartesian \ - model= data= \ - evaluator= -m +python egomimic/trainHydra.py -m \ + +experiment=arc_real/stationary_velocity \ + hydra/launcher= ``` -Always inspect the fully resolved config saved with the run before treating a -launch as reproducible. +This command submits a job; run it only after launch authorization and +cluster-specific preflight. Verify GPU/task layout and effective global batch +in the resolved launcher configuration. Always retain the fully resolved +config, source commit and normalization identity with the run.