config: consolidate experiment entrypoints and component names

AGENTS.md

@@ -33,7 +33,7 @@ datasets, normalizers, environments, recorded source, and unverified checkpoints
 Use bounded task directories; never discover files by recursively scanning a
 whole scratch/project root. Candidate inventory alone does not authorize deletion.
 
-Publish source from the authenticated Skynet account with Graphite `gt submit`,
+Publish source with Graphite `gt submit` using the user's authorized GitHub account,
 preserving branch parents. Do not merge without an explicit user request.
 
 ## Aidan stationery source

README.md

@@ -6,6 +6,8 @@ stack for training and evaluation.
 For cluster training and maintenance, start with
 [Pipeline and cluster operations](docs/cluster-pipeline.md). It distinguishes
 current source, historical run checkouts, and shared data/runtime dependencies.
+For current task recipes and the data/model/evaluator configuration layout,
+start with the [configuration guide](docs/CONFIGURATION.md).
 For the integration branch, start with the [migration guide](docs/integration/MIGRATION.md)
 for the locked environment, retained HPT/PI recipes, data tools, and checkpoint
 rules. [Integration status](docs/integration/PLAN.md) records the draft stack and
@@ -146,14 +148,22 @@ python egomimic/scripts/data_download/sync_s3.py \
 ```
 
 ### Training
-Select a complete Pipeline experiment; the root config deliberately has no
-implicit model, data, or evaluator:
-``` bash
-python egomimic/trainHydra.py --config-name=train_zarr_cartesian \
-  model=<pipeline-model-config> data=<dataset-config> \
-  evaluator=<evaluator-config>
-```
-For full instructions on training see [``training.md``](./training.md)
+Select a complete task recipe. Each public real experiment explicitly links its
+model, data, evaluator, trainer, callbacks and logger. Inspect it without
+starting training:
+
+```bash
+python egomimic/trainHydra.py \
+  +experiment=arc_real/stationary_velocity --cfg job --resolve
+```
+
+The [experiment menu](egomimic/hydra_configs/experiment/README.md) lists
+`baseline`, `arc_vel` and `arc_dur` for each real task. A cluster launch adds
+`-m hydra/launcher=<approved-submitit-config>` after resource/data preflight;
+machine-specific paths and credentials must be configured. Advanced direct
+`model=... data=... evaluator=...` composition remains supported; the root
+config deliberately does not select those components implicitly.
+See [training.md](./training.md) for the graph and launch workflow.
 
 ### Converting your own data
 See [``embodiment_tutorial.ipynb``](./egomimic/scripts/tutorials/embodiment_tutorial.ipynb) as reference to write a conversion script for your own data.

docs/ARC_CHUNKING_MODES.md

@@ -3,7 +3,8 @@
 > Historical reference, not a current API or launch menu. Removed E1/legacy
 > codecs, configs and launchers described below require their recorded source.
 > Current usage: [clock contract](ARC_CLOCK_CONTRACT.md) and
-> [experiment menu](../egomimic/hydra_configs/experiment/README.md).
+> [experiment menu](../egomimic/hydra_configs/experiment/README.md), with the
+> current folder layout in [CONFIGURATION.md](CONFIGURATION.md).
 
 Every retained `abc_arc/robot_bc` and `abc_arc/human_bc` visual experiment
 accepts `abc.arc_chunking_mode=race`, `multistream`, or `joint_distance`.

docs/ARC_CLOCK_CONTRACT.md

@@ -1,10 +1,10 @@
-# Cartesian ARC: one codec, two clocks, two stream layouts
+# Cartesian ARC: one codec, two clocks, four stream layouts
 
 This contract replaces the historical Cartesian mean/wide/stacked/clock/E1
 families. It builds on PR #263 and preserves the local first-row-delay
 implementation. It does not change the separate planar or Libero codecs.
 
-New run presets are grouped by task under `hydra_configs/experiment/`:
+New run presets are grouped by task under `hydra_configs/experiment/arc_real/`:
 `organize_stationary`, `fold_and_stack_towels`, `wireless_earbuds`, and
 `multitask`, each with `arc_vel`, `arc_dur`, and `baseline`.
 See the [experiment menu](../egomimic/hydra_configs/experiment/README.md)
@@ -47,8 +47,8 @@ represent independent gripper movement.
 ## Timing rows
 
 Row zero of every timing column represents that stream's initial delay.
-Rows i > 0 represent the incoming interval from waypoint i-1 to waypoint i.
-There is no repeated final padding clock.
+The meaning of later rows depends on `clock_mode`: incoming rates for velocity,
+absolute waypoint seconds for duration. There is no repeated final padding clock.
 
 - velocity: row zero is seconds; later rows are the distance between emitted
   supports divided by the source interval's seconds. Units are m/s, rad/s,
@@ -62,15 +62,17 @@ There is no repeated final padding clock.
   `first_delay_absolute_v1`; velocity keeps `first_delay_incoming_v1`.
 
 Translation distances are Euclidean, rotation distances are SO(3) geodesic,
-and gripper distances are absolute value differences. No velocity smoothing,
-physical speed clamp, or decoder-duration floor is introduced.
+and gripper distances are absolute value differences. Velocity-mode rates are
+not smoothed or physically speed-clamped. Duration-mode clock projection does
+enforce its explicit 5 m/s translation bound. Neither mode adds an arbitrary
+decoder-duration floor.
 
 An initial wait retains constant geometry before the first movement. Rotation
 and distance-gripper target times use first arrival, so a final stationary
 plateau does not stretch the last moving interval.
 
 Full-window holds retain constant waypoints. Velocity interval rates are zero.
-Duration intervals span the configured hold_window_frames × dt. Constant
+Duration absolute clocks span the configured hold_window_frames × dt. Constant
 streams do not terminate a chunk. Internal dwells are not explicitly allocated
 duplicate supports; zero speed alone cannot encode a zero-distance dwell time.
 
@@ -82,7 +84,8 @@ Command-change supports and endpoint padding retain the prior local semantics.
 Both use the same codec and `control_steps()`:
 
 1. Keep an exact integer waypoint prefix, e.g. 30 of M100.
-2. Decode every stream using its own start delay and incoming intervals.
+2. Decode every stream using its selected clock representation: start delay
+   plus distance/rate intervals, or absolute waypoint seconds.
 3. Replan when the first moving stream reaches its prefix endpoint.
 4. Ignore fully constant streams in this race.
 
@@ -113,12 +116,14 @@ quantile widening (`norm_stats.widen_degenerate_quantiles: true`).
 
 The first-delay six-stream refactor was compared numerically against its
 preceding local implementation across 32 synthetic clock/gripper/hold cases,
-including token, detokenizer, and termination parity. New regression tests cover
-all four clock/stream combinations and validation/rollout agreement.
+including token, detokenizer, and termination parity. Regression tests cover
+clock/stream combinations and validation/rollout agreement.
 
-Verification for this migration: 536 focused CPU tests passed, including
+Historical verification for the initial two-layout migration: 536 focused CPU
+tests passed, including
 normalization-cache rejection, baseline graph forward/backward, source adapters,
-and planar ARC regressions. All 379 shipped Hydra YAML files composed.
+and planar ARC regressions. All 379 Hydra YAML files shipped at that revision
+composed; these are historical counts, not the current configuration inventory.
 The 32 six-stream comparisons used predecessor
 `f364923a3e675106882823da17f840a9505ab972`, with token tolerance 1e-10,
 decoded-trajectory tolerance 1e-9, and identical execution-step counts.

docs/ARC_OAT.md

@@ -450,18 +450,18 @@ examples use the measured LIBERO-10 selections in the
 [dated replay report](results/libero_arc_timed_replay_20260921.md):
 
 ```sh
-python -m egomimic.trainHydra +experiment=sim/oat/libero_oattok \
+python -m egomimic.trainHydra +experiment=arc_sim/oat/libero_oattok \
   benchmark.suite=libero_10 benchmark.dataset=/datasets/arc_oat/libero_10.zarr \
   hydra.run.dir=/runs/tokenizer
-python -m egomimic.trainHydra +experiment=sim/oat/libero_oatpolicy \
+python -m egomimic.trainHydra +experiment=arc_sim/oat/libero_oatpolicy \
   benchmark.suite=libero_10 benchmark.dataset=/datasets/arc_oat/libero_10.zarr \
   benchmark.tokenizer_checkpoint=/runs/tokenizer/checkpoints/last.ckpt \
   hydra.run.dir=/runs/oat
-python -m egomimic.trainHydra +experiment=sim/oat/libero_arc_dur_policy \
+python -m egomimic.trainHydra +experiment=arc_sim/oat/libero_arc_dur_policy \
   benchmark.suite=libero_10 benchmark.dataset=/datasets/arc_oat/libero_10.zarr \
   benchmark.arc_waypoints=32 benchmark.arc_max_translation=0.8 \
   benchmark.arc_max_rotation_degrees=192 hydra.run.dir=/runs/arc_dur
-python -m egomimic.trainHydra +experiment=sim/oat/libero_arc_stk_policy \
+python -m egomimic.trainHydra +experiment=arc_sim/oat/libero_arc_stk_policy \
   benchmark.suite=libero_10 benchmark.dataset=/datasets/arc_oat/libero_10.zarr \
   benchmark.arc_waypoints=32 benchmark.arc_max_translation=1.6 \
   benchmark.arc_max_rotation_degrees=128 hydra.run.dir=/runs/arc_stk

docs/CONFIGURATION.md

@@ -0,0 +1,139 @@
+# Configuration guide
+
+Start with a task experiment, then follow its Hydra `defaults` links. Folder
+names organize recipes; Python `_target_` declarations identify implementations.
+The generic pipeline does not choose behavior from a model or task name.
+
+```text
+egomimic/hydra_configs/
+├── experiment/
+│   ├── arc_real/
+│   │   ├── stationary_{time,duration,velocity}.yaml
+│   │   ├── towels_{time,duration,velocity}.yaml
+│   │   └── earbuds_{time,duration,velocity}.yaml
+│   ├── abc_arc/robot_bc/     retained ABC-lab recipes, including multitask
+│   └── arc_sim/              PushT, LIBERO, OAT and synthetic recipes
+├── data/                     episode selection, transforms and loaders
+├── model/
+│   ├── base/                 shared model definitions and settings
+│   ├── hpt/                  HPT recipe leaves
+│   ├── pi/                   PI recipe leaves
+│   └── …                     distinct non-HPT/PI model families
+├── model_contract/           shared deployment and data contracts
+├── evaluator/
+│   ├── eval_open_loop_sim.yaml
+│   ├── eval_cartesian_*.yaml  behavior/frame-specific metrics and videos
+│   ├── _base/                inherited evaluator defaults
+│   └── viz/                  reusable renderers
+└── hydra/launcher/            cluster allocation, separate from experiment
+```
+
+## Select and inspect one experiment
+
+The three real tasks each expose `_time`, `_velocity` and `_duration`:
+nine standard entry points, preserving the current task pools and protocols.
+Every leaf directly links model, data,
+evaluator, trainer, callbacks and logger. The model owns the optimizer and
+scheduler. Real recipes do not inherit other experiments: each contains its
+task/run overrides and imports reusable definitions from the component groups.
+
+Activate the project environment, configure required data-cache/weight paths,
+then inspect without running training:
+
+```bash
+python egomimic/trainHydra.py \
+  +experiment=arc_real/stationary_velocity --cfg job --resolve
+```
+
+The [experiment menu](../egomimic/hydra_configs/experiment/README.md) records
+task pools and numeric defaults. The top-level experiment directories `abc/`
+and `abc_arc/human_bc/`, and `arc_real/arcdur_abstime/`, are removed. Their old
+commands are historical, not aliases for the retained task recipes. The
+removed absolute-time recipe menu does not remove duration mode:
+`arc_real/<task>_duration` uses absolute waypoint seconds.
+
+For an authorized cluster launch, use that same experiment with `-m` and an
+explicit approved `hydra/launcher=...` selection. Verify the resolved resource
+request, effective global batch, pinned data and normalization before launch.
+See [training.md](../training.md) and [cluster operations](cluster-pipeline.md).
+
+## What belongs in each layer
+
+| Layer | Owns | Does not imply |
+| --- | --- | --- |
+| `experiment/arc_real/<task>_<mode>` | Complete component selection, task values and run schedule | Another task-specific Python runtime |
+| `data/` | SQL filters or pinned episode lists, splits, transforms, loader behavior | The model architecture |
+| `model/hpt/`, `model/pi/` | Model graph and family-specific recipe overrides | A model-specific evaluator implementation |
+| `model/base/` | Identical inherited graphs/settings factored out of leaves | An experiment with data and evaluation selected |
+| `model_contract/` | Reusable inference/deployment and preprocessing declarations | Permission to reinterpret a checkpoint |
+| `evaluator/` | Metrics, frame reversion, execution protocol and artifacts | Training-time tokenization choices inferred from a model name |
+
+`model/base/` contains shared HPT flow, cotrain, keypoint and Qwen definitions,
+PI defaults, and YAM HPT/diffusion settings. Distinct architectures and
+contracts remain distinct; only equivalent definitions should be shared.
+The Python networks stay in `egomimic/models/`, and configured graph stages
+stay in `egomimic/pipeline/`. Moving YAML does not move or resize a network.
+
+### Direct component links; no shared experiment parents
+
+`experiment/_shared/` and the real experiment-parent chain are removed. Every
+one of the nine `arc_real/` recipes selects its six components directly. Model
+dimensions live in `model/base/real_*_settings`, training defaults in
+`trainer/real_*`, and the checkpoint schedule in
+`callbacks/real_yam`. Task identity, dataset selection, caps and run
+overrides remain visible in the experiment itself.
+
+Canonical clock dimensions/provenance live in
+`model_contract/arc_clock_settings`. YAM ARC experiments explicitly
+select `model/hpt/yam_300m_arc` and the matching `data/abc_arc/*_arc`
+adapter; the data adapter retains the original pinned pool and binds the
+model-declared transforms through `data/_base/yam_arc`.
+
+Component inheritance still deduplicates model graphs and dataset lists.
+There is no earbuds-to-towels-to-stationery experiment inheritance, and no
+extra experiment file is needed to launch any real recipe. Simulation recipes
+retain their existing parent structure under the renamed `arc_sim/` folder.
+
+## Evaluator selection
+
+| Selection | Purpose |
+| --- | --- |
+| `eval_open_loop_sim` | Recorded-episode execution simulation for real baseline and ARC recipes; metrics, DTW and executed-chunk videos |
+| `eval_bimanual_cartesian` | Time-indexed chunk errors and overlays, without sequential execution/replanning |
+| `eval_cartesian_chunk_metrics` | Metrics-only chunk comparison using the configured YAM batch budget; no renderer |
+| `eval_cartesian_camera`, `eval_cartesian_wrist` | Cartesian pose metrics with the chosen frame reversion |
+| `eval_cartesian_*_prompt` | Corresponding metrics with sampled-prompt overlays; `6d` variants use continuous-6D rotations |
+
+For example, the old `eval_hpt` selector becomes `eval_cartesian_camera`, and
+the old root `eval_pi` becomes `eval_cartesian_camera_prompt`. A model name is
+not sufficient to choose the right evaluator: frame, rotation encoding,
+ground-truth key and execution protocol must match the data/model contract.
+Renderer-only defaults live in `evaluator/viz/`; common evaluator definitions
+live in `evaluator/_base/`.
+
+`eval_open_loop_sim.yaml` groups settings by execution, canonical codec,
+complete-episode scoring and artifacts. Real baselines predict 100 actions,
+execute the first 25 at 30 Hz, then replan against the next recorded
+observation. ARC keeps a waypoint prefix and ends when the first moving stream
+finishes. This is open-loop replay of recorded observations, not closed-loop
+physics or robot evaluation. Ordered complete episodes and matching video/GT
+coverage are part of its contract, not optional batch-sampling details.
+
+The [Cartesian clock contract](ARC_CLOCK_CONTRACT.md) defines velocity versus
+absolute-seconds duration, four stream layouts and start-delay semantics.
+Planar and LIBERO codecs have their own contracts; the removed Cartesian
+legacy options must not be reintroduced into the open-loop YAML.
+
+## Provenance and migration
+
+Use the new selectors for new compositions. Keep the original source commit,
+resolved config, normalization and checkpoint for an existing run. Do not
+resume a different action representation under the same run identity or infer
+decoder compatibility from M×20/M×18/M×16 shape alone. YAML organization does
+not migrate trained weights. See [checkpoint rules](integration/CHECKPOINTS.md)
+and [current real-task recipes](STATTEMPO.md).
+
+The exact old-to-new model, evaluator and data selector map is in
+[`config_cleanup_migration.json`](config_cleanup_migration.json). Removed
+experiment menus have no automatic replacement: choosing another task recipe
+could change its data pool or training protocol.

docs/CURRENT_FAMILY_PROFILES.md

@@ -61,7 +61,7 @@ selected emitted configs remain separate immutable task artifacts.
 
 ## PushShapes: explicit multiplier versus historical physical speed
 
-AF selects `sim/pusht/action_flow_cotrain_uc_multiplier_interpolation`; its typed
+AF selects `arc_sim/pusht/action_flow_cotrain_uc_multiplier_interpolation`; its typed
 route requires fresh initialization, scalar raw dimensionless multiplier, no
 physical-speed reference, Euler50, batches32+32 and fixed EMA0.9978. The latest
 explicit PushT preference uses normal native per-source image processing, not

docs/LIBERO_ARC_OAT_DP.md

@@ -47,7 +47,7 @@ only the camera/state encoder receives the observation LR.
 
 Use `--arc-backbone oat_dp` with a standalone `--arc-profile` and one ARC mode
 in `scripts/benchmarks/launch_libero_osmo.py`. The new experiment is
-`sim/oat/libero_arc_oat_dp_policy`; the existing default remains the U-Net.
+`arc_sim/oat/libero_arc_oat_dp_policy`; the existing default remains the U-Net.
 
 A fresh policy may reuse `--arc-replay-runs-file` without `--resume-from-run`.
 The worker still checks the complete replay confirmation, chosen R/D/M,

docs/LIBERO_ARC_OAT_HYBRID.md

@@ -42,8 +42,8 @@ episodes, two observations, a 32-action horizon and 16 executed actions per
 replan. These are separate experiment runs; previous ARC/OAT weights are not
 used to initialize the models.
 
-The YAML recipes are `sim/oat/libero_arc_oattok` and
-`sim/oat/libero_arc_oatpolicy`. They compose the existing `LiberoArcStage`,
+The YAML recipes are `arc_sim/oat/libero_arc_oattok` and
+`arc_sim/oat/libero_arc_oatpolicy`. They compose the existing `LiberoArcStage`,
 `OATTokenizerStage`/`OATPolicyStage`, and ARC decoder in `PipelineAlgo`.
 The shared trainer, decoded replay loader, normalizer and evaluator are used.
 Validation reports dense controller-action errors after decoding, while the

docs/LIBERO_ARC_REPLAY.md

@@ -60,7 +60,7 @@ selection 2–9, and final testing 35–49. Each mode freezes its own R/D/M choi
 These specs use 16 workers, recycling after four episodes, and 128 GiB memory.
 Use `--replay-spec libero_arc_replay_stk` or `libero_arc_replay_dur` when rendering
 an OSMO workflow. Their corresponding policy recipes are
-`sim/oat/libero_arc_stk_policy` and `sim/oat/libero_arc_dur_policy`.
+`arc_sim/oat/libero_arc_stk_policy` and `arc_sim/oat/libero_arc_dur_policy`.
 
 The earlier LIBERO codec/results are explicitly **`joint_dur`**, a shared-clock
 M×11 baseline. They cannot be reused as STK or independent-clock DUR evidence.