[Docs 2/2] map the feature-sized rebuild and current configuration contracts

AGENTS.md

@@ -1,7 +1,7 @@
 # Repo Agent Rules
 
-See [the verification policy](docs/VERIFICATION_POLICY.md): keep tests and
-verification fixtures outside this repository; never commit them.
+See [the verification policy](docs/VERIFICATION_POLICY.md): do not commit tests,
+fixtures or local verification scripts.
 
 ## Shell / Command Execution
 To run commands in the interactive shell, source `emimic/bin/activate` when it
@@ -39,6 +39,14 @@ whole scratch/project root. Candidate inventory alone does not authorize deletio
 Publish source with Graphite `gt submit` using the user's authorized GitHub account,
 preserving branch parents. Do not merge without an explicit user request.
 
+## Aidan stationery source
+
+For current ARC work, use the feature-sized rebuild rooted on main.
+See [docs/ARC_REBUILD.md](docs/ARC_REBUILD.md); #263–#279 remain reference history.
+Read [docs/STATTEMPO.md](docs/STATTEMPO.md) and the experiment README for the
+retained task recipes and source-preservation rules. Older branches are
+historical run sources. Preserve existing jobs' recorded commits.
+
 ## Current PushT recipes
 
 Read the current selection section in `docs/cluster-pipeline.md` before choosing
@@ -68,10 +76,10 @@ not imply a separate runtime for that task or model.
 | Shared Lightning training and validation hooks | [pl_utils/pl_model.py](egomimic/pl_utils/pl_model.py), [pl_utils/pl_data_utils.py](egomimic/pl_utils/pl_data_utils.py) |
 | Episode loading, metadata, normalization, bounds and sampling | [rldb/zarr/zarr_dataset_multi.py](egomimic/rldb/zarr/zarr_dataset_multi.py) |
 | Embodiment keymaps and coordinate frames | `egomimic/rldb/embodiment/{human,eva,yam}.py`; see [data guide](egomimic/rldb/AGENTS.md) |
-| Generic bimanual ARC inputs used by E1 | [embodiment/bimanual_arc.py](egomimic/rldb/embodiment/bimanual_arc.py) |
+| Generic bimanual time/ARC inputs | [embodiment/bimanual_arc.py](egomimic/rldb/embodiment/bimanual_arc.py) |
 | Pose transforms and action encodings | [zarr/action_chunk_transforms.py](egomimic/rldb/zarr/action_chunk_transforms.py), [utils/pose_utils.py](egomimic/utils/pose_utils.py), [utils/action_encoding.py](egomimic/utils/action_encoding.py) |
 | Shared HPT/PI robot metrics and videos | [eval/bimanual_cartesian_eval.py](egomimic/eval/bimanual_cartesian_eval.py), [eval/video.py](egomimic/eval/video.py); see [evaluation guide](egomimic/eval/AGENTS.md) |
-| ARC and tempo evaluation | `egomimic/eval/{arc_bimanual_cartesian_eval,bimanual_tempo_eval,arc_metrics,e1_metrics}.py` |
+| ARC evaluation | `egomimic/eval/{arc_bimanual_cartesian_eval,open_loop_sim,arc_metrics}.py`; [clock contract](docs/ARC_CLOCK_CONTRACT.md) |
 | Dataset filters, task prompts, camera/frame choices and experiment parameters | `egomimic/hydra_configs/`; calibration matrices in `hydra_configs/calibration/` |
 | Shared Eva/Yam collection, local graph rollout, Zarr replay and upload | [robot/AGENTS.md](egomimic/robot/AGENTS.md), [docs/YAM_RUNTIME.md](docs/YAM_RUNTIME.md) |
 | Offline normalization export for any model | [scripts/data/precompute_norm_stats.py](scripts/data/precompute_norm_stats.py), or `trainHydra.py norm_stats_only=true` |
@@ -100,14 +108,12 @@ rg -n 'class BimanualCartesianEval|class PI05Stage|class MultiDataset' egomimic
 - Evaluators expose `data_requirements()` and `trainer_overrides()`. Complete
   episode requests must reject partial loaders or validation-loop limits before
   model construction. See [the integration contract](docs/GENERIC_PIPELINE_CONTRACT.md).
-- `egomimic/campaigns/pi05/`, `e1_fold.py`, and `pi05_graph_eval.py` retain old
+- `egomimic/campaigns/pi05/` and `pi05_graph_eval.py` retain old
   import names. Their implementations now live at the canonical paths above.
   Use current YAML examples when migrating older serialized configuration APIs.
-- Do not commit tests, test fixtures, or local verification scripts. Keep any
-  regression checks outside the repository and report their results in the PR.
-  Verify normalization, pose conventions and checkpoint loading with local CPU
-  checks before a cluster run; hardware and pretrained-model validation remain
-  separate. Historical test sources remain available in Git history.
+- Keep regression checks and fixtures outside the repository; never commit tests.
+  Verify geometry, normalization and checkpoint behavior locally and report results.
+  Hardware and pretrained-model validation remain separate.
 
 ## Preserve work and verify the target
 

README.md

@@ -3,9 +3,15 @@
 This repository contains EgoVerse data processing plus a dependency-aware Pipeline
 stack for training and evaluation.
 
+For the current review stack and retained/removed features, see the
+[feature-sized ARC rebuild](docs/ARC_REBUILD.md). Tests and local verification
+scripts are not shipped; see the [verification policy](docs/VERIFICATION_POLICY.md).
+
 For cluster training and maintenance, start with
 [Pipeline and cluster operations](docs/cluster-pipeline.md). It distinguishes
 current source, historical run checkouts, and shared data/runtime dependencies.
+For current task recipes and the data/model/evaluator configuration layout,
+start with the [configuration guide](docs/CONFIGURATION.md).
 For the integration branch, start with the [migration guide](docs/integration/MIGRATION.md)
 for the locked environment, retained HPT/PI recipes, data tools, and checkpoint
 rules. [Integration status](docs/integration/PLAN.md) records the draft stack and
@@ -117,6 +123,11 @@ To programatically view the SQL table of all episodes + metadata see [``sql_tuto
 
 #### Interactive Dataset Browser
 
+For data-config → modern multistream ARC tokenization → paired t-SNE, use the
+[offline ARC representation explorer](docs/ARC_TSNE.md). It exports self-contained
+HTML with non-ARC time baselines, optional clock removal, linked anchor selection,
+source trajectories, and configuration provenance. No checkpoint is required.
+
 `latent_inspector.py` also ships a local web app for browsing a **folder of per-episode zarrs** — scrub any episode, overlay the recorded actions (cartesian trajectory / orientation axes / MANO keypoints), and toggle language annotations. Frames are rendered server-side using each episode's `zarr.json` camera intrinsics, so it works for any embodiment (the overlay is drawn by that episode's embodiment class, e.g. `Human`/`Eva`; human poses are projected in the head frame).
 
 ```bash
@@ -141,14 +152,22 @@ python egomimic/scripts/data_download/sync_s3.py \
 ```
 
 ### Training
-Select a complete Pipeline experiment; the root config deliberately has no
-implicit model, data, or evaluator:
-``` bash
-python egomimic/trainHydra.py --config-name=train_zarr_cartesian \
-  model=<pipeline-model-config> data=<dataset-config> \
-  evaluator=<evaluator-config>
-```
-For full instructions on training see [``training.md``](./training.md)
+Select a complete task recipe. Each public real experiment explicitly links its
+model, data, evaluator, trainer, callbacks and logger. Inspect it without
+starting training:
+
+```bash
+python egomimic/trainHydra.py \
+  +experiment=arc_real/stationary_velocity --cfg job --resolve
+```
+
+The [experiment menu](egomimic/hydra_configs/experiment/README.md) lists
+`baseline`, `arc_vel` and `arc_dur` for each real task. A cluster launch adds
+`-m hydra/launcher=<approved-submitit-config>` after resource/data preflight;
+machine-specific paths and credentials must be configured. Advanced direct
+`model=... data=... evaluator=...` composition remains supported; the root
+config deliberately does not select those components implicitly.
+See [training.md](./training.md) for the graph and launch workflow.
 
 ### Converting your own data
 See [``embodiment_tutorial.ipynb``](./egomimic/scripts/tutorials/embodiment_tutorial.ipynb) as reference to write a conversion script for your own data.

docs/ARC_CHUNKING_MODES.md

@@ -0,0 +1,14 @@
+# Cartesian ARC stream selection
+
+The former race, joint-distance and multistream selectors have been removed.
+Cartesian ARC always uses independent stream supports and clocks, with
+first-moving-stream replanning. Choose only the clock representation and stream
+layout described in [ARC_CLOCK_CONTRACT.md](ARC_CLOCK_CONTRACT.md).
+
+The supported layouts are full separation, translation/rotation,
+translation-only and rotation-only. Projection is maintained on an optional
+separate review branch, not in this stack's default source.
+
+Historical checkpoints require their recorded Git source and configuration.
+The historical mode documentation remains available in Git history; its timing,
+normalization and evaluation contracts are not interchangeable with arrival-v2.

docs/ARC_CLOCK_CONTRACT.md

@@ -0,0 +1,186 @@
+# Cartesian ARC: explicit-current-pose arrival clocks
+
+This is the current Cartesian contract. It is a new representation relative to
+the former first-delay token: row zero now describes the current measured pose
+to the first future target. No row has a special start-delay meaning.
+Separate planar and Libero codecs are unchanged.
+
+Select a complete experiment from the
+[experiment menu](../egomimic/hydra_configs/experiment/README.md). The example
+below describes the codec API, not a universal experiment's D/R/M settings:
+
+```yaml
+_target_: egomimic.rldb.zarr.arc_length_tokenizer.TokenizeBimanualArcLengthCartesian
+initial_action_key: arc.current_action
+clock_mode: velocity                 # velocity | duration
+streams: full_separation              # full_separation | translation_rotation | translation | rotation
+resampled_vector_length: 100
+min_distance_unit: 0.42               # meters, per arm
+rotation_distance_unit: 1.7453292519943295  # radians, per arm (100 degrees)
+dt: 0.03333333333333333
+gripper_stream_mode: events           # events | distance; separate streams only
+gripper_distance_unit: null           # required positive for distance mode
+```
+
+## Current state and future source
+
+The transform requires two distinct inputs:
+
+- `arc.current_action`: the finite 14D measured current pose at time zero,
+  transformed into exactly the same coordinate frame, YPR convention and
+  gripper convention as the action geometry.
+- `actions_cartesian`: future rows beginning at source index `idx+1`.
+  Raw row i arrives at `(i+1) * dt`; the current observation row is not
+  duplicated into this window.
+
+The current pose is supplied separately to tokenization and every decoding
+entrypoint. It must never be guessed from token row zero or replaced by a zero
+pose except in an explicitly constructed synthetic example whose current pose
+really is zero. Timing uses the configured control-grid dt, not irregular
+measured timestamps.
+
+## Geometry, supports and clock order
+
+Geometry occupies the first 14 columns:
+`[left xyz, left YPR, left gripper, right xyz, right YPR, right gripper]`.
+
+| streams | Shape | Timing columns |
+| --- | --- | --- |
+| full_separation | M × 20 | T-left, T-right, R-left, R-right, G-left, G-right |
+| translation_rotation | M × 18 | T-left, T-right, R-left, R-right |
+| translation | M × 16 | T-left, T-right |
+| rotation | M × 16 | R-left, R-right |
+
+Translation-only samples XYZ, YPR and gripper on the translation supports.
+Rotation-only samples all three on the rotation supports. In
+translation_rotation, the gripper follows translation. Full separation gives
+each component its own supports and clock. A token row therefore need not
+represent simultaneous source times across separate streams.
+
+D and R bound cumulative travel **within the supplied future trajectory**.
+The measured-current-to-first-future-target segment is included in incoming
+rates and decoding, but is not retroactively included in those support budgets.
+Anchor-inclusive emitted travel can exceed the source budget by that incoming
+segment. Translation uses Euclidean distances; rotation uses SO(3) geodesic
+distances; gripper distance uses absolute command differences.
+
+Exactly zero-progress geometry uses supports distributed over the actual
+source indices. This retains the real available time span without inventing
+movement or extending the trajectory to a configured hold duration.
+
+## The same timing meaning in every row
+
+Let q[-1] be the measured current pose, t[-1]=0, and q[i] the emitted future
+waypoint with source arrival time t[i].
+
+### Velocity
+
+Every row, including row zero, stores the incoming segment speed:
+
+```text
+v[i] = distance(q[i-1], q[i]) / (t[i] - t[i-1])
+```
+
+Thus row zero is measured-current-to-first-target speed, not seconds and not
+a start delay. The numerator is computed between emitted supports, matching
+the segment reconstructed by the decoder. Units are m/s, rad/s or gripper
+units/s. Tiny zero intervals use numerical guards. Rates are not smoothed or
+physically speed-clamped; decode uses the rate magnitude.
+
+Normalization provenance identifies this timing layout as `incoming_rate_v2`.
+
+### Duration
+
+Every row stores its raw absolute arrival time from the current observation:
+
+```text
+clock[i] = t[i]
+```
+
+There is no logarithm, exponentiation, or special row-zero delay. For future
+targets at 0.1, 0.2 and 0.3 seconds, the token stores `[0.1, 0.2, 0.3]`.
+The decoder derives incoming intervals with `diff([0, corrected_times])`.
+
+Predicted times are projected to a non-decreasing absolute clock by isotonic
+regression. Translation has a 5 m/s maximum-speed bound that includes the
+current-pose-to-row-zero segment:
+
+```text
+fastest[i] = sum(incoming_distance[0:i+1]) / 5
+corrected_times = fastest + max(isotonic(predicted_times - fastest), 0)
+```
+
+Rotation and gripper clocks receive monotone projection without a translation
+speed bound. No arbitrary minimum chunk duration is added.
+Normalization provenance uses `absolute_arrival_seconds_v2`.
+
+## Pauses and grippers
+
+There is no motion-start-delay extractor, synthetic 400-frame hold clock,
+or dedicated internal-pause support allocation.
+
+Duration can preserve a stationary span when it is represented in the emitted
+support times. Fully static duration tokens retain the actual source span;
+they are not extended to 400 frames. Velocity is zero on a zero-distance
+interval and cannot encode how long that dwell lasted. This includes terminal
+gripper dwells and all-static chunks.
+
+For separate event grippers, supports remain initial future command, command
+changes, source endpoint and repeated endpoint padding. If there are more
+events than M supports, they are downsampled. Reconstruction is step/hold
+command behavior, not fitted aperture ramps. Distance grippers use their
+independent cumulative-travel budget.
+
+Single-stream velocity has the same limitation: if the selected component
+does not move but another component does, that selected component's zero rate
+cannot encode the other component's timing. Use a separated clock layout or
+duration when that timing must be represented.
+
+## Decode and execution boundary
+
+Both evaluation and rollout provide the measured current pose explicitly:
+
+```python
+codec.control_steps(token_prefix, initial_action=current_pose, max_steps=horizon)
+codec.detokenize(token_prefix, horizon, initial_action=current_pose)
+```
+
+The codec starts at the measured pose at t=0 and emits future controls at
+`dt, 2*dt, ..., horizon*dt`. XYZ uses linear interpolation, rotations use
+SO(3) interpolation, and event grippers use held commands.
+
+For first-stream execution:
+
+1. Keep an exact integer prefix of future waypoint rows.
+2. Decode all incoming intervals, including current-to-row-zero.
+3. Replan when the first moving stream finishes that prefix.
+4. Exclude constant streams from the race while another stream moves.
+
+A fully static velocity token contains no duration. `control_steps` therefore
+requires the caller's positive `max_steps` and returns that explicit bound.
+For fully static duration tokens it uses the earliest final absolute stream
+arrival. A moving PW stream with unusable rates cannot win the termination
+race; the explicit caller horizon can bound it. Nonfinite tokens or anchors
+fail rather than being repaired silently.
+
+`clock_token_seconds` requires the same explicit anchor when converting
+duration columns. Sixteen-column helpers also require explicit `streams`;
+shape alone cannot distinguish translation-only from rotation-only.
+
+## Migration and checkpoint safety
+
+The former mean/wide/stacked/log-clock/E1 APIs are not supported.
+Projection is outside the main contract. `hold_window_frames`,
+`projection_offset_m`, `velocity_mode`, `velocity_layout` and
+`velocity_norm` are not codec constructor options.
+
+First-delay checkpoints and normalization statistics are incompatible with
+these arrival clocks despite matching tensor shapes. Preserve their recorded
+source for historical reloads; do not infer the layout from token width or
+resume an old run after this change. New runs require fresh normalization
+statistics and the new timing-layout provenance.
+
+Checks and fixtures remain outside the application repository under the
+[verification policy](VERIFICATION_POLICY.md). Local numerical checks do not
+establish GPU throughput, pretrained-checkpoint compatibility or hardware
+safety. This source update does not start a cluster or robot process.

docs/ARC_OAT.md

@@ -450,18 +450,18 @@ examples use the measured LIBERO-10 selections in the
 [dated replay report](results/libero_arc_timed_replay_20260921.md):
 
 ```sh
-python -m egomimic.trainHydra +experiment=oat/libero_oattok \
+python -m egomimic.trainHydra +experiment=arc_sim/oat/libero_oattok \
   benchmark.suite=libero_10 benchmark.dataset=/datasets/arc_oat/libero_10.zarr \
   hydra.run.dir=/runs/tokenizer
-python -m egomimic.trainHydra +experiment=oat/libero_oatpolicy \
+python -m egomimic.trainHydra +experiment=arc_sim/oat/libero_oatpolicy \
   benchmark.suite=libero_10 benchmark.dataset=/datasets/arc_oat/libero_10.zarr \
   benchmark.tokenizer_checkpoint=/runs/tokenizer/checkpoints/last.ckpt \
   hydra.run.dir=/runs/oat
-python -m egomimic.trainHydra +experiment=oat/libero_arc_dur_policy \
+python -m egomimic.trainHydra +experiment=arc_sim/oat/libero_arc_dur_policy \
   benchmark.suite=libero_10 benchmark.dataset=/datasets/arc_oat/libero_10.zarr \
   benchmark.arc_waypoints=32 benchmark.arc_max_translation=0.8 \
   benchmark.arc_max_rotation_degrees=192 hydra.run.dir=/runs/arc_dur
-python -m egomimic.trainHydra +experiment=oat/libero_arc_stk_policy \
+python -m egomimic.trainHydra +experiment=arc_sim/oat/libero_arc_stk_policy \
   benchmark.suite=libero_10 benchmark.dataset=/datasets/arc_oat/libero_10.zarr \
   benchmark.arc_waypoints=32 benchmark.arc_max_translation=1.6 \
   benchmark.arc_max_rotation_degrees=128 hydra.run.dir=/runs/arc_stk