Skip to content

Bring Aniketh's ARC work onto main with the YAM task recipes and the station rollout runtime - #263

Open
aidang3019 wants to merge 43 commits into
mainfrom
aidan/yam-tasks-consolidated-20261018
Open

aidang3019 wants to merge 43 commits into
mainfrom
aidan/yam-tasks-consolidated-20261018

Conversation

@aidang3019

@aidang3019 aidang3019 commented Oct 19, 2026 •

Copy link
Copy Markdown

On main (a525d738). Aniketh's PR #259 work reconciled onto main as one co-authored commit (5666d7a8), then Aidan's stationery / earbuds / towels recipes and the station rollout runtime, so the three YAM task runs train and deploy from one checkout. Head 97b48673: 272 files, +40,511 / −5,524.

Pruned on 2026-10-18 at Aidan's request: the recipe tree keeps only the _base/ recipes and one leaf set per task at M 200 waypoints / D 0.33 m (plus the M/D-free time leaves), and the checkpoint-era tokenizer snapshots (#230's robot/arc_codecs/, the station's pinned_arc_bbb813a0/) are not carried.

Review in four groups

1. Aniketh's #259 onto main (5666d7a8, one commit, co-authored). His 60 commits were on the 2026-09-24 main, 26 merged PRs behind; this commit is his content adapted to main's data-context architecture: data/abc_arc recipes move to ZarrDataModule + prepare_context, the ABC ARC export guard becomes model_contract/abc_visual.yaml + inference / data_requirements declarations on his models (stage_ids.sampler 4 for HPT, 3 for DiT), open_loop_sim gains main's evaluation hooks and the history-stack rule, MultiDataset.to_state keeps contract-free digests, ABC-DiT skips external weights under restoring_parameters(). Multistream / six-clock / log-clock evaluator and tokenizer, ABC-DiT, ResNet-MLP stems and his experiment/abc_arc recipes are otherwise his. @aniketh: #259 can take this commit as its rebase onto main.

2. Recipes. _base/stattempo (FT recipe on the frozen slow-pace 366 pool), _base/stattempo_openloop, _base/arc_projection, _base/m200_d033, _base/towels_hpt300 (HPT-300M, tempo evaluator), _base/earbuds_updated_hpt300, _base/m200_d033_hpt300; the shared loader data/abc_arc/stationery_yam_loader and the three frozen pools (slow-pace 366, earbuds_updated, towels since0924plus74 / new290plus74) with their manifests. Leaves:

  • stationery: bc_rl2_stattempo_slowpace366_{arcdur,arcvel,arcdur_abstime,arcvel_nosmooth,arc_projection_dur,arc_projection_vel}_m200_d033
  • earbuds: earbuds_updated_time_hpt300, earbuds_updated_{arcdur,arcvel,arcdur_abstime,arcvel_nosmooth}_m200_d033_hpt300
  • towels: towels_{since0924plus74,new290plus74}_time_hpt300, towels_{since0924plus74,new290plus74}_{arcdur,arcvel}_m200_d033_hpt300

Every task launches through scripts/e1/stationery_ft.sbatch with EXPERIMENT=yam_arc_grid/<leaf>; a relaunch changes only logger.wandb.id (docs/STATTEMPO.md). The six stationery leaves resolve byte-for-byte to the configs authored on 10-17 (tests/fixtures/stattempo_contracts.json pins them at 8e80f090). The hybrid / tri / event / reference, r72 / r360, tempo-bucket, Elmo+Aidan, Aria-cotrain and M 100 recipes are gone from the tree; their codecs, decoders and evaluator modes stay, and scripts/e1/build_slowpace366.py regenerates them from the manifest if a run needs them.

3. Rollout runtime (da1429cd … fa072050):

  • Elmo's Consolidate YAM rollout recording, active timer, and ARC controls #230, cherry-picked with his authorship: declared decoder_attribute controls and transactional apply_control_bindings, execution_prefix, replay / hold tempo, fastest-stream chunk termination with its waypoint cap, HDF5 episode recording, the rollout timer, station yaml without inline inference_graph. Its vendored codec copies are removed again in group 4.
  • The camera alignment monitor (plus the station's direct-D405 --source realsense), the HDF5 tempo checker, the stalled-episode / max_steps fixes.
  • The 10-12 IK branch: opt-in pose relaxation, bounded differential IK recovery, TRAC-IK endpoint selector (yam-tracik extra), --check-model CPU preflight, towels / earbuds HPT300 rollout profiles.

4. Decoding the stack's own tokens + station extras:

  • BimanualArcDecoder decodes every layout these recipes train with the live egomimic.rldb.zarr tokenizers (absolute timing, hybrid R, event / reference / projection, speed floors) with the same tempo and termination controls; the live E1 codec gains decode-only helpers (detokenize(times=), as_duration_token, stream_clocks, stream_ends), tokenize untouched. FirstStreamArcDecoder executes the live multistream interval tokens to their first moving stream. The E1 contracts declare first_stream: 1, execute_percent: 50 and the four controls (model.arc_decoder_controls); the time contract wraps its chunk in a TimeChunkRetimer for the naive speed-up. No checkpoint-era tokenizer copies: a bundle trained by another tokenizer revision needs that revision's runtime.
  • Station-only extra: view-only reset guides on both dashboards (reset_overlay_templates.yaml, egomimic.scripts.build_reset_guides).
  • docs/YAM_RUNTIME.md documents the decoder, the declared controls, the IK options, the monitors and the reset guides.

Decisions on the open questions from the #229 handoff

  1. Legacy unbound checkpoints stay unsupported (main's loader); the shared profile defaults to the bound towels HPT300 180k bundle, YAM_HPTFLOW_* env variables select another.
  2. No vendored codecs: tokens decode with the live tokenizer; older bundles keep their own runtime.
  3. No encode-side timing change is ported.
  4. Decoder controls are declared decoder_attribute targets; fastest-stream is on by default in the E1 contracts (what the station ran), the dashboard turns it off.
  5. Prompt follows the default bundle (fold towels); the per-task profiles pin their own.

Not carried on purpose: the overlapped-inference loop (tried 10-08, reverted on the arms), the station's hash-bound pinned_arc_bbb813a0 routing and the live-tree BimanualArcClockDecoder (both depend on byte-exact tokenizer snapshots), and the station's uncommitted hpt_stems / teleop edits (main already has them).

Verification (WSL CPU venv)

  • ruff 0.8.6 check + format clean (main's gate).
  • scripts/audit_hydra_configs.py: 415 / 415 YAMLs compose and resolve; all 17 task leaves and 7 bases resolve to their variant, M, D, W&B id and declared decoder.
  • scripts/e1/validate_stattempo_configs.py: PASS, the 6 stationery recipes match 8e80f090.
  • Recipe / decoder / rollout / contract suites: 606 passed, 15 skipped; the 6 remaining failures (MuJoCo / mink missing, no torchvision write_video, W&B-resume and ABC layout tests) are identical on the pre-rollout tag.
  • Not done: no training smoke, no checkpoint reload, nothing on the arms. First station use: fetch into a new worktree, --check-config then --check-model on the towels profile, then a rollout as the validation of the decoder path.

🤖 Generated with Claude Code

aidang3019 and others added 30 commits October 18, 2026 21:19
Aniketh's codex/absolute-log-clock6 (60 commits on the 2026-09-24 main
161e3a0, tip f021aa1) reconciled with current main a525d73 as one commit,
so the work can be reviewed and stacked without replaying the 26 PRs his base
lacked. His history stays on PR #259; this is his content adapted to main's
data-context architecture.

What the branch carried:
- Multistream ARC execution and scoring: each arm's own rotation clock, raced
  with translation; six-stream action clocks (translation, rotation, gripper
  per arm); four-clock duration tokens; log_clock and absolute log-duration
  clock layouts; the wide velocity layout as the bimanual default; per-arm
  multistream DTW with segment caps; M28 DTW and rotation-cap repairs; the
  hold-chunk timing fix and position-only interval timing.
- Open-loop evaluator: multistream executes until its first stream ends,
  chunk-held and executed-chunk videos by default, episode-boundary and full
  validation coverage fixes, per-embodiment metrics and W&B sections,
  configurable ARC read window, optional dump of every validation record.
- Models and recipes: ABC-DiT restored on the M28 runtime with CLIP task
  conditioning, HPT ResNet-MLP image stems with training color jitter, towel
  multistream and log_clock recipes (HPT180 / HPT300, ABC-DiT), organize robot
  + human cotraining recipes, event-mode gripper fixtures (PR 202), the
  regenerated frozen tokenizer parity corpus, joint human+robot ARC t-SNE
  explorer, sanitized default W&B run ids.

Reconciliation with main (the former catch-up commit 3eb7448):
- data/abc_arc recipes move from MultiDataModuleWrapper to the ZarrDataModule
  data context (_recursive_: false, source_fps: 30.0); trainHydra keeps main's
  prepare_context flow and threads run_provenance.action_contract into
  normalization fitting so ARC caches stay bound to their codec.
- The code-level ABC ARC export guard becomes model-owned declarations:
  model_contract/abc_visual.yaml (100-frame cartesian contract, ARC
  "unsupported" entry, selected per abc.action_mode); hpt_yam_visual,
  hpt_bimanual_visual (population-scoped data requirements) and
  abc_dit_yam_visual declare inference / data_requirements; stage_ids fixed to
  the actual sampler positions.
- Flow denoiser accepts main's condition_ndim and ABC-DiT's token conditions;
  DualHPTTrunkStage kept beside main's HPTTrunkStage.execute.
- open_loop_sim keeps the multistream / clock evaluator and adopts main's
  EvaluationDataRequirements hooks, the history-stack observation rule,
  _record_video_frames, and flushes held chunks before the collective video
  gather; baseline mode accepts a chunking mode without a rotation distance.
- MultiDataset.to_state serializes action_contracts only when present; ABC-DiT
  skips its DINOv3 / CLIP initializers under restoring_parameters().
- Retired ABC ablation configs stay deleted; audit contexts, env placeholders
  and three main tests use the retained recipes;
  evaluator/eval_arc_bimanual_cartesian_D40_M100.yaml removed (no consumers).

Co-authored-by: AnikethCheluva <ani.cheluva@gmail.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Re-stacked from aidan/stattempo-consolidated-main-20261011 (a572d274) onto
the codex/absolute-log-clock6 catch-up. The shared evaluator and tokenizer
files (open_loop_sim, arc_length_tokenizer, e1_arc_tokenizer,
distance_budget_dtw, bimanual_arc) were merged three-way against the fork
point 67863ca of the two lines, so Aniketh's later multistream/clock work
stays intact and only the E1 additions (speed floors, tick-prefix scoring,
evt/ref layouts, fixed ranges, clipped flow sampling) are layered on. The
retired legacy fixed-row ARC truncation path and its "legacy" cap mode follow
the branch and are not reintroduced.

The 50 resolved stationery recipes still match the pinned 51cd2bf8 contracts:
the stationery evaluator selections pin arc_video_trajectory_cap_mode:
distance (the recorded value; the shared evaluator now defaults to the
chunking mode), and the contract filter documents the evaluator defaults the
catch-up added (null diagnostics and the executed_chunk overlay style of
videos these recipes disable).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit fc35bc7b89b03455730f36fea796763d174245a3)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…erships

Each pool adds the 74 Aidan fold_towels takes uploaded 2026-10-15 to train and keeps the earlier validation set. since0924 drops the 30 rows soft-deleted on 2026-10-13 (26 train, 4 val): new290plus74 340/24, since0924plus74 443/17.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…op decode

OpenLoopSimEval passes duration_timing to its E1 tokenizer so absolute waypoint
times are scored as absolute. The stationery leaf matches the 10-05 Lambda
baseline bc_rl2_stattempo_slowpace_arcdur_lambda except for duration timing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Matches the 10-05 Lambda baseline bc_rl2_stattempo_slowpace_arcvel_lambda
except for speed_smooth_frames 7 -> 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Aidan 2026-10-17: retrain arcdur, arcdur absolute time, arcvel, arcvel without smoothing and
both ARC Projection codecs with 200 waypoints and a 0.33 m arc budget. The shared override
routes every literal M / D read (data transforms, arc_tokenizer, token-row horizons and the
open-loop evaluator) through e1, and turns on tick-prefix scoring for all six. Adds plain
arcdur / arcvel leaves (the 10-05 baselines on the stack), drops the earbuds leaves that came
with the timing cherry-picks, and keeps their stationery tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CSaa5mnstFBVJi8329KLDc
Elmo 2026-10-16: absolute time fixed arm syncing on earbuds; placement lacks precision.
One change against earbuds_updated_arcdur_abstime_hpt300, applied everywhere the value is read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Elmo 2026-10-16: absolute time fixed arm syncing on earbuds; placement lacks precision.
One change against earbuds_updated_arcdur_abstime_hpt300, applied everywhere the value is read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Elmo 2026-10-16: the earbuds rollout lacks placement precision (long chunk, too few waypoints).
One change against earbuds_updated_arcvel_nosmooth_hpt300, applied everywhere the value is read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Elmo 2026-10-16: the earbuds rollout lacks placement precision (long chunk, too few waypoints).
One change against earbuds_updated_arcvel_nosmooth_hpt300, applied everywhere the value is read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sk recipes

The stationery M200/D033 commit on Phoenix deleted
earbuds_updated_arcdur_abstime_hpt300 and earbuds_updated_arcvel_nosmooth_hpt300
when it renamed the earbuds tests to stationery ones; the earbuds M200 / D03
leaves inherit them, so both come back as recorded on the earbuds branches.
The frozen towels/earbuds pool data configs get an audit context that supplies
e1.*, and the recipe commits that only ever lived in Phoenix worktrees pass
main's ruff gate (import order, formatting). Every shipped YAML composes again
(540/540) and the earbuds/stationery/projection recipe tests pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
docs/STATTEMPO.md names the branch, the shared one-GPU launcher and, per
task (stationery slow-pace 366, wireless_earbuds_updated, fold_towels), the
frozen pool, the recipe leaves and the one config change a relaunch needs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
egomimic/robot/camera_align: a loopback dashboard (python -m
egomimic.robot.camera_align.monitor, run.sh / stop.sh) that reads the rollout
dashboard's own front-camera stream as a receive-only websocket client and
reports, four times a second:

- camera roll / tilt / pan against a saved empty-table reference frame (SIFT
  similarity, the D405's own intrinsics from yam_rl2_agentview_extrinsics.yaml),
  with an advice line in screen terms;
- desk motion separated from camera motion: the table's left and right edges
  located to sub-pixel precision in two rows above the robot bases, and SIFT
  features on the table top and in the background fitted separately, converted
  to millimetres at the calibrated table range;
- ArUco tag poses (4x4_50, 200 mm by default) and, after "Save tag reference",
  the same angles from the tag corners, with a moved-tag warning from the
  inter-tag baseline.

State (reference frames, tag reference, snapshots) lives in --data-dir, not in
the repository. It never sends a command to the rollout dashboard and opens no
camera or robot; connecting supersedes the rollout dashboard's single browser
tab, and reopening that tab supersedes the monitor (close code 4001) until
"Reconnect stream" is pressed.

Tests: tests/test_camera_align_monitor.py (10): screen-motion sign conventions,
advice text, sub-pixel edge location, desk geometry and mask, desk verdicts,
tag detection and pose on a rendered marker, tag comparison and the moved-tag
warning, reference loading, calibration loading.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit ce51e57)
(cherry picked from commit abb5f5ea5f3b8d12c0a012eb4c0f6213c5d787ed)
(cherry picked from commit cf78a4f)
(cherry picked from commit e64e3d82a8267b1b5b6e4a5faa302beeea9e9210)
Review findings on the alignment monitor, both reproduced on scenes rendered
through the station K and camera_T_base:

- advice_for never read the fitted scale, so a camera pushed 20-40 mm along
  its optical axis (scale 1.008-1.016, table edges 3.5-8 px off) read ALIGNED
  while the page's own table coloured the scale red. It now adds a DISTANCE
  line at |scale - 1| >= 0.005 (nothing moved: 0.9997-1.0001).
- desk_compare built its verdict from horizontal terms only, so a desk slid
  20-50 mm toward or away from the camera (table 6-17 px up or down against
  the room) read "desk steady". It now compares the table's and the
  background's vertical shift, as it already did for the horizontal one. A
  steady desk under camera pan, tilt, roll and noise read at most 0.54 px
  against the 2 px threshold; a 10 mm slide reads 2.3 px.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit ba41e0c7592a4da18cdf193c6c343d384bddf505)
…reference

Review findings on the alignment monitor, reproduced against the real
RolloutDashboard on loopback:

- The last verdict was never invalidated. With the stream dropped or
  superseded, the front camera gone from the stream, every analysis failing,
  or the dashboard re-sending an old frame while the rollout loop is busy
  (model load, homing), the page kept showing a green ALIGNED and "desk
  steady", and the save buttons stored that old frame as the new reference.
  /metrics.json now replaces both with "STALE: no live measurement" when the
  stream is not connected, the last analysis is older than 2 s, or the
  dashboard reports the frame itself as older than 2 s; the two reference
  saves refuse in the same state. A successful analysis clears the error line.
- Saving a frame reference ignored cv2.imwrite's result and retired the old
  reference.json before the new image was known to exist, so a save on a full
  disk left the pointer naming a missing PNG and the next start had no
  reference. The image is now checked and loaded first, and both pointers are
  written to a temp file and swapped in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit a6ed2e04ae0a5cd2d3755e246d48a189177870d2)
The monitor latched itself out only on close code 4001. After anything else
(rollout exited, dashboard not up yet, Reconnect pressed with no rollout up)
it retried every 2 s, so a monitor left running connected to the next
rollout's dashboard by itself and closed the operator's freshly opened tab
with 4001, which that page never reopens: Start, Pause and Stop dead until a
reload (reproduced with the real Monitor, RolloutDashboard and run_rollout on
loopback, 4 of 4 when the tab opened within the 2 s poll).

It now latches after every attempt, so each connection is an operator action:
one at start, one per "Reconnect stream" press. Trade-off: the monitor no
longer follows a rollout restart on its own. The page says which of the two
states it is in (superseded by the tab, or just disconnected).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 48a442a67ffac46130fe8974e9ec47e15ada498f)
…the alignment monitor

Port of what aidan/yam-rollout-main-port-20261009 (86845377, 767e1547) carried
beyond PR #230's episode recorder: max_steps is validated before any device
opens, a per-tick recording failure ends only the episode, the tempo checker
skips an unusable episode instead of aborting the scan, and the never-filled
episode-folder placeholder is dropped.

The camera alignment monitor gains the station's uncommitted --source option
(rl2-yam ~/dev/EgoVerse-pr-merge-20261009): "realsense" opens the front D405
directly so the monitor runs with no rollout dashboard; "stream" (the CLI
default) keeps reading the dashboard websocket. run.sh defaults to realsense.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 72e8e2ea1deefdc3cf9ddec9a6dfc10046eab862)
This was referenced Oct 23, 2026
aidang3019 added a commit that referenced this pull request Nov 4, 2026
Stacked on #317. Brings back the rollout knobs the Tuesday (#263) dashboard had. #317's ARC profile shows only sampling steps, and it opens at 10.

- **Station-owned controls** (`policy.station_controls` in the three YAM rollout profiles): Actions before replanning, Fastest-stream termination, Execute first % of waypoints, Replay speed, Hold speed. A control is added only when the model declares none of that name and, for decoder controls, only when the loaded decoder has the attribute. Time bundles keep their own controls (the station adds nothing to them). Checkpoint-bound declarations are untouched, so bundles already exported get the knobs as is.
- **`policy.inference_defaults`**: sampling steps (Euler steps) open at 50, the trained and validated value, for any model that exposes the control.
- **ARC replay tempo**: `FirstStreamArcDecoder` reuses the `ReplayTempo` warp (moving vs hold speed) on the translation clock. Decode-only codec helpers: `detokenize(times=)`, `stream_clocks`, `control_duration`.

Checks (CPU, WSL):
- At 100 % / 100 % the decode is byte-identical to 776656e on 60 real val chunks, duration/translation and velocity/full_separation, M=200.
- At 2×, row k equals 1× row 2k+1 exactly and the fastest-stream step count is ceil(1×/2).
- Hold 100 % with replay 200 % lands between the two.
- `audit_hydra_configs`: 325/325. `audit_components`: 276/325, the same failure set as 776656e (OAT / LIBERO / Qwen / egobridge env-only).

Not yet run on the arms.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants