Repository navigation
Bring Aniketh's ARC work onto main with the YAM task recipes and the station rollout runtime - #263
Open
aidang3019 wants to merge 43 commits into
Open
aidang3019 wants to merge 43 commits into
aidang3019 wants to merge 43 commits into
Conversation
Aniketh's codex/absolute-log-clock6 (60 commits on the 2026-09-24 main 161e3a0, tip f021aa1) reconciled with current main a525d73 as one commit, so the work can be reviewed and stacked without replaying the 26 PRs his base lacked. His history stays on PR #259; this is his content adapted to main's data-context architecture. What the branch carried: - Multistream ARC execution and scoring: each arm's own rotation clock, raced with translation; six-stream action clocks (translation, rotation, gripper per arm); four-clock duration tokens; log_clock and absolute log-duration clock layouts; the wide velocity layout as the bimanual default; per-arm multistream DTW with segment caps; M28 DTW and rotation-cap repairs; the hold-chunk timing fix and position-only interval timing. - Open-loop evaluator: multistream executes until its first stream ends, chunk-held and executed-chunk videos by default, episode-boundary and full validation coverage fixes, per-embodiment metrics and W&B sections, configurable ARC read window, optional dump of every validation record. - Models and recipes: ABC-DiT restored on the M28 runtime with CLIP task conditioning, HPT ResNet-MLP image stems with training color jitter, towel multistream and log_clock recipes (HPT180 / HPT300, ABC-DiT), organize robot + human cotraining recipes, event-mode gripper fixtures (PR 202), the regenerated frozen tokenizer parity corpus, joint human+robot ARC t-SNE explorer, sanitized default W&B run ids. Reconciliation with main (the former catch-up commit 3eb7448): - data/abc_arc recipes move from MultiDataModuleWrapper to the ZarrDataModule data context (_recursive_: false, source_fps: 30.0); trainHydra keeps main's prepare_context flow and threads run_provenance.action_contract into normalization fitting so ARC caches stay bound to their codec. - The code-level ABC ARC export guard becomes model-owned declarations: model_contract/abc_visual.yaml (100-frame cartesian contract, ARC "unsupported" entry, selected per abc.action_mode); hpt_yam_visual, hpt_bimanual_visual (population-scoped data requirements) and abc_dit_yam_visual declare inference / data_requirements; stage_ids fixed to the actual sampler positions. - Flow denoiser accepts main's condition_ndim and ABC-DiT's token conditions; DualHPTTrunkStage kept beside main's HPTTrunkStage.execute. - open_loop_sim keeps the multistream / clock evaluator and adopts main's EvaluationDataRequirements hooks, the history-stack observation rule, _record_video_frames, and flushes held chunks before the collective video gather; baseline mode accepts a chunking mode without a rotation distance. - MultiDataset.to_state serializes action_contracts only when present; ABC-DiT skips its DINOv3 / CLIP initializers under restoring_parameters(). - Retired ABC ablation configs stay deleted; audit contexts, env placeholders and three main tests use the retained recipes; evaluator/eval_arc_bimanual_cartesian_D40_M100.yaml removed (no consumers). Co-authored-by: AnikethCheluva <ani.cheluva@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Re-stacked from aidan/stattempo-consolidated-main-20261011 (a572d274) onto the codex/absolute-log-clock6 catch-up. The shared evaluator and tokenizer files (open_loop_sim, arc_length_tokenizer, e1_arc_tokenizer, distance_budget_dtw, bimanual_arc) were merged three-way against the fork point 67863ca of the two lines, so Aniketh's later multistream/clock work stays intact and only the E1 additions (speed floors, tick-prefix scoring, evt/ref layouts, fixed ranges, clipped flow sampling) are layered on. The retired legacy fixed-row ARC truncation path and its "legacy" cap mode follow the branch and are not reintroduced. The 50 resolved stationery recipes still match the pinned 51cd2bf8 contracts: the stationery evaluator selections pin arc_video_trajectory_cap_mode: distance (the recorded value; the shared evaluator now defaults to the chunking mode), and the contract filter documents the evaluator defaults the catch-up added (null diagnostics and the executed_chunk overlay style of videos these recipes disable). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit fc35bc7b89b03455730f36fea796763d174245a3)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…erships Each pool adds the 74 Aidan fold_towels takes uploaded 2026-10-15 to train and keeps the earlier validation set. since0924 drops the 30 rows soft-deleted on 2026-10-13 (26 train, 4 val): new290plus74 340/24, since0924plus74 443/17. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…op decode OpenLoopSimEval passes duration_timing to its E1 tokenizer so absolute waypoint times are scored as absolute. The stationery leaf matches the 10-05 Lambda baseline bc_rl2_stattempo_slowpace_arcdur_lambda except for duration timing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Matches the 10-05 Lambda baseline bc_rl2_stattempo_slowpace_arcvel_lambda except for speed_smooth_frames 7 -> 1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Aidan 2026-10-17: retrain arcdur, arcdur absolute time, arcvel, arcvel without smoothing and both ARC Projection codecs with 200 waypoints and a 0.33 m arc budget. The shared override routes every literal M / D read (data transforms, arc_tokenizer, token-row horizons and the open-loop evaluator) through e1, and turns on tick-prefix scoring for all six. Adds plain arcdur / arcvel leaves (the 10-05 baselines on the stack), drops the earbuds leaves that came with the timing cherry-picks, and keeps their stationery tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CSaa5mnstFBVJi8329KLDc
Elmo 2026-10-16: absolute time fixed arm syncing on earbuds; placement lacks precision. One change against earbuds_updated_arcdur_abstime_hpt300, applied everywhere the value is read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Elmo 2026-10-16: absolute time fixed arm syncing on earbuds; placement lacks precision. One change against earbuds_updated_arcdur_abstime_hpt300, applied everywhere the value is read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Elmo 2026-10-16: the earbuds rollout lacks placement precision (long chunk, too few waypoints). One change against earbuds_updated_arcvel_nosmooth_hpt300, applied everywhere the value is read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Elmo 2026-10-16: the earbuds rollout lacks placement precision (long chunk, too few waypoints). One change against earbuds_updated_arcvel_nosmooth_hpt300, applied everywhere the value is read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sk recipes The stationery M200/D033 commit on Phoenix deleted earbuds_updated_arcdur_abstime_hpt300 and earbuds_updated_arcvel_nosmooth_hpt300 when it renamed the earbuds tests to stationery ones; the earbuds M200 / D03 leaves inherit them, so both come back as recorded on the earbuds branches. The frozen towels/earbuds pool data configs get an audit context that supplies e1.*, and the recipe commits that only ever lived in Phoenix worktrees pass main's ruff gate (import order, formatting). Every shipped YAML composes again (540/540) and the earbuds/stationery/projection recipe tests pass. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
docs/STATTEMPO.md names the branch, the shared one-GPU launcher and, per task (stationery slow-pace 366, wireless_earbuds_updated, fold_towels), the frozen pool, the recipe leaves and the one config change a relaunch needs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 6111f23)
…exts (cherry picked from commit c96d355)
egomimic/robot/camera_align: a loopback dashboard (python -m egomimic.robot.camera_align.monitor, run.sh / stop.sh) that reads the rollout dashboard's own front-camera stream as a receive-only websocket client and reports, four times a second: - camera roll / tilt / pan against a saved empty-table reference frame (SIFT similarity, the D405's own intrinsics from yam_rl2_agentview_extrinsics.yaml), with an advice line in screen terms; - desk motion separated from camera motion: the table's left and right edges located to sub-pixel precision in two rows above the robot bases, and SIFT features on the table top and in the background fitted separately, converted to millimetres at the calibrated table range; - ArUco tag poses (4x4_50, 200 mm by default) and, after "Save tag reference", the same angles from the tag corners, with a moved-tag warning from the inter-tag baseline. State (reference frames, tag reference, snapshots) lives in --data-dir, not in the repository. It never sends a command to the rollout dashboard and opens no camera or robot; connecting supersedes the rollout dashboard's single browser tab, and reopening that tab supersedes the monitor (close code 4001) until "Reconnect stream" is pressed. Tests: tests/test_camera_align_monitor.py (10): screen-motion sign conventions, advice text, sub-pixel edge location, desk geometry and mask, desk verdicts, tag detection and pose on a rendered marker, tag comparison and the moved-tag warning, reference loading, calibration loading. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit ce51e57) (cherry picked from commit abb5f5ea5f3b8d12c0a012eb4c0f6213c5d787ed)
(cherry picked from commit cf78a4f) (cherry picked from commit e64e3d82a8267b1b5b6e4a5faa302beeea9e9210)
Review findings on the alignment monitor, both reproduced on scenes rendered through the station K and camera_T_base: - advice_for never read the fitted scale, so a camera pushed 20-40 mm along its optical axis (scale 1.008-1.016, table edges 3.5-8 px off) read ALIGNED while the page's own table coloured the scale red. It now adds a DISTANCE line at |scale - 1| >= 0.005 (nothing moved: 0.9997-1.0001). - desk_compare built its verdict from horizontal terms only, so a desk slid 20-50 mm toward or away from the camera (table 6-17 px up or down against the room) read "desk steady". It now compares the table's and the background's vertical shift, as it already did for the horizontal one. A steady desk under camera pan, tilt, roll and noise read at most 0.54 px against the 2 px threshold; a 10 mm slide reads 2.3 px. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit ba41e0c7592a4da18cdf193c6c343d384bddf505)
…reference Review findings on the alignment monitor, reproduced against the real RolloutDashboard on loopback: - The last verdict was never invalidated. With the stream dropped or superseded, the front camera gone from the stream, every analysis failing, or the dashboard re-sending an old frame while the rollout loop is busy (model load, homing), the page kept showing a green ALIGNED and "desk steady", and the save buttons stored that old frame as the new reference. /metrics.json now replaces both with "STALE: no live measurement" when the stream is not connected, the last analysis is older than 2 s, or the dashboard reports the frame itself as older than 2 s; the two reference saves refuse in the same state. A successful analysis clears the error line. - Saving a frame reference ignored cv2.imwrite's result and retired the old reference.json before the new image was known to exist, so a save on a full disk left the pointer naming a missing PNG and the next start had no reference. The image is now checked and loaded first, and both pointers are written to a temp file and swapped in. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit a6ed2e04ae0a5cd2d3755e246d48a189177870d2)
The monitor latched itself out only on close code 4001. After anything else (rollout exited, dashboard not up yet, Reconnect pressed with no rollout up) it retried every 2 s, so a monitor left running connected to the next rollout's dashboard by itself and closed the operator's freshly opened tab with 4001, which that page never reopens: Start, Pause and Stop dead until a reload (reproduced with the real Monitor, RolloutDashboard and run_rollout on loopback, 4 of 4 when the tab opened within the 2 s poll). It now latches after every attempt, so each connection is an operator action: one at start, one per "Reconnect stream" press. Trade-off: the monitor no longer follows a rollout restart on its own. The page says which of the two states it is in (superseded by the tab, or just disconnected). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 48a442a67ffac46130fe8974e9ec47e15ada498f)
…the alignment monitor Port of what aidan/yam-rollout-main-port-20261009 (86845377, 767e1547) carried beyond PR #230's episode recorder: max_steps is validated before any device opens, a per-tick recording failure ends only the episode, the tempo checker skips an unusable episode instead of aborting the scan, and the never-filled episode-folder placeholder is dropped. The camera alignment monitor gains the station's uncommitted --source option (rl2-yam ~/dev/EgoVerse-pr-merge-20261009): "realsense" opens the front D405 directly so the monitor runs with no rollout dashboard; "stream" (the CLI default) keeps reading the dashboard websocket. run.sh defaults to realsense. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 72e8e2ea1deefdc3cf9ddec9a6dfc10046eab862)
This was referenced Oct 23, 2026
Merged
aidang3019
added a commit
that referenced
this pull request
Nov 4, 2026
Stacked on #317. Brings back the rollout knobs the Tuesday (#263) dashboard had. #317's ARC profile shows only sampling steps, and it opens at 10. - **Station-owned controls** (`policy.station_controls` in the three YAM rollout profiles): Actions before replanning, Fastest-stream termination, Execute first % of waypoints, Replay speed, Hold speed. A control is added only when the model declares none of that name and, for decoder controls, only when the loaded decoder has the attribute. Time bundles keep their own controls (the station adds nothing to them). Checkpoint-bound declarations are untouched, so bundles already exported get the knobs as is. - **`policy.inference_defaults`**: sampling steps (Euler steps) open at 50, the trained and validated value, for any model that exposes the control. - **ARC replay tempo**: `FirstStreamArcDecoder` reuses the `ReplayTempo` warp (moving vs hold speed) on the translation clock. Decode-only codec helpers: `detokenize(times=)`, `stream_clocks`, `control_duration`. Checks (CPU, WSL): - At 100 % / 100 % the decode is byte-identical to 776656e on 60 real val chunks, duration/translation and velocity/full_separation, M=200. - At 2×, row k equals 1× row 2k+1 exactly and the fastest-stream step count is ceil(1×/2). - Hold 100 % with replay 200 % lands between the two. - `audit_hydra_configs`: 325/325. `audit_components`: 276/325, the same failure set as 776656e (OAT / LIBERO / Qwen / egobridge env-only). Not yet run on the arms.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On
main(a525d738). Aniketh's PR #259 work reconciled onto main as one co-authored commit (5666d7a8), then Aidan's stationery / earbuds / towels recipes and the station rollout runtime, so the three YAM task runs train and deploy from one checkout. Head97b48673: 272 files, +40,511 / −5,524.Pruned on 2026-10-18 at Aidan's request: the recipe tree keeps only the
_base/recipes and one leaf set per task at M 200 waypoints / D 0.33 m (plus the M/D-freetimeleaves), and the checkpoint-era tokenizer snapshots (#230'srobot/arc_codecs/, the station'spinned_arc_bbb813a0/) are not carried.Review in four groups
1. Aniketh's #259 onto main (
5666d7a8, one commit, co-authored). His 60 commits were on the 2026-09-24 main, 26 merged PRs behind; this commit is his content adapted to main's data-context architecture:data/abc_arcrecipes move toZarrDataModule+prepare_context, the ABC ARC export guard becomesmodel_contract/abc_visual.yaml+inference/data_requirementsdeclarations on his models (stage_ids.sampler4 for HPT, 3 for DiT),open_loop_simgains main's evaluation hooks and the history-stack rule,MultiDataset.to_statekeeps contract-free digests, ABC-DiT skips external weights underrestoring_parameters(). Multistream / six-clock / log-clock evaluator and tokenizer, ABC-DiT, ResNet-MLP stems and hisexperiment/abc_arcrecipes are otherwise his. @aniketh: #259 can take this commit as its rebase onto main.2. Recipes.
_base/stattempo(FT recipe on the frozen slow-pace 366 pool),_base/stattempo_openloop,_base/arc_projection,_base/m200_d033,_base/towels_hpt300(HPT-300M, tempo evaluator),_base/earbuds_updated_hpt300,_base/m200_d033_hpt300; the shared loaderdata/abc_arc/stationery_yam_loaderand the three frozen pools (slow-pace 366,earbuds_updated, towelssince0924plus74/new290plus74) with their manifests. Leaves:bc_rl2_stattempo_slowpace366_{arcdur,arcvel,arcdur_abstime,arcvel_nosmooth,arc_projection_dur,arc_projection_vel}_m200_d033earbuds_updated_time_hpt300,earbuds_updated_{arcdur,arcvel,arcdur_abstime,arcvel_nosmooth}_m200_d033_hpt300towels_{since0924plus74,new290plus74}_time_hpt300,towels_{since0924plus74,new290plus74}_{arcdur,arcvel}_m200_d033_hpt300Every task launches through
scripts/e1/stationery_ft.sbatchwithEXPERIMENT=yam_arc_grid/<leaf>; a relaunch changes onlylogger.wandb.id(docs/STATTEMPO.md). The six stationery leaves resolve byte-for-byte to the configs authored on 10-17 (tests/fixtures/stattempo_contracts.jsonpins them at8e80f090). The hybrid / tri / event / reference, r72 / r360, tempo-bucket, Elmo+Aidan, Aria-cotrain and M 100 recipes are gone from the tree; their codecs, decoders and evaluator modes stay, andscripts/e1/build_slowpace366.pyregenerates them from the manifest if a run needs them.3. Rollout runtime (
da1429cd…fa072050):decoder_attributecontrols and transactionalapply_control_bindings,execution_prefix, replay / hold tempo, fastest-stream chunk termination with its waypoint cap, HDF5 episode recording, the rollout timer, station yaml without inlineinference_graph. Its vendored codec copies are removed again in group 4.--source realsense), the HDF5 tempo checker, the stalled-episode /max_stepsfixes.yam-tracikextra),--check-modelCPU preflight, towels / earbuds HPT300 rollout profiles.4. Decoding the stack's own tokens + station extras:
BimanualArcDecoderdecodes every layout these recipes train with the liveegomimic.rldb.zarrtokenizers (absolute timing, hybrid R, event / reference / projection, speed floors) with the same tempo and termination controls; the live E1 codec gains decode-only helpers (detokenize(times=),as_duration_token,stream_clocks,stream_ends), tokenize untouched.FirstStreamArcDecoderexecutes the live multistream interval tokens to their first moving stream. The E1 contracts declarefirst_stream: 1,execute_percent: 50and the four controls (model.arc_decoder_controls); thetimecontract wraps its chunk in aTimeChunkRetimerfor the naive speed-up. No checkpoint-era tokenizer copies: a bundle trained by another tokenizer revision needs that revision's runtime.reset_overlay_templates.yaml,egomimic.scripts.build_reset_guides).docs/YAM_RUNTIME.mddocuments the decoder, the declared controls, the IK options, the monitors and the reset guides.Decisions on the open questions from the #229 handoff
YAM_HPTFLOW_*env variables select another.decoder_attributetargets; fastest-stream is on by default in the E1 contracts (what the station ran), the dashboard turns it off.fold towels); the per-task profiles pin their own.Not carried on purpose: the overlapped-inference loop (tried 10-08, reverted on the arms), the station's hash-bound
pinned_arc_bbb813a0routing and the live-treeBimanualArcClockDecoder(both depend on byte-exact tokenizer snapshots), and the station's uncommittedhpt_stems/ teleop edits (main already has them).Verification (WSL CPU venv)
ruff 0.8.6check + format clean (main's gate).scripts/audit_hydra_configs.py: 415 / 415 YAMLs compose and resolve; all 17 task leaves and 7 bases resolve to their variant, M, D, W&B id and declared decoder.scripts/e1/validate_stattempo_configs.py: PASS, the 6 stationery recipes match8e80f090.--check-configthen--check-modelon the towels profile, then a rollout as the validation of the decoder path.🤖 Generated with Claude Code