[yuecideng]

Description

Add a source-backed Real2Sim path from annotated trajectory bundles through Atomic Skill assembly, bounded kinematic refinement, spatial/time expansion, sequential Task Program execution, and synchronous LeRobot recording. Pick, MoveHeldObject and Place retain their existing identities and complete-plan semantics.

• Normalize named joints, units, frames and hard events into content-pinned JSON/NPZ prototypes. Keep measured state, synthetic reference and submitted commands distinct; support qpos, EEF and explicitly phased hybrid authority.
• Retarget compatible tabletop geometry and optionally refine EEF motion against task goals, antipodal mesh candidates, calibrated gripper geometry, the confirmed held relation, and physical support footprint/release-gap checks. Sparse fixed-grid pose fitting preserves gravity and corrected contact/hold boundaries. IK, sampled motion/collision checks and measured effects remain separate acceptance gates; repair failure never falls back to native planning.
• Apply existing same-grid operators or mapped-time recipes to the corrected baseline with intersected phase permissions. Episode identity includes the refinement profile; realized pose hashes remain call-level evidence so small physical variations do not rename a deterministic recipe.
• Save actual submitted actions, T+1 observations, source bundles, original/refined TCP arrays, mappings, measured outcomes and matching receipts. Verify refinement digests and correction budgets before confirming persistence; preserve pending payloads for write-only repair.

Refs #709 and #708. Qualification is synthetic, B=1, same robot/assets, with a dynamic box, static support box and fixed ground. Real-data/hardware qualification, general joint-space collision optimization, continuous collision, RL, cross-robot transfer, parallel calls and B>1 host execution remain outside this slice.

Relationship to #799

This is an independent implementation against main and overlaps #799, including incompatible definitions of Real2SimTrajectory. The two PRs require an explicit integration or selection decision before both merge; this PR does not depend on #799.

Dependencies: existing SciPy, OSQP, Pinocchio/Coal and LeRobot dependencies; no package dependency changes.

Type of change

• New feature (non-breaking change which adds functionality)

Validation

• Refinement regression: 54/54 error-injected episodes across nine layouts and six nominal/joint/task-space/time recipes, with 0 failed attempts. Maximum IK/FK position residual 0.545 mm, minimum measured lift 9.99 cm, maximum placement XY error 2.97 mm.
• Final seven-case suite: clean, biased, drifting, repeated Pick/Move/Place, same-seed repeat, unrefined replay rejection and over-budget rejection all passed. Rejected cases retain explicit attempt evidence and have no confirmed episode. Same-seed recipe/prototype/profile identities match; final object position differs by 0.0684 mm.
• 60 successful episodes in 15 datasets independently re-audited for source/asset hashes, original/refined TCP digests, correction budgets, hard-event maps, measured outcomes, attachments and receipts. An initial trace-copy validation defect was fixed; its staged episode was confirmed by write-only repair without another rollout.
• Analytic ablation on identical inputs: replay position RMSE about 6.67 mm, rigid-anchor correction 1.05 mm, sparse refinement 0.015 mm. Independent straight-line/rotation fixtures also cover confidence-weighted outliers, authority protection, correction/iteration budgets and invalid measurements.
• CPU numerical benchmark: float64, 128/512/2048 samples, two warmups and seven repeats; completed-call latency and separately measured retained RSS are reported alongside accuracy. This excludes robot IK and simulation, and does not claim transient peak memory or GPU performance.
• 638 affected tests passed across numerical computation/import boundaries, Atomic Skills, support/collision, configured integration, recording/receipts, CLI, docs tooling and project context. Existing production fixtures were extended for source repair, RNG isolation, invalid candidate costs and tampered refinement receipts; tutorial scripts are not test targets.
• Black 26.10.1, diff checks, context validation and API coverage 3353/3353 pass. Sphinx dummy build passes with cross-reference/docstring warnings, using Python 3.10 native autodoc and Python 3.13 source-only architecture extraction.
• The 54-case run precedes the final additional validation/identity gates; the final seven-case suite and focused tests exercise those gates. All recorded episodes were re-audited after the refinement checks were added. The checked-in report records production-file hashes and this revision scope.

See examples/expansion/trajectory/real2sim/REFINEMENT.md for the verification ladder and rerun commands, refinement_validation_report.json for machine-readable results, and refinement_comparison.png for the analytic comparison. The original nominal and repeated-program acceptance reports remain available separately.

Greptile review follow-up

• Give each retained episode its own write state. Receiptless staging/submission failures resume the outstanding submission; final failed receipts advance it once. Receipt publication validates every identity, fsync retries cover existing links, and publication/reset/cleanup failures do not duplicate a completed submission or successful reset.
• Validate external Pick's final TCP against the requested world-Z lift target and grasp orientation, using the same pose tolerances as its grasp anchor.
• Preflight normalization sample count and estimated generated-array/work-buffer bytes before allocating the control grid or casting event indices. Python preparation/loading and CLI expose reader-owned budgets; manifests cannot enlarge them. The estimate is not a process-RSS guarantee.
• Apply grasp-frame-to-EEF calibration before sampled-candidate correction budgets and IK, while evaluating jaw width in the canonical grasp frame. Explicit/fixed EEF goals retain their existing frame semantics.
• Correct the CLI command inventory and freeze the compatibility test to historical utility exports, rather than requiring every new compute algorithm in the legacy namespace. The two unrelated RigidFormer failures from the previous CI merge are already repaired on main; no optional dependencies were added here.

Validation of follow-up commit de94d3f7a:

• 802 affected tests passed: 661 normalization, owner, Atomic Skill, Task Program and CLI/compatibility tests; 141 recorder/context tests. New regressions exercise receiptless failures before/after a successful episode, interrupted retries with the real ExpansionSession, directory-sync/reset/cleanup recovery, requested lift direction/height, nonidentity calibration and pre-allocation rejection. Existing fixtures cover the remaining paths; tutorial scripts are not test targets.
• One fresh, bias-injected refined Pick → MoveHeldObject → Place episode passed in DexSim and synchronous LeRobot. Independent file-based audit verified all three calls, source/refinement attachments, hard-event mappings, measured outcomes and the confirmed receipt: 707 steps, 10.03 cm lift, 2.70 mm final placement XY error, 0 failures. This is a focused post-review smoke; the larger matrix and seven-case results above remain evidence from the earlier revisions, not reruns at this commit.
• Black 26.10.1, diff/header checks, context validation and API coverage 3353/3353 pass. Verification used a process-local correction for an editable installation pointing at another worktree; neither the shared environment nor repository packaging was changed.
• Context now documents the normalization budget. The recovery, lift and calibration changes restore the existing owner/skill contracts, so no additional routing or architecture change is needed.

Screenshots

Checklist

• I have run black . to format the code base.
• I reviewed affected documentation and agent context and updated the Real2Sim owner and motion navigation.
• Public API changes are reflected in the API docs.
• Behavioral coverage and test necessity were reviewed; analytic, integration, rejection and physical/persistence evidence are distinguished.
• Dependencies have been reviewed; no dependency changes are needed.

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge based on the reviewed changes.

Summary

Adds source-backed Real2Sim prototypes, bounded pose refinement, trajectory expansion, sequential execution, and synchronous recording with matching receipts.

• Annotated trajectories become pinned prototypes with clear joint, frame, and event data.
• Pick, MoveHeldObject, and Place can use source motion with bounded TCP repair.
• Expansion recipes change prototype motion in space or time before task execution.
• Task Programs route each selected source and transform to the right call.
• The tabletop collector traces, records, and confirms each source-backed episode.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Annotated source bundle] --> B[Normalize and pin prototype]
  B --> C[Retarget and optionally refine]
  C --> D[Assemble skill plan]
  D --> E[Apply permitted expansion]
  E --> F[Check motion and sampled collisions]
  F --> G[Execute sequential program]
  G --> H[Check measured outcomes]
  H --> I[Stage and save episode]
  I --> J[Publish matching receipt]
  J --> K[Reset environment]

Loading

_{Reviews (5) · Last reviewed commit: "test(sim): tolerate cleanup during mater..." · Reviewed by Greptile}