Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion agent_context/MAP.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -283,7 +283,8 @@ topics:
- examples/learning/nerd/_support/
- examples/learning/nerd/configs/
- extensions/embodichain_models/embodichain_models/dynamics/rigidformer/
watch_paths: [extensions/embodichain_models/, examples/learning/nerd/,
- examples/learning/rigidformer/
watch_paths: [extensions/embodichain_models/, examples/learning/nerd/, examples/learning/rigidformer/,
docs/source/features/neural_dynamics.md,
docs/source/api_reference/model_catalogs.rst]
related_topics: [supervised-learning, learning-layers, data-pipeline, data-assets]
Expand Down
9 changes: 7 additions & 2 deletions agent_context/topics/neural-dynamics/neural-dynamics.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,8 +57,13 @@ and at least two valid objects. Attention backend and source semantics belong
to recipe/recovery identity. Source-owned epoch sampling does not claim the
legacy research DataLoader's worker-RNG trajectory.

Numerical agreement, recovery and full benchmark/retraining qualification
remain separate evidence requirements.
The [showcase](../../../examples/learning/rigidformer/README.md) owns downloads,
conversion, fixed-weight evaluation and independent author-Trainer comparison.
Its `demo` command in `_demo.py` compares live Newton spheres with render-only
model predictions after four observed frames, through a native window or an
engine-camera recording. It uses explicit prepared geometry and weight paths.
Assets/results stay outside the wheel. Numerical agreement, recovery and full
benchmark/retraining qualification remain distinct evidence requirements.

## Resource and physical identity

Expand Down
7 changes: 5 additions & 2 deletions docs/source/api_reference/model_rigidformer.rst
Original file line number Diff line number Diff line change
Expand Up @@ -146,5 +146,8 @@ source identity also binds splits, scaling, sampling and seed. EMA shadows remai
FP32 and retain the author's warmup policy. Official weight import is distinct
from native optimizer recovery; export selects EMA or raw weights explicitly.

Applications own acquisition and independent reference experiments. Short
numerical comparisons do not establish published accuracy.
The showcase owns asset preparation, the author evaluator and independent
reference comparison. Fixed-checkpoint test evaluation does not select a
checkpoint. New learning recipes should select using validation data. Reports
record source/data/checkpoint identities, attention backend and subset/full scope.
Short numerical comparisons do not establish published accuracy.
6 changes: 5 additions & 1 deletion embodichain/learning/sl/types.py
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,11 @@ def on_update(self, learner: SupervisedLearner) -> None:
...

def state_dict(self) -> Mapping[str, Any]:
"""Return an owned serializable snapshot of the committed state."""
"""Return an owned serializable snapshot of the committed state.

Returns:
State whose mutable values do not alias the live participant.
"""
...

def validate_state_dict(self, state: Mapping[str, Any]) -> None:
Expand Down
6 changes: 6 additions & 0 deletions examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ embodichain run-example sim/motion/planners/curobo_planner --help
embodichain run-example ods/simple_bc_demo/train --help
embodichain run-example learning/nerd --help
embodichain run-example learning/rwm --help
embodichain run-example learning/rigidformer demo --help
embodichain run-example sim/atomic_action --skill place --variants 8
embodichain run-example sim/atomic_action/coordinated_pickment --help
```
Expand Down Expand Up @@ -48,6 +49,11 @@ state-transition fitting and state/action gradients. Its generated recipes use
the installable `embodichain_models.dynamics.rwm` model family and shared SL
trainer; its corpus generation and acceptance helpers remain private to the example.

The [RigidFormer showcase](learning/rigidformer/README.md) prepares official
weights and geometry, trains and evaluates the model, and runs an engine demo
with Newton spheres beside a free model rollout. The demo opens a native window
or records an engine-camera video with `--headless`.

The [Atomic Skill example catalog](sim/atomic_action/README.md) owns all per-skill,
coordinated-manipulation and recovery demonstrations. The two source-checkout
walkthroughs under `scripts/tutorials/atomic_action/` teach single-arm and dual-arm
Expand Down
228 changes: 228 additions & 0 deletions examples/learning/rigidformer/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,228 @@
# RigidFormer author implementation

This showcase integrates the author's RigidFormer implementation at
`160fe0852c51cffae7b7440452b58d0b49db6df6` from
[Frank-ZY-Dou/Dynamics-Modeling](https://github.com/Frank-ZY-Dou/Dynamics-Modeling/tree/160fe0852c51cffae7b7440452b58d0b49db6df6/RigidFormer).
The model lives at `embodichain_models.dynamics.rigidformer` and trains through
the common SL trainer. The community `rigidformer` package is not a runtime dependency.

## Setup and ownership

Use a Python 3.11 environment and a CUDA-capable Torch installation:

```bash
conda activate py311
python -m pip install 'embodichain[rigidformer]'
embodichain run-example learning/rigidformer --help
```

From a source checkout, use `python -m pip install -e '.[rigidformer]'` to install
that checkout and its optional model dependencies.

The standard native attention backend is Torch SDPA. The explicit `flash`
backend requires a Flash-Attention build compatible with Torch, CUDA and the
GPU. Reference evidence records the actual dependency/backend versions.
Imports, architecture construction and checkpoint inspection work on CPU;
geometry prediction uses the author's CUDA KNN path and needs at least two
valid objects per scene.

Model/config/checkpoint/data/objective/recipe code is installable under the
model family. Downloading assets, conversion and reference experiments remain
showcase-owned. Private author-derived source retains MIT attribution in
`THIRD_PARTY_LICENSES`. Official scenes are CC BY 4.0; collision meshes and
derived geometry are CC BY-SA 4.0. Those licenses remain attached to prepared
data. Data, checkpoints, source checkouts and reports are not bundled in the wheel.

## Prepare official assets

```bash
embodichain run-example learning/rigidformer prepare --dataset movi_a
```

Data preparation is independent of pretrained weights. Download those separately
with `prepare --dataset movi_a --assets weights`, or select `--assets all`.
Training from scratch requires no pretrained checkpoint.

The downloader pins dataset revision
`e78514b479b11acfa5d23d28f1211d7830646ff9` and weights revision
`8f2c0a3a157b64e4b310c64a01eee6e4e7e25af6`. It verifies published checksums.
Use `movi_sphere`, `movi_a`, or `movi_b`. Numeric scenes and collision meshes
are sufficient; videos are not downloaded.

Prepared files use the author's packed layout, with aligned material points,
complete object identities (including recovered objects), physical properties,
reference anchors/FPS order, and a content-hashed manifest. Stored vertices are
**unscaled asset vertices transformed into world coordinates**. The size sidecar
applies each object's metadata size about its origin exactly once during reads.

By default preparation requires all 960 training, 120 validation and 120 test
scenes. `--limit-per-split N` creates a clearly labelled debug subset.
`--local-files-only` requires already downloaded, verified assets.
Every output dataset needs a fresh directory. Failures after conversion starts
remain recorded; checksum preflight rejects bad assets before creating output.
Readers reject altered payloads and geometry omitted from the manifest. FPS and
anchor caches are committed inputs, rather than untracked files generated during
training. Debug preparation verifies collision meshes as well as scene shards.

## Engine demo

Prepare the Sphere weights and a small geometry subset, then open the engine
window:

```bash
embodichain run-example learning/rigidformer prepare \
--dataset movi_sphere --assets all --limit-per-split 1

embodichain run-example learning/rigidformer demo
```

The left side contains dynamic Newton spheres. The right side contains engine
`VisualObject` spheres driven by the official RigidFormer model. Both sides use
the same geometry, masses, friction and restitution from a prepared test scene.
The model observes four sampled simulation frames, then predicts the remaining
trajectory in one free rollout. Predicted spheres have no colliders and provide
no feedback to the physical simulation. The window holds the final frame for
three seconds; Ctrl+C closes the demo early.

To record on a machine without a display, pass `--headless`:

```bash
embodichain run-example learning/rigidformer demo --headless \
--output outputs/rigidformer/demo-headless
```

The output directory must be fresh. It contains engine-camera `rollout.mp4`,
`frame.png`, aligned `trajectory.npz`, and a `summary.json` with source hashes,
timing and center errors. Frames stream to the video writer as they are captured.
The defaults show 51 samples at stride 5 from 240 Hz physics, played at 12 fps.
`--frames`, `--stride`, `--fps` and `--hold-seconds` control the demonstration.
`--scene` selects a prepared test scene; `--object-ids` selects two or three
template objects. `--device cuda:1` selects a different visible GPU.

Use existing resources or compatible exported inference weights explicitly:

```bash
embodichain run-example learning/rigidformer demo --headless \
--data-root /path/to/prepared/MOVi-Sphere \
--checkpoint /path/to/rigidformer_movisphere.pth \
--output outputs/rigidformer/my-demo
```

The scene uses Newton XPBD and newly chosen initial positions and velocities.
Its center errors describe this engine comparison. The fixed-checkpoint
`evaluate` command below reports the official MOVi data metrics. The preparation
limit bounds converted scenes; downloading may still acquire the dataset's
numeric shards and shared meshes.

## Train with shared SL

```bash
embodichain train-sl --config examples/learning/rigidformer/configs/sl.movi_a.yaml
```

Paths resolve from the caller's current working directory. The preset recipes
use eight-frame windows, four anchors, variable integration strides 1/5/10,
author augmentation, anchor-only before/after-Kabsch position and acceleration
losses, grouped AdamW, gradient clipping, epoch scheduling and FP32 EMA.

The author model integrates with **model dt=1**. Physical time is
`step_size / 240` seconds. Loss scaling and conditioning use the recorded
step size; do not replace the model integration unit with seconds.
FP32 geometry augmentation precedes BF16 network autocast; integration and
rigid registration retain the author's numerical policy.

Each source shuffles and shards its epoch traversal, drops an incomplete batch,
and stages the cursor until optimizer success. The source-sampling stream is
explicit and recoverable; the legacy research DataLoader's worker-RNG trajectory
is not claimed bitwise identical. Checkpoints preserve sampling, model/optimizer,
scheduler, EMA and per-rank state. A changed source/split/scale/recipe/world size
is rejected before restoration.

For multiple GPUs, use the same launcher:

```bash
OMP_NUM_THREADS=1 torchrun --standalone --nnodes=1 --nproc-per-node=4 \
-m embodichain train-sl --config examples/learning/rigidformer/configs/sl.movi_a.yaml
```

Batch size is per rank, and learning rates are not scaled automatically.
The released MOVi-A/Sphere training layout uses four GPUs x 32 scenes;
MOVi-B uses six x 21. A smaller training run is useful for integration checks
but is not equivalent to that reference training budget. Native DDP preserves
the author's synchronized encoder statistics and uses a global valid-anchor
loss mean exactly once.

Resume through the common entry point; change only the output directory when
continuing an older checkpoint into a fresh run destination:

```bash
embodichain train-sl --config my-recipe.yaml --resume outputs/run/last.pt
```

## Export and evaluate

```bash
embodichain run-example learning/rigidformer export \
--checkpoint outputs/run/last.pt --weights ema --output outputs/model-ema.pt

embodichain run-example learning/rigidformer evaluate \
--checkpoint outputs/model-ema.pt \
--data-root outputs/rigidformer/data/MOVi-A \
--split validation --output outputs/rigidformer/eval-validation
```

Official weights can be passed directly to `evaluate`. Exported native weights
use the same named tensor/config layout; importing weights does not recover an
official optimizer process. EMA/raw selection is explicit, and an absent EMA
does not silently fall back to raw weights.

The fixed-checkpoint evaluator uses four ground-truth warmup frames, step sizes
1/5/10, and the author's HopNet-aligned position/rotation metrics at physical
frames 25/50/75/100/200/300/400. Off-grid checkpoints are not interpolated.
Reports include corpus scope and dependency/source/checkpoint identities.
A debug-subset result does not establish complete benchmark accuracy.

New native runs do not select a checkpoint from test scores. Use validation to
select, freeze the checkpoint, then evaluate test. The author's default research
training recipe monitors test metrics; that selection-protocol difference must
remain visible in any retraining comparison.

## Independent reference comparison

Supply an unmodified author checkout's `RigidFormer` directory at the pinned
revision. The runner verifies source hashes in `REFERENCE_SOURCES.json` before
importing the independent original model and Trainer.

```bash
embodichain run-example learning/rigidformer compare \
--reference /path/to/Dynamics-Modeling/RigidFormer \
--checkpoint outputs/model-ema.pt \
--data-root outputs/rigidformer/data/MOVi-A \
--attention-backend flash --updates 3 --output outputs/rigidformer/compare
```

Comparison checks prediction, free rollout, losses, gradients and paired optimizer
updates on identical fixed inputs and randomness. `flash` executes the genuine
Flash-Attention package. Explicit `sdpa` uses a documented compatibility operator
for the reference's unmasked attention; its results do not qualify Flash execution.
Reports distinguish official released weights from other explicitly provided weights.
They also compare optimizer moments/groups, buffers, EMA and epoch schedules,
and record selected frames, source hashes and the executed numerical runtime.
Each paired update represents one fixed-batch comparison epoch; it is separate
from the source's normal training-epoch length.

Check exact continuous versus resumed training using the same installable recipe:

```bash
embodichain run-example learning/rigidformer check-recovery \
--config my-four-update-recipe.yaml --stop-after 2 --output outputs/recovery-check
```

This creates fresh full/split/resumed runs, compares complete committed state
exactly, and verifies raw/EMA exports. The [acceptance protocol](research/acceptance.md)
also covers epoch/warmup boundaries, two-rank NCCL and installed-wheel checks.

This is a numerical-integration check. Full published-accuracy reproduction
additionally requires frozen official weights/data/evaluation, or complete
retraining with the declared compute and selection protocol. No short-run
comparison is labelled a full training reproduction.
21 changes: 21 additions & 0 deletions examples/learning/rigidformer/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# ----------------------------------------------------------------------------
# Copyright (c) 2021-2026 DexForce Technology Co., Ltd.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ----------------------------------------------------------------------------

"""Author-based RigidFormer integration and frozen reference workflows."""

from __future__ import annotations

__all__: list[str] = []
Loading
Loading