Cosmos3 Inverse/Forward Dynamics: how to obtain observation.state (LeRobot-compatible) alongside predicted actions?

[BlackMamba19]

I am evaluating Cosmos3 inverse dynamics (video → action) and I would like to build LeRobot-format episodes for downstream training/finetuning (e.g., observation.images.*, observation.state, action, timestamps, etc.).
Cosmos3 inference currently returns an action tensor (e.g. [T, raw_action_dim]) but it is unclear how I should obtain the corresponding state observations (observation.state) required by LeRobot/robot-learning datasets.

---

Context / Goal

• I run Cosmos3 inverse dynamics on an input video and obtain:

  • action.data with shape [T, raw_action_dim]

• I want to append new episodes to an existing dataset in LeRobot/GR00T format (parquet + meta + mp4), which expects at minimum:

  • observation.images.<cam> (video frames)
  • observation.state (per-timestep state vector)
  • action (per-timestep action vector)
  • timestamps / frame_index / episode_index, etc.

My blocker: given only (video, predicted action), how should I obtain observation.state to create a coherent episode?

[mli0603]

Hi @BlackMamba19

We are releasing ID code / data examples. You will be able to see how different datasets provide the information. + @lzhang57 for vis.

[Rohithmatham12]

I opened a small docs PR to clarify the current boundary here: #212.

The key point documented there is that inverse dynamics predicts action trajectories from video, but it does not synthesize a full LeRobot episode by itself. Fields such as observation.state, timestamps, frame indexes, and camera streams need to come from the source dataset or robot logging pipeline, then the predicted action rows should be aligned to those original per-timestep observations. The PR also points to the checked-in DROID LeRobot sample metadata as the current concrete reference.