From World Models to World Action Models: A Concise Tutorial for Robotics

Language: English | 中文

Xiaoxiong Zhang, Xiong Zeng, and Wei Zhang
Southern University of Science and Technology; LimX Dynamics

ArXiv

The following is a short summary for the tutorial. More details can be found in the arXiv Paper.

Architectural view

A world is the set of task-relevant entities, including the robot and its environment. A physical AI task asks a policy to drive the world from an initial state toward a goal set while satisfying task-specific constraints.

World	Physical AI tasks	Policy framework

World models and world action models

A world model predicts how future observations or states evolve under candidate actions, typically conditioned on the current observation. A world action model is a policy whose action generation is coupled, during training or inference, with a model or representation of future world evolution.

World model	World action model

Design space

We organize world models according to the space in which prediction is performed:

• Observation-space world models directly predict future observations, such as RGB images, multi-view RGB, RGB-D frames, or point clouds.
• State-space world models first abstract observations into structured states, then model future evolution in that state space.

Observation-space world models	State-space world models

World action model taxonomy

World action models couple future-oriented visual prediction with physical decision making. The tutorial groups representative methods into four paradigms: imagine-then-execute, video-feature-conditioned action prediction, joint video-action modeling, and auxiliary video prediction for policy learning.

Resource browser

The companion website includes a filterable paper list aligned with the survey taxonomy:

https://clearlab-sustech.github.io/WorldModelSurvey/#resources

Citation

@article{zhang2026worldactionmodels,
  title   = {From World Models to World Action Models: A Concise Tutorial for Robotics},
  author  = {Zhang, Xiaoxiong and Zeng, Xiong and Zhang, Wei},
  year    = {2026},
  note    = {Survey manuscript}
}

Contact

Suggestions, taxonomy improvements, missing papers, and discussion are welcome:

12433017@mail.sustech.edu.cn