Diffusion-based World Models: A Survey

A Living Survey, Curated Paper List, Taxonomy, Dataset Hub, and Research Roadmap

Overview · Premium Visuals · Original Figures · Project Structure · Datasets · Paper List · Citation

Toward controllable, interactive, and physically grounded world intelligence.

---

📢 News

• [2026] We release this project-page-style GitHub repository for Diffusion-based World Models.
• [2026] The repository now includes both original paper figures and redesigned premium visuals for better presentation.
• [2026] The paper list currently covers 360+ references spanning 2016–2026.
• [2026] We organize the topic into autonomous driving, embodied intelligence, and general-purpose worlds.

---

🌐 Overview

World models aim to learn an internal simulator of the real world, enabling intelligent agents to perceive, predict, imagine, reason, plan, and act. Diffusion models have recently become a powerful foundation for world modeling because they support multimodal generation, high-fidelity future synthesis, conditional controllability, and long-horizon imagination.

This repository is designed in a project page style rather than a plain paper list. It provides:

📚 Curated Paper List A living collection of representative papers on diffusion-based world models and related foundations.

🧭 Unified Taxonomy A structured map of tasks, domains, capabilities, datasets, and research trends.

🖼️ Visual Assets Both paper-original figures and redesigned premium visuals for GitHub presentation.

🔭 Research Roadmap A concise summary of open problems and future opportunities for the field.

---

🧠 Core Themes

Theme	Description
Autonomous Driving	Driving scene generation, trajectory-conditioned simulation, occupancy/BEV forecasting, closed-loop planning, and long-tail scenario synthesis.
Embodied Intelligence	Action-conditioned prediction, robot manipulation, VLA reasoning, policy learning, sim-to-real transfer, and interactive imagination.
General-purpose Worlds	Long video generation, digital twins, game-like simulation, 3D/4D worlds, and scalable world foundation models.
Key Capabilities	Long-horizon evolution, multimodal fusion, interactivity, spatiotemporal consistency, and environment diversification.
Open Problems	Efficiency, controllability, causal reasoning, physical grounding, evaluation, and verification.

---

🧪 Datasets & Benchmarks

Domain	Representative Datasets
Autonomous Driving	nuScenes, Waymo, Argoverse 2, nuPlan, NAVSIM, OpenDV
Embodied Intelligence	Open X-Embodiment, RT-1, CALVIN, LIBERO, BridgeData, Ego4D
General-purpose Worlds	UCF101, MSR-VTT, SSv2, WebVid, Minecraft, Procgen, MiniGrid

For a more organized summary, see:

• papers/datasets.md
• resources/benchmarks.md

---

🗂️ Project Structure

Diffusion_based_World_Models_GitHub_Final/
├── README.md
├── CONTRIBUTING.md
├── assets/
│   ├── project-page-preview.png
│   ├── dwm-banner-premium.png
│   ├── dwm-principle-premium.png
│   ├── dwm-original-figures-premium.png
│   ├── dwm-banner.png
│   ├── dwm-concept.png
│   ├── dwm-paper-galaxy.png
│   ├── dwm-datasets.png
│   └── paper_figures/
├── papers/
│   ├── README.md
│   ├── autonomous-driving.md
│   ├── embodied-intelligence.md
│   ├── general-worlds.md
│   └── datasets.md
└── resources/
    ├── README.md
    ├── awesome-world-models.md
    ├── benchmarks.md
    └── survey-notes.md

---

📑 Paper List

The following paper list is continuously updated. We welcome pull requests for missing papers, code links, project pages, datasets, and benchmark resources.

📑 Year Directory

• 2026
• 2025
• 2024
• 2023
• 2022
• 2021
• 2020
• 2019
• 2018
• 2016

---

2026

• VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control CVPR 2026 | Paper | Code
• NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos CVPR 2026 | Paper | Code
• LongStream: Long-Sequence Streaming Autoregressive Visual Geometry CVPR 2026 | Paper | Code
• VideoWorld 2: Learning Transferable Knowledge from Real-world Videos arXiv 26.02 | Paper
• ProPhy: Progressive Physical Alignment for Dynamic World Simulation arXiv 26.04 | Paper | Code
• Chain of Event-Centric Causal Thought for Physically Plausible Video Generation CVPR 2026 | Paper
• Taming Video Models for 3D and 4D Generation via Zero-Shot Camera Control CVPR 2026 | Paper
• DriveLaW: Unifying Planning and Video Generation in a Latent Driving World CVPR 2026 | Paper | Code
• ABot-PhysWorld: Interactive World Foundation Model for Robotic Manipulation with Physics Alignment arXiv 26.05 | Paper
• SimScale: Learning to Drive via Real-World Simulation at Scale CVPR 2026 | Paper
• 4DWorldBench: A Comprehensive Evaluation Framework for 3D/4D World Generation Models arXiv 25.11 | Paper
• WorldLens: Full-Spectrum Evaluations of Driving World Models in Real World arXiv 25.12 | Paper | Code
• GeoWorld: Geometric World Models CVPR 2026 | Paper | Code
• Free-Lunch Long Video Generation via Layer-Adaptive O.O.D Correction CVPR 2026 | Paper | Code
• Astra: General interactive world model with autoregressive denoising ICLR 2026 | Paper | Code
• seq-JEPA: Autoregressive Predictive Learning of Invariant-Equivariant World Models arXiv 25.05 | Paper | Code

---

2025

• UniWorld-V1: High-Resolution Semantic Encoders for Unified Visual Understanding and Generation arXiv 25.06 | Paper
• MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Control ICCV 2025 | Paper | Code
• VLM-SAFE: Vision-Language Model-Guided Safety-Aware Reinforcement Learning with World Models for Autonomous Driving arXiv 25.05 | Paper | Code
• Statespacediffuser: Bringing long context to diffusion world models arXiv 25.05 | Paper
• Understanding world or predicting future? a comprehensive survey of world models arXiv 24.11 | Paper
• The role of world models in shaping autonomous driving: A comprehensive survey arXiv 25.02 | Paper
• A survey: Learning embodied intelligence from physical simulators and world models arXiv 25.07 | Paper
• I think, therefore i diffuse: Enabling multimodal in-context reasoning in diffusion models arXiv 25.02 | Paper | Code
• Vidar: Embodied video diffusion model for generalist manipulation arXiv 25.07 | Paper
• Epona: Autoregressive diffusion world model for autonomous driving ICCV 2025 | Paper | Code
• Gaia-2: A controllable multi-view generative world model for autonomous driving arXiv 25.03 | Paper
• Cosmos-drive-dreams: Scalable synthetic driving data generation with world foundation models arXiv 25.06 | Paper | Code
• Avd2: Accident video diffusion for accident video description ICRA 2025 | Paper | Code
• Cvd-storm: Cross-view video diffusion with spatial-temporal reconstruction model for autonomous driving arXiv 25.10 | Paper
• Dive: Efficient multi-view driving scenes generation based on video diffusion transformer arXiv 25.04 | Paper
• Dist-4d: Disentangled spatiotemporal diffusion with metric depth for 4d driving scene generation arXiv 25.03 | Paper | Code
• Dreamland: Controllable world creation with simulator and generative models arXiv 25.06 | Paper
• Drive&gen: Co-evaluating end-to-end driving and video generation models IROS 2025 | Paper
• Driverse: Navigation world model for driving simulation via multimodal trajectory prompting and motion alignment arXiv 25.04 | Paper | Code
• Dynamiccity: Large-scale 4d occupancy generation from dynamic scenes ICLR 2025 | Paper | Code
• Lidarcrafter: Dynamic 4d world modeling from lidar sequences AAAI 2026 | Paper | Code
• Mila: Multi-view intensive-fidelity long-term video generation world model for autonomous driving arXiv 25.03 | Paper
• Maskgwm: A generalizable driving world model with video mask reconstruction arXiv 25.02 | Paper
• Orbis: Overcoming challenges of long-horizon prediction in driving world models arXiv 25.07 | Paper | Code
• Scenediffuser++: City-scale traffic simulation via a generative world model CVPR 2025 | Paper
• Uniscene: Unified occupancy-centric driving scene generation arXiv 24.12 | Paper | Code
• Imagine-2-drive: Leveraging high-fidelity world models via multi-modal diffusion policies arXiv 24.11 | Paper
• Imagidrive: A unified imagination-and-planning framework for autonomous driving arXiv 25.08 | Paper | Code
• UniFuture: A 4D Driving World Model for Future Generation and Perception ICRA 2026 | Paper | Code
• Cosmos-transfer1: Conditional world generation with adaptive multimodal control arXiv 25.03 | Paper | Code
• Cosmos World Foundation Model Platform for Physical AI arXiv 25.01 | Paper | Code
• Diwa: Diffusion policy adaptation with world models arXiv 25.08 | Paper | Code
• Dreamgen: Unlocking generalization in robot learning through video world models arXiv 25.05 | Paper
• Ladi-wm: A latent diffusion-based world model for predictive manipulation CoRL 2025 | Paper
• Navigation world models CVPR 2025 | Paper | Code
• Vid2world: Crafting video diffusion models to interactive world models arXiv 25.05 | Paper | Code
• Cot-vla: Visual chain-of-thought reasoning for vision-language-action models arXiv 25.03 | Paper
• Ctrl-world: A controllable generative world model for robot manipulation arXiv 25.10 | Paper
• Echoworld: Learning motion-aware world models for echocardiography probe guidance CVPR 2025 | Paper | Code
• Flare: Robot learning with implicit world modeling arXiv 25.05 | Paper
• Gwm: Towards scalable gaussian world models for robotic manipulation ICCV 2025 | Paper
• Keyworld: Key frame reasoning enables effective and efficient world models arXiv 25.09 | Paper
• Longscape: Advancing long-horizon embodied world models with context-aware moe arXiv 25.09 | Paper
• Mind: Learning a dual-system world model for real-time planning and implicit risk analysis arXiv 25.06 | Paper
• Mindjourney: Test-time scaling with world models for spatial reasoning arXiv 25.07 | Paper | Code
• Mowm: Mixture-of-world-models for embodied planning via latent-to-pixel feature modulation arXiv 25.09 | Paper
• Playerone: Egocentric world simulator arXiv 25.06 | Paper | Code
• Vistav2: World imagination for indoor vision-and-language navigation arXiv 25.12 | Paper
• Whole-body conditioned egocentric video prediction arXiv 25.06 | Paper
• Wmpo: World model-based policy optimization for vision-language-action models arXiv 25.11 | Paper | Code
• World4rl: Diffusion world models for policy refinement with reinforcement learning for robotic manipulation arXiv 25.09 | Paper
• World-in-world: World models in a closed-loop world arXiv 25.10 | Paper | Code
• Motion prompting: Controlling video generation with motion trajectories CVPR 2025 | Paper
• Storyweaver: A unified world model for knowledge-enhanced story character customization arXiv 24.12 | Paper | Code
• Aether: Geometric-aware unified world modeling arXiv 25.03 | Paper | Code
• Jedi: Latent end-to-end diffusion mitigates agent-human performance asymmetry in model-based reinforcement learning arXiv 25.05 | Paper
• Long-context state-space video world models arXiv 25.05 | Paper
• Longvie 2: Multimodal controllable ultra-long video world model arXiv 25.12 | Paper
• Martian world model: Controllable video synthesis with physically accurate 3d reconstructions arXiv 25.07 | Paper
• Model as a game: On numerical and spatial consistency for generative games arXiv 25.03 | Paper
• Morphosim: An interactive, controllable, and editable language-guided 4d world simulator arXiv 25.10 | Paper
• Omniworld: A multi-domain and multi-modal dataset for 4d world modeling arXiv 25.09 | Paper | Code
• Pre-trained video generative models as world simulators arXiv 25.02 | Paper
• Simulating before planning: Constructing intrinsic user world model for user-tailored dialogue policy planning arXiv 25.04 | Paper
• Terra: Explorable native 3d world model with point latents arXiv 25.10 | Paper
• Voyager: Long-range and world-consistent video diffusion for explorable 3d scene generation arXiv 25.06 | Paper
• Yume: An interactive world generation model arXiv 25.07 | Paper
• Fantasyworld: Geometry-consistent world modeling via unified video and 3d prediction arXiv 25.09 | Paper | Code
• π0.5: a vision-language-action model with open-world generalization arXiv 25.04 | Paper
• Open x-embodiment: Robotic learning datasets and rt-x models arXiv 25.05 | Paper | Code | Dataset
• Egodex: Learning dexterous manipulation from large-scale egocentric video ICLR 2026 | Paper | Code
• Robomind: Benchmark on multi-embodiment intelligence normative data for robot manipulation arXiv 24.12 | Paper
• Robotwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation arXiv 25.06 | Paper
• Evaluating precise geolocation inference capabilities of vision language models AAAI 2025 | Paper
• Diffuse and disperse: Image generation with representation regularization arXiv 25.06 | Paper
• CorrectAD: A Self-Correcting Agentic System to Improve End-to-end Planning in Autonomous Driving arXiv 25.11 | Paper
• 4D Driving Scene Generation With Stereo Forcing arXiv 25.09 | Paper
• CoGen: 3D Consistent Video Generation via Adaptive Conditioning for Autonomous Driving arXiv 25.03 | Paper
• UniMLVG: Unified Framework for Multi-view Long Video Generation with Comprehensive Control Capabilities for Autonomous Driving arXiv 24.12 | Paper
• Genie 3 DeepMind Blog 2025 | Paper | Code
• 3DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World Model arXiv 25.06 | Paper
• A Good Regulator May Provide a World Model for Intelligent Systems arXiv 25.06 | Paper
• AdaPower: Specializing World Foundation Models for Predictive Manipulation arXiv 25.12 | Paper
• Bridging the Gap Between Multimodal Foundation Models and World Models arXiv 25.10 | Paper
• Bootstrapping World Models from Dynamics Models in Multimodal Foundation Models arXiv 25.06 | Paper | Code
• Consistent World Models via Foresight Diffusion arXiv 25.05 | Paper
• Clone Deterministic 3D Worlds with Geometrically-Regularized World Models arXiv 25.10 | Paper
• Counterfactual World Models via Digital Twin-conditioned Video Diffusion arXiv 25.11 | Paper
• FlowDreamer: A RGB-D World Model with Flow-based Motion Representations for Robot Manipulation arXiv 25.05 | Paper | Code
• GAF: Gaussian Action Field as a Dynamic World Model for Robotic Manipulation arXiv 25.06 | Paper | Code
• Generative World Modelling for Humanoids: 1X World Model Challenge Technical Report arXiv 25.10 | Paper
• GigaWorld-0: World Models as Data Engine to Empower Embodied AI arXiv 25.11 | Paper | Code
• GrndCtrl: Grounding World Models via Self-Supervised Reward Alignment arXiv 25.12 | Paper
• HERO: Hierarchical Extrapolation and Refresh for Efficient World Models arXiv 25.08 | Paper
• Hunyuan-GameCraft-2: Instruction-following Interactive Game World Model arXiv 25.11 | Paper
• HunyuanWorld 1.0: Generating Immersive, Explorable, and Interactive 3D Worlds from Words or Pixels arXiv 25.07 | Paper
• Audio-Visual World Models: Towards Multisensory Imagination in Sight and Sound arXiv 25.12 | Paper
• ManipDreamer: Boosting Robotic Manipulation World Model with Action Tree and Visual Guidance arXiv 25.04 | Paper
• Matrix-Game 2.0: An Open-Source, Real-Time, and Streaming Interactive World Model arXiv 25.08 | Paper | Code
• Memory Forcing: Spatio-Temporal Memory for Consistent Scene Generation on Minecraft arXiv 25.10 | Paper | Code
• Motus: A Unified Latent Action World Model arXiv 25.12 | Paper | Code
• NORA-1.5: A Vision-Language-Action Model Trained using World Model- and Action-based Preference Rewards arXiv 25.11 | Paper | Code
• ORV: 4D Occupancy-centric Robot Video Generation arXiv 25.06 | Paper | Code
• PhysicalAgent: Towards General Cognitive Robotics with Foundation World Models arXiv 25.09 | Paper
• Probing the effectiveness of World Models for Spatial Reasoning through Test-time Scaling arXiv 25.12 | Paper
• RELIC: Interactive Video World Model with Long-Horizon Memory arXiv 25.12 | Paper
• Remote Sensing-Oriented World Model arXiv 25.09 | Paper
• Scalable Policy Evaluation with Video World Models arXiv 25.11 | Paper
• Semantic Communications with World Models arXiv 25.10 | Paper
• TesserAct: Learning 4D Embodied World Models arXiv 25.04 | Paper | Code
• Toward Stable World Models: Measuring and Addressing World Instability in Generative Environments arXiv 25.03 | Paper
• MTV-World: Towards High-Consistency Embodied World Model with Multi-View Trajectory Videos arXiv 25.11 | Paper
• Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets arXiv 25.04 | Paper | Code
• WorldPlanner: Monte Carlo Tree Search and MPC with Action-Conditioned Visual World Models arXiv 25.11 | Paper
• WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling arXiv 25.12 | Paper
• WoW: Towards a World omniscient World model Through Embodied Interaction arXiv 25.09 | Paper
• WristWorld: Generating Wrist-Views via 4D World Models for Robotic Manipulation arXiv 25.10 | Paper
• NRSeg: Noise-Resilient Learning for BEV Semantic Segmentation via Driving World Models arXiv 25.07 | Paper
• OpenTwinMap: An Open-Source Digital Twin Generator for Urban Autonomous Driving arXiv 25.11 | Paper
• Unlocking Efficient Vehicle Dynamics Modeling via Analytic World Models AAAI 2026 | Paper
• Deterministic World Models for Verification of Closed-loop Vision-based Systems arXiv 25.12 | Paper
• Dynamic Sparsity: Challenging Common Sparsity Assumptions for Learning World Models in Robotic Reinforcement Learning Benchmarks arXiv 25.11 | Paper
• ProTerrain: Probabilistic Physics-Informed Rough Terrain World Modeling arXiv 25.10 | Paper
• Simple, Good, Fast: Self-Supervised World Models Free of Baggage arXiv 25.06 | Paper
• Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformers CVPR 2025 | Paper | Code
• Delta-Triplane Transformers as Occupancy World Models arXiv 25.03 | Paper
• DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Driving arXiv 25.10 | Paper | Code
• DriveX: Omni Scene Modeling for Learning Generalizable World Knowledge in Autonomous Driving arXiv 25.05 | Paper
• FASTopoWM: Fast-Slow Lane Segment Topology Reasoning with Latent World Models arXiv 25.07 | Paper
• From Forecasting to Planning: Policy World Model for Collaborative State-Action Prediction NeurIPS 2025 | Paper | Code
• FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving arXiv 25.05 | Paper | Code
• I²-World: Intra-Inter Tokenization for Efficient Dynamic 4D Scene Forecasting arXiv 25.07 | Paper
• Latent Chain-of-Thought World Modeling for End-to-End Autonomous Driving CVPR 2026 | Paper
• MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft arXiv 25.04 | Paper | Code
• OccProphet: Pushing Efficiency Frontier of Camera-Only 4D Occupancy Forecasting ICLR 2025 | Paper | Code
• OccTENS: 3D Occupancy World Model via Temporal Next-Scale Prediction arXiv 25.09 | Paper
• OmniNWM: Omniscient Driving Navigation World Models arXiv 25.10 | Paper | Code
• SparseWorld-TC: Trajectory-Conditioned Sparse Occupancy World Model CVPR 2026 | Paper | Code
• STAGE: A Stream-Centric Generative World Model for Long-Horizon Driving-Scene Simulation arXiv 25.06 | Paper
• Think Before You Drive: World Model-Inspired Multimodal Grounding for Autonomous Vehicles arXiv 25.12 | Paper
• VaViM and VaVAM: Autonomous Driving through Video Generative Modeling arXiv 25.02 | Paper | Code
• VGGT-World: Transforming VGGT into an Autoregressive Geometry World Model arXiv 26.03 | Paper
• World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Model ICCV 2025 | Paper
• GenieDrive: Towards Physics-Aware Driving World Model with 4D Occupancy Guided Video Generation arXiv 25.12 | Paper | Code
• GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Control arXiv 25.05 | Paper
• LongDWM: Cross-Granularity Distillation for Building a Long-Term Driving World Model arXiv 25.06 | Paper | Code
• MoVieDrive: Urban Scene Synthesis with Multi-Modal Multi-View Video Diffusion Transformer CVPR 2026 | Paper
• Other Vehicle Trajectories Are Also Needed: A Driving World Model Unifies Ego-Other Vehicle Trajectories in Video Latent Space arXiv 25.03 | Paper
• ProphetDWM: A Driving World Model for Rolling Out Future Actions and Videos arXiv 25.05 | Paper
• Realistic and Controllable 3D Gaussian-Guided Object Editing for Driving Video Generation arXiv 25.08 | Paper
• ReSim: Reliable World Simulation for Autonomous Driving arXiv 25.06 | Paper | Code
• Seeing Clearly, Forgetting Deeply: Revisiting Fine-Tuned Video Generators for Driving Simulation arXiv 25.08 | Paper
• SimWorld: An Open-ended Realistic Simulator for Autonomous Agents in Physical and Social Worlds arXiv 25.12 | Paper | Code
• TeraSim-World: Worldwide Safety-Critical Data Synthesis for End-to-End Autonomous Driving arXiv 25.09 | Paper
• U4D: Uncertainty-Aware 4D World Modeling from LiDAR Sequences CVPR 2026 | Paper | Code
• Semi-SMD: Semi-Supervised Metric Depth Estimation via Surrounding Cameras for Autonomous Driving arXiv 25.03 | Paper
• Adapting Vision-Language Models for Evaluating World Models NeurIPS LAW 2025 | Paper | Code
• Adapting a World Model for Trajectory Following in a 3D Game arXiv 25.04 | Paper
• Better World Models Can Lead to Better Post-Training Performance arXiv 25.12 | Paper
• Context and Diversity Matter: The Emergence of In-Context Learning in World Models arXiv 25.09 | Paper
• GAWM: Global-Aware World Model for Multi-Agent Reinforcement Learning arXiv 25.01 | Paper
• Higher Embedding Dimension Creates a Stronger World Model for a Simple Sorting Task arXiv 25.10 | Paper
• Learning Primitive Embodied World Models: Towards Scalable Robotic Learning arXiv 25.08 | Paper
• Learning To Explore With Predictive World Model Via Self-Supervised Learning arXiv 25.02 | Paper
• Long-Context Autoregressive Video Modeling with Next-Frame Prediction arXiv 25.03 | Paper
• Missing Target-Relevant Information Prediction with World Model for Accurate Zero-Shot Composed Image Retrieval CVPR 2025 | Paper
• NavForesee: A Unified Vision-Language World Model for Hierarchical Planning and Dual-Horizon Navigation Prediction arXiv 25.12 | Paper
• Occupancy World Model for Robots arXiv 25.05 | Paper
• One Model for All: Large Language Models are Domain-Agnostic Recommendation Systems arXiv 23.10 | Paper
• ParticleFormer: A 3D Point Cloud World Model for Multi-Object, Multi-Material Robotic Manipulation arXiv 25.06 | Paper
• Prismatic World Model: Learning Compositional Dynamics for Planning in Hybrid Systems arXiv 25.12 | Paper
• Reimagination with Test-time Observation Interventions: Distractor-Robust World Model Predictions for Visual Model Predictive Control arXiv 25.06 | Paper
• RoboScape-R: Unified Reward-Observation World Models for Generalizable Robotics Training via RL arXiv 25.12 | Paper
• Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Robotics arXiv 25.01 | Paper | Code
• RynnVLA-002: A Unified Vision-Language-Action and World Model arXiv 25.11 | Paper | Code
• Sparse Imagination for Efficient Visual World Model Planning ICLR 2026 | Paper | Code
• The brain-AI convergence: Predictive and generative world models for general-purpose computation arXiv 25.12 | Paper
• TraceGen: World Modeling in 3D Trace Space Enables Learning from Cross-Embodiment Videos arXiv 25.11 | Paper | Code
• TransDreamerV3: Implanting Transformer In DreamerV3 arXiv 25.06 | Paper
• World Models Should Prioritize the Unification of Physical and Social Dynamics arXiv 25.10 | Paper
• GigaBrain-0: A World Model-Powered Vision-Language-Action Model arXiv 25.10 | Paper | Code
• ViPRA: Video Prediction for Robot Actions ICLR 2026 | Paper | Code

---

2024

• Visual Point Cloud Forecasting enables Scalable Autonomous Driving CVPR 2024 | Paper | Code
• Genie: Generative interactive environments arXiv 24.02 | Paper | Code
• Sora: Creating video from text OpenAI Blog 2024 | Paper
• Video generation models as world simulators OpenAI Blog 2024 | Paper
• Diffusion for world modeling: Visual details matter in atari NeurIPS 2024 | Paper | Code
• Vista: A generalizable driving world model with high fidelity and versatile controllability NeurIPS 2024 | Paper | Code
• Learning interactive real-world simulators arXiv 23.10 | Paper
• Streetscapes: Large-scale consistent street view generation using autoregressive video diffusion arXiv 24.07 | Paper
• Worlddreamer: Towards general world models for video generation via predicting masked tokens arXiv 24.01 | Paper | Code
• Drivedreamer-2: Llm-enhanced world models for diverse driving video generation arXiv 24.03 | Paper | Code
• Copilot4d: Learning unsupervised world models for autonomous driving via discrete diffusion ICLR 2024 | Paper | Code
• Generalized predictive model for autonomous driving CVPR 2024 | Paper
• Scenediffuser: Efficient and controllable driving simulation initialization and rollout NeurIPS 2024 | Paper
• Simgen: Simulator-conditioned driving scene generation arXiv 24.06 | Paper
• GenAD: Generalized Predictive Model for Autonomous Driving CVPR 2024 | Paper
• Subjectdrive: Scaling generative data in autonomous driving via subject control arXiv 24.03 | Paper
• Unleashing generalization of end-to-end autonomous driving with controllable long video generation arXiv 24.06 | Paper
• Urbanworld: An urban world model for 3d city generation arXiv 24.07 | Paper | Code
• Drivedreamer4d: World models are effective data machines for 4d driving scene representation arXiv 24.10 | Paper
• 3d-vla: A 3d vision-language-action generative world model CVPR 2024 | Paper | Code
• Avid: Adapting video diffusion models to world models arXiv 24.10 | Paper
• Physdreamer: Physics-based interaction with 3d objects via video generation ECCV 2024 | Paper
• Advancing humanoid locomotion: Mastering challenging terrains with denoising world model learning RSS 2024 | Paper
• Robodreamer: Learning compositional world models for robot imagination arXiv 24.04 | Paper
• Vidman: Exploiting implicit dynamics from video diffusion model for effective robot manipulation NeurIPS 2024 | Paper
• Consisti2v: Enhancing visual consistency for image-to-video generation arXiv 24.02 | Paper | Code
• Worldgpt: Empowering llm as multimodal world model arXiv 24.04 | Paper | Code
• Owl-1: Omni world model for consistent long video generation arXiv 24.12 | Paper
• Pandora: Towards general world model with natural language actions and video states arXiv 24.06 | Paper | Code
• Gem: A generalizable ego-vision multimodal world model for fine-grained ego-motion, object dynamics, and scene composition control arXiv 24.12 | Paper
• Diffusion world models ICLR 2024 | Paper
• Navsim: Data-driven non-reactive autonomous vehicle simulation and benchmarking NeurIPS 2024 | Paper | Code
• Bridgedata v2: A dataset for robot learning at scale arXiv 23.08 | Paper | Code | Dataset
• Towards generalist robot policies: What matters in building vision-language-action models arXiv 24.05 | Paper
• Womd-lidar: Raw sensor dataset benchmark for motion forecasting ICRA 2024 | Paper | Dataset
• Video as the new language for real-world decision making arXiv 24.02 | Paper
• Infinitydrive: Breaking time limits in driving world models arXiv 24.12 | Paper
• Diffusion policy: Visuomotor policy learning via action diffusion RSS 2023 | Paper | Code
• Genie 2: A large-scale foundation world model DeepMind Blog 2024 | Paper
• π0: A vision-language-action flow model for general robot control RSS 2025 | Paper
• Bevworld: A multimodal world simulator for autonomous driving via scene-level bev latents arXiv 24.07 | Paper
• Diffusion models are real-time game engines ICLR 2025 | Paper
• LidarDM: Generative LiDAR Simulation in a Generated World arXiv 24.04 | Paper | Code
• DrivePhysica: Physical Informed Driving World Model arXiv 24.12 | Paper
• DriveGenVLM: Real-world Video Generation for Vision Language Model based Autonomous Driving arXiv 24.08 | Paper
• DOME: Taming Diffusion Model into High-Fidelity Controllable Occupancy World Model arXiv 24.10 | Paper | Code
• Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving AAAI 2025 | Paper | Code
• CarFormer: Self-Driving with Learned Object-Centric Representations ECCV 2024 | Paper | Code
• SafeDreamer: Safe Reinforcement Learning with World Models ICLR 2024 | Paper | Code
• Planning with Adaptive World Models for Autonomous Driving ICRA 2025 | Paper
• Compete and Compose: Learning Independent Mechanisms for Modular World Models arXiv 24.04 | Paper
• DINO-Foresight: Looking into the Future with DINO NeurIPS 2025 | Paper | Code
• MegaSaM: Accurate, Fast, and Robust Structure and Motion from Casual Dynamic Videos arXiv 24.12 | Paper | Code
• InfiniCube: Unbounded and Controllable Dynamic 3D Driving Scene Generation with World-Guided Video Models ICCV 2025 | Paper | Code
• OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Driving arXiv 24.05 | Paper | Code
• GenEx: Generating an Explorable World arXiv 24.12 | Paper | Code
• How Far is Video Generation from World Model: A Physical Law Perspective ICML 2025 | Paper
• Automated construction of cognitive maps with visual predictive coding arXiv 24.08 | Paper
• D^2-World: An Efficient World Model through Decoupled Dynamic Flow CVPR 2024 | Paper
• DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning arXiv 24.11 | Paper | Code
• Evaluating the World Model Implicit in a Generative Model arXiv 24.06 | Paper
• Grounded Answers for Multi-agent Decision-making Problem through Generative World Model arXiv 24.10 | Paper
• Improving Token-Based World Models with Parallel Observation Prediction arXiv 24.02 | Paper | Code
• One-shot World Models Using a Transformer Trained on a Synthetic Prior arXiv 24.09 | Paper | Code
• PIVOT-R: Primitive-Driven Waypoint-Aware World Model for Robotic Manipulation NeurIPS 2024 | Paper | Code
• Revisiting Feature Prediction for Learning Visual Representations from Video arXiv 24.04 | Paper
• Scaling Laws for Pre-training Agents and World Models arXiv 24.11 | Paper
• Transformers and Slot Encoding for Sample Efficient Physical World Modelling arXiv 24.05 | Paper
• Transformers Use Causal World Models in Maze-Solving Tasks arXiv 24.12 | Paper
• A Causal World Model Underlying Next Token Prediction: Exploring GPT in a Controlled Environment arXiv 24.12 | Paper
• From Imitation to Exploration: End-to-end Autonomous Driving based on World Model arXiv 24.10 | Paper
• Navigation-Guided Sparse Scene Representation for End-to-End Autonomous Driving ICLR 2025 | Paper
• Mitigating Covariate Shift in Imitation Learning for Autonomous Vehicles Using Latent Space Generative World Models ICRA 2025 | Paper
• Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learning arXiv 24.02 | Paper
• Statespacediffuser: Bringing long context to diffusion world models arXiv 25.05 | Paper
• Understanding world or predicting future? a comprehensive survey of world models arXiv 24.11 | Paper
• The role of world models in shaping autonomous driving: A comprehensive survey arXiv 25.02 | Paper
• A survey: Learning embodied intelligence from physical simulators and world models arXiv 25.07 | Paper

---

2023

• Instructpix2pix: Learning to follow image editing instructions arXiv 23.01 | Paper | Code
• Scalable diffusion models with transformers arXiv 23.05 | Paper | Code
• Compositional visual generation with composable diffusion models ECCV 2022 | Paper | Code
• Training-free structured diffusion guidance for compositional text-to-image synthesis ICLR 2023 | Paper | Code
• Gaia-1: A generative world model for autonomous driving arXiv 23.09 | Paper
• Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving arXiv 23.11 | Paper | Code
• Adriver-i: A general world model for autonomous driving arXiv 23.11 | Paper
• Motiondiffuser: Controllable multi-agent motion prediction using diffusion CVPR 2023 | Paper
• Drivedreamer: Towards real-world-driven world models for autonomous driving arXiv 23.09 | Paper | Code
• Magicdrive: Street view generation with diverse 3d geometry control arXiv 23.10 | Paper | Code
• Panacea: Panoramic and controllable video generation for autonomous driving arXiv 23.11 | Paper | Code
• Wovogen: World volume-aware diffusion for controllable multi-camera driving scene generation ECCV 2024 | Paper | Code
• Nuwa-xl: Diffusion over diffusion for extremely long video generation arXiv 23.05 | Paper | Dataset
• Peekaboo: Interactive video generation via masked-diffusion arXiv 23.12 | Paper
• Libero: Benchmarking knowledge transfer for lifelong robot learning arXiv 23.06 | Paper | Code
• Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks arXiv 23.06 | Paper | Code
• Argoverse 2: Next generation datasets for self-driving perception and forecasting arXiv 23.01 | Paper | Code | Dataset
• Stable video diffusion: Scaling latent video diffusion models to large datasets arXiv 23.11 | Paper
• OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving arXiv 23.11 | Paper | Code
• MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representations arXiv 23.11 | Paper
• Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture IEEE 2023 | Paper | Code
• Transformers are Sample-Efficient World Models ICLR 2023 | Paper | Code
• TrafficBots: Towards World Models for Autonomous Driving Simulation and Motion Prediction ICRA 2023 | Paper | Code
• Categorical Traffic Transformer: Interpretable and Diverse Behavior Prediction with Tokenized Latent arXiv 23.11 | Paper

2022

• Survey of hallucination in natural language generation arXiv 22.02 | Paper
• A path towards autonomous machine intelligence version 0.9.2, 2022-06-27 arXiv 22.06 | Paper | Code
• High-resolution image synthesis with latent diffusion models CVPR 2022 | Paper | Code
• Video diffusion models arXiv 22.04 | Paper | Code
• Progressive distillation for fast sampling of diffusion models ICLR 2022 | Paper | Code
• Classifier-free diffusion guidance NeurIPS 2021 | Paper | Code
• Nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles arXiv 22.02 | Paper | Code | Dataset
• Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks IEEE RA-L 2022 | Paper | Code | Dataset
• Interactive language: Talking to robots in real time arXiv 22.10 | Paper | Dataset
• Ego4d: Around the world in 3,000 hours of egocentric video CVPR 2022 | Paper | Code | Dataset
• Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation arXiv 22.05 | Paper
• Rt-1: Robotics transformer for real-world control at scale arXiv 22.05 | Paper | Code | Dataset
• Tartandrive: A large-scale dataset for learning off-road dynamics models arXiv 22.05 | Paper | Code | Dataset
• Diffusion causal models for counterfactual estimation CLeaR 2022 | Paper
• Planning with diffusion for flexible behavior synthesis ICML 2022 | Paper | Code
• CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Driving ECCV 2022 | Paper
• Diffdreamer: Towards consistent unsupervised single-view scene extrapolation with conditional diffusion models arXiv 22.11 | Paper | Code
• Hierarchical Model-Based Imitation Learning for Planning in Autonomous Driving IROS 2022 | Paper
• Socially Compliant Navigation Dataset (SCAND): A Large-Scale Dataset of Demonstrations for Social Navigation arXiv 22.05 | Paper | Dataset
• Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields arXiv 22.05 | Paper | Dataset

2021

• Finetuned language models are zero-shot learners arXiv 21.09 | Paper
• Diffusion models beat gans on image synthesis arXiv 21.05 | Paper | Code
• Tackling the generative learning trilemma with denoising diffusion gans ICLR 2022 | Paper | Code
• Bridge data: Boosting generalization of robotic skills with cross-domain datasets arXiv 21.09 | Paper | Dataset
• Frozen in time: A joint video and image encoder for end-to-end retrieval ICCV 2021 | Paper | Code
• D4rl: Datasets for deep data-driven reinforcement learning arXiv 20.04 | Paper | Code | Dataset
• Rapid Exploration for Open-World Navigation with Latent Goal Models CoRL 2021 | Paper
• CLIP arXiv 20.02 | Paper | Code

2020

• Language models are few-shot learners arXiv 20.05 | Paper | Code
• Experience grounds language EMNLP 2020 | Paper
• Learning to simulate dynamic environments with gamegan CVPR 2020 | Paper | Code
• Denoising diffusion probabilistic models arXiv 20.06 | Paper | Code
• nuscenes: A multimodal dataset for autonomous driving CVPR 2020 | Paper | Dataset
• Scalability in perception for autonomous driving: Waymo open dataset CVPR 2020 | Paper | Code | Dataset
• Leveraging Procedural Generation to Benchmark Reinforcement Learning arXiv 20.07 | Paper | Dataset

2019

• HowTo100M ICCV 2019 | Paper | Dataset
• LaSOT arXiv 18.09 | Paper | Dataset
• Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines SIGGRAPH 2019 | Paper | Dataset

2018

• World models arXiv 18.03 | Paper | Code
• Recurrent world models facilitate policy evolution arXiv 18.09 | Paper
• GS arXiv 18.03 | Paper | Dataset

2016

• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild arXiv 12.12 | Paper | Dataset
• MSR-VTT: A Large Video Description Dataset for Bridging Video and Language IEEE 2016 | Paper | Dataset
• A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation IEEE 2016 | Paper | Dataset

🤝 Contributing

We warmly welcome contributions. You can help by:

• adding missing papers,
• correcting metadata,
• supplementing code / project links,
• improving taxonomy or tags,
• refining dataset and benchmark summaries.

Please see CONTRIBUTING.md for details.

---

📚 Paper Navigation

For category-wise browsing, see:

• papers/README.md
• papers/autonomous-driving.md
• papers/embodied-intelligence.md
• papers/general-worlds.md
• papers/datasets.md

---

📌 Citation

@article{wang2026diffusionworldmodels,
  title   = {Diffusion-based World Models: A Survey},
  author  = {Wang, Gang and Liu, Zhen and Zhou, Mingliang and Zhang, Yugui and Yang, Guang and Yang, Lei and Song, Ziying},
  journal = {arXiv preprint arXiv:2504.10724},
  year    = {2026}
}