POInfra focuses on policy optimization for generative models. We study how diffusion and flow models can serve as policies, how these policies can improve through reinforcement learning, and how additional computation at inference time can improve their decisions.
Our research began with D2PPO, which combines dispersive regularization for diffusion policy pretraining with PPO fine-tuning. The research extends to efficient one-step policies, coordinated multi-agent generation, online fine-tuning, and inference-time guidance and planning. The common goal is to turn expressive generative models into effective decision-making policies.
PO stands for Policy Optimization, and Infra reflects our goal of building reusable infrastructure around this research. This repository brings together the methods and resources. Available implementations currently run through their individual project repositories.
Browse the POInfra collection on Hugging Face, or open a project resource directly below.
| Project | Models | Datasets |
|---|---|---|
| OGPO | Hugging Face | Hugging Face |
| CoFlow | Hugging Face | Hugging Face |
| MA-FPPO | Hugging Face | — |
| G2MAF | Hugging Face | — |
| MA-WAM | Hugging Face | — |
A two-minute introduction to POInfra, with real robot demonstrations and multi-agent environments.
POInfra_2min_English_Music.mp4
-
[2026/09] 🔥 POInfra is launched, bringing together our research on policy optimization for generative models, beginning with D2PPO.
-
[2026/09] 🔥 MA-FPPO is now available on arXiv. The project page includes policy rollouts and a presentation of multi-agent flow pretraining and online fine-tuning. Project · Paper · Code · Models.
-
[2026/07] 🔥 G2MAF explores test-time gradient guidance for multi-agent flow policies. Project · Paper · Code.
-
[2026/07] 🔥 MA-WAM explores multi-agent world-action models for test-time planning. Project · Paper · Code.
-
[2026/05] 🔥 CoFlow introduces coordinated few-step flow for offline multi-agent decision-making. Project · Paper · Code.
-
[2026/09] 🔥 The updated OGPO preprint (v2) is available. The work is now accepted at ACM MM 2026. Project · Paper · Code.
-
[2025/10] 🔥 DM1 introduces MeanFlow with dispersive regularization for one-step robotic manipulation. Project · Paper · Code.
-
[2025/08] 🔥 D2PPO introduces diffusion policy optimization with dispersive loss. The work is now accepted at AAAI 2026. Project · Paper · Code.
POInfra aims to bring embodied intelligence into practical use, enabling robots to perform useful work in factories and homes and create tangible value. We focus on a central question: How can generative models move from learning existing behavior to becoming decision-making policies that continually improve, adapt to new tasks, and act efficiently?
For robot control and multi-agent cooperation, we organize our research around five connected stages: data, pretraining, policy optimization, inference, and evaluation. Together, these stages address how to learn from experience, improve through feedback, and make effective decisions with limited interaction and computation.
Offline demonstrations and interaction trajectories provide the behavioral experience for pretraining. D2PPO introduces dispersive regularization during diffusion policy pretraining, and DM1 studies this regularization for MeanFlow policies in robotic manipulation. These projects investigate representation diversity and efficient action generation as foundations for subsequent policy learning. CoFlow extends learning from offline data to coordinated few-step trajectory generation for multiple agents.
Policy optimization is at the core of POInfra. D2PPO fine-tunes diffusion policies with PPO, OGPO develops one-step generative policy optimization for real-time robot control, and MA-FPPO studies flow policy pretraining and online fine-tuning for multi-agent decision-making. Across these model families, we study how policies can use prior experience while exploring better behavior through task feedback.
DM1 and OGPO study one-step policies to reduce the computation needed to generate actions. G2MAF uses Q-value gradient guidance to steer multi-agent flow policies toward actions with higher estimated value. MA-WAM studies multi-agent world-action models for test-time planning. These directions address action generation speed, action value, and anticipation of future outcomes.
Evaluation covers task success rate, return, win rate, inference time, and performance on physical robots, as appropriate for each task. We consider both task performance and the data, interaction, and computation required to achieve it. These results inform the design of pretraining, policy optimization, and inference methods, connecting methodological progress to practical deployment.
| Method | Research focus | Paper | Project | Code |
|---|---|---|---|---|
| D2PPO | Diffusion policy pretraining with dispersive loss and PPO fine-tuning | AAAI 2026 / arXiv | Website | D2PPO-release |
| DM1 | MeanFlow with dispersive regularization for one-step robotic manipulation | arXiv | Website | DM1-release |
| OGPO | One-step generative policy optimization for real-time robot control | ACM MM 2026 / arXiv | Website | OGPO |
| Method | Research focus | Paper | Project | Code |
|---|---|---|---|---|
| CoFlow | Coordinated few-step flow for offline multi-agent decision-making | arXiv | Website | CoFlow-release |
| MA-FPPO | Multi-agent flow-pretrained policy optimization | arXiv · Models | Website | MA-FPPO-release |
| G2MAF | Test-time gradient guidance for multi-agent flow policies | arXiv | Website | G2MAF-release |
| MA-WAM | Multi-agent world-action modeling for test-time planning | arXiv | Website | MA-WAM-release |
All linked repositories are public. G2MAF and MA-WAM currently contain a project README, with no implementation files yet. Implementation links above refer to separate project releases, rather than modules integrated into a single package.
Choose the implementation that matches the question you want to explore:
| Goal | Starting point |
|---|---|
| Pretrain a diffusion policy and fine-tune it with PPO | D2PPO setup and examples |
| Study one-step generative policies for manipulation | DM1 setup and evaluation |
| Study online optimization of one-step policies | OGPO training and evaluation |
| Train coordinated flow models on offline multi-agent data | CoFlow setup and experiments |
For example, obtain the D2PPO implementation with:
git clone https://github.com/Guowei-Zou/d2ppo-release.git
cd d2ppo-releaseThen follow the repository's installation instructions, environment setup, and task configurations. Each project documents its own dependencies, datasets, checkpoints, and evaluation procedure.
We welcome discussions about generative policy optimization, reproducibility, and components that could be shared across projects. Open an issue for cross-project ideas. For a method-specific question or bug, use the corresponding implementation repository.
Please cite the papers corresponding to the methods and resources you use. BibTeX entries are available in each project's repository or website.
Guowei Zou · Sun Yat-sen University
Following the path from learning to deployment, we are extending this research in eight directions:
- Expand training data. Explore world models as a source of generated data to address data scarcity in embodied intelligence and provide richer scenarios and behavioral experience for policy learning.
- Build training environments closer to reality. Learn world models from real interactions and use them as simulation environments for reinforcement learning, exploring ways to reduce the gap between simulation and reality.
- Improve performance beyond dataset coverage. Optimize pretrained policies through online reinforcement learning so robots can adjust their behavior using environment feedback and explore effective solutions beyond existing data.
- Reduce the cost of adapting to new tasks. Study in-context learning so robots can use task examples and interaction experience to adapt, reducing the need for parameter fine-tuning during deployment.
- Meet real-time control requirements. Study one-step generative models and their policy optimization to reduce action generation latency while improving task performance.
- Enable decisions that anticipate future outcomes. Study latent world models and world-model-based test-time planning to improve current decisions by predicting future states and action outcomes.
- Support failure recovery and long-horizon tasks. Explore Agent RL so robots can identify failures from execution feedback, revise plans, attempt recovery, and coordinate multiple steps to complete tasks.
- Advance cooperation among embodied robots. Develop large-scale datasets and simulation environments for multi-agent embodied intelligence, and study policy optimization for physical interaction and cooperation so multiple robots can work together in industrial and everyday settings.