Discussion about sim2sim pipeline in mujoco

[MRonaldo-gif]

Hi, nice to see your excellent work and I am trying to develop the sim2sim of "carrybox" and "linedown". I use the pretrained checkpoints in the /resources dir and see the robot working successfully in isaacgym, but the situation is quite different in mujoco sim.I have set same obs, controller and so on, the testing video is as follow. I want to ask for some help or reasons of bad mujoco testing, and if you need or you think my try is helpful, I am glad to join in your open-sourced plan on sim2sim pipeline.

output.mp4

[why618188]

Thank you for your interest in our work! I think the first thing to check is whether the task observations are fully consistent — for example, the object’s rotation, position, and other related information — as well as whether the observation and action scales are the same. If you don’t mind sharing your code with us, I’d be happy to help take a closer look and work together on developing the sim2sim pipeline.

[windigal]

Hi @why618188 @MRonaldo-gif, I have also implemented the sim2sim pipeline on carrybox task and encountered similar performance issues. The robot appears to be unable to stand and walk stably and fall backwards. I have opened a Draft PR here: #9.
I'd appreciate it if we could check the implementation details together in that PR. Thanks!

[SakuBorder]

Hi @why618188 @MRonaldo-gif, I have also implemented the sim2sim pipeline on carrybox task and encountered similar performance issues. The robot appears to be unable to stand and walk stably and fall backwards. I have opened a Draft PR here: #9. I'd appreciate it if we could check the implementation details together in that PR. Thanks!

Did you manage to solve this issue? I tested other tasks on my side as well, and they also can’t run properly in sim2sim and the robot slips and falls very easily.

[windigal]

Did you manage to solve this issue? I tested other tasks on my side as well, and they also can’t run properly in sim2sim and the robot slips and falls very easily.

No, I haven't solved it yet. Given that so many people are reporting the exact same instability issues across different tasks, I think that the issue lies with the pre-trained model itself rather than the sim2sim pipeline.
It seems the current checkpoints might lack the robustness required for MuJoCo. I don't plan to retrain the policy from scratch for now, so I'll wait for the authors' insights on this matter.

[gaolin0621]

你好，很高兴看到你的出色工作，我正在尝试开发“carrybox”和“linedown”的sim2sim。我使用了/resources目录里的预训练检查点，看到机器人在Isaacgym中能成功工作，但Mujoco模拟器的情况却完全不同。我设置了相同的OBS、控制器等，测试视频如下。我想请教一些关于mujoco测试不好的帮助或原因，如果你需要或觉得我的尝试有帮助，我很高兴加入你关于sim2sim流水线的开源计划。

output.mp4

Hello, I’m currently working on sim2sim code implementation as well, and I’m writing to politely ask if you would be willing to share your sim2sim code with me.
The core issue I’m stuck on is that the robot exhibits persistent uncontrolled jitter and twitching when running with my code. I would greatly appreciate the opportunity to refer to your implementation as a reference to troubleshoot this problem.
My email address is: 2957028747@qq.com
Thank you so much for your time and consideration.