Question about reproduction gap on NAVSIM v1 released checkpoints (Stage 2 PDMS 86.29 vs 87.4, Stage 3 PDMS 89.7 vs 91.1)

[induge]

Hi, thanks for releasing the code and checkpoints.

I am trying to reproduce the NAVSIM v1 results using the released checkpoints, but I observe a noticeable gap compared with the numbers reported in the paper.

Specifically:

• Stage 2: Diffusion Planner Imitation Learning

  • Paper: 87.4 PDMS
  • My reproduction with the released checkpoint: 86.29 PDMS

• Stage 3: Diffusion Planner Reinforcement Learning Training

  • Paper: 91.1 PDMS
  • My reproduction with the released checkpoint: 89.7 PDMS

I would like to ask whether there are any additional details that are important for reproducing the reported numbers. For example:

1. Are the released checkpoints exactly the same as the ones used to report the paper results?
2. Are there any missing evaluation settings, data filtering steps, or preprocessing details that are not included in the current release?

If helpful, I can also provide more details about my environment, commit version, and evaluation command.

Any clarification would be greatly appreciated. Thanks!

[SII-Whaleice]

Thank you for your interest in our work.
You are absolutely correct in your observation. The code I exported from the company and adapted locally reproduced a result of 86.7 PDMS on NAVSIM v1 stage 2, while another colleague obtained 86.8 PDMS(H200GPUS). I suspect this discrepancy may be due to referencing a different commit or branch, or possibly due to minor manual debugging changes I made (such as adjustments to certain index values). It is also quite likely that my colleague further improved the results based on the company’s internal codebase.(H20GPUS)
That said, the overall logic and implementation of the code should remain largely consistent.
Regarding the SGDrive results, I have also retrieved the corresponding screenshots as supporting evidence.

As mentioned in the email, I will follow up to investigate the issue. For now, I will keep this issue open.

[chenxiahan]

Hi, thanks for releasing the code and checkpoints.

I am trying to reproduce the NAVSIM v1 results using the released checkpoints, but I observe a noticeable gap compared with the numbers reported in the paper.

Specifically:

• Stage 2: Diffusion Planner Imitation Learning

  • Paper: 87.4 PDMS
  • My reproduction with the released checkpoint: 86.29 PDMS

• Stage 3: Diffusion Planner Reinforcement Learning Training

  • Paper: 91.1 PDMS
  • My reproduction with the released checkpoint: 89.7 PDMS

I would like to ask whether there are any additional details that are important for reproducing the reported numbers. For example:

1. Are the released checkpoints exactly the same as the ones used to report the paper results?
2. Are there any missing evaluation settings, data filtering steps, or preprocessing details that are not included in the current release?

If helpful, I can also provide more details about my environment, commit version, and evaluation command.

Any clarification would be greatly appreciated. Thanks!

My reproduction(A100GPUS) with the released checkpoint is the same as yours