Inquiry about the detailed config of reproduction

[fzhiy]

Hi, authors,

Thank you very much for your excellent work. I believe this will undoubtedly advance further research in practical reinforcement learning.

I am attempting to reproduce the results of QeRL-supported GRPO on Qwen 2.5-3B using an RTX 5090. I trained the base model quantized via NVFP4 using the hyperparameters from Table 4 in the paper (max_completion_length=4096, max_seq_length=5500, num_iterations=4, lora_rank=32). When evaluated on GSM8K, the accuracy was only 66.5%, which is significantly lower than the 83.7% reported in Table 1(a).

What could be causing this discrepancy? Have I overlooked any critical parameter configurations?
If possible, could you share the detailed configuration file used to reproduce the main results in Table 1 and Table 2?

Thank you in advance.

Best regards,
Feng

[oncleJules]

I have the same concern.

The scripts' filenames started with 'dapo', does it mean the hyperparameters in these scripts are only for dapo, not for grpo?

since the experiments of grpo are significantly shorter, i believe most practitioners will try grpo experiments first - would greatly appreciate a release of the grpo hyperparameters

[Aaronhuang-778]

Hi, @fzhiy

What is your evaluation script? Did you use the GSM8K script provided in QeRL?

[Aaronhuang-778]

@oncleJules

Thanks for your remind. We will release both optimal setting of GRPO and DAPO scripts.

[Aaronhuang-778]

Hi, authors,

Thank you very much for your excellent work. I believe this will undoubtedly advance further research in practical reinforcement learning.

I am attempting to reproduce the results of QeRL-supported GRPO on Qwen 2.5-3B using an RTX 5090. I trained the base model quantized via NVFP4 using the hyperparameters from Table 4 in the paper (max_completion_length=4096, max_seq_length=5500, num_iterations=4, lora_rank=32). When evaluated on GSM8K, the accuracy was only 66.5%, which is significantly lower than the 83.7% reported in Table 1(a).

What could be causing this discrepancy? Have I overlooked any critical parameter configurations? If possible, could you share the detailed configuration file used to reproduce the main results in Table 1 and Table 2?

Thank you in advance.

Best regards, Feng

Hi, please just set larger batch size for the training.

The scripts provided in training path are for just for limited resource training. In our experiments, we used larger batch-size as we can in H100, which provided stable results. Hope this will be helpful :-) .

[fzhiy]

Hi, @fzhiy

What is your evaluation script? Did you use the GSM8K script provided in QeRL?

Before making any modifications to eval/gsm8k.py, I used the evaluation script provided by QeRL: python -m eval.gsm8k --model “ckpt/qwen2.5-3B-single-gpus-quantized_1e-5_0.2_gsm8k_b2_g1_r16_True/checkpoint-7473” --load nvfp4 and python -m eval.gsm8k --model “ckpt/qwen2.5-3B-single-gpus-quantized_1e-5_0.2_gsm8k_b2_g1_r16_True/checkpoint-7473” --load nvfp4 --data gsm8k yields accuracies of 0.15 and 0.21911.

It appears that using the flexible-extract filter in lm_eval achieves an accuracy of 0.6331. Therefore, I referenced this approach to rewrite the correctness_reward_func implementation as follows:

def flexible_extract_answer(text: str | None) -> str | None:
    if not text:
        return None
    matches = FLEXIBLE_EXTRACT_REGEX.findall(text)
    if not matches:
        return None
    # Each match is a tuple of the two capturing groups; pick the last non-empty entry.
    raw = matches[-1]
    candidate = next((m for m in raw if m), None)
    if candidate is None:
        return None
    candidate = candidate.replace(",", "").replace("$", "").strip()
    # Drop trailing punctuation like a standalone period.
    candidate = candidate.rstrip(".")
    return candidate or None

def correctness_reward_func(prompts, response, answer, **kwargs) -> list[float]:
    q = prompts
    extracted_text = extract_xml_answer(response)
    pred = flexible_extract_answer(extracted_text)
    gold = flexible_extract_answer(answer)
    print(
        "-" * 20,
        f"Question:\n{q}",
        f"\nAnswer:\n{answer}",
        f"\nResponse:\n{response}",
        f"\nExtracted:\n{extracted_text}",
        f"\nParsed Prediction:\n{pred}",
        f"\nParsed Gold:\n{gold}",
    )
    is_correct = 1.0 if (pred is not None and gold is not None and pred == gold) else 0.0
    return [is_correct]

After rerunning the evaluation script with the --data gsm8k flag, the accuracy reached 0.665.

[fzhiy]

Hi, authors,
Thank you very much for your excellent work. I believe this will undoubtedly advance further research in practical reinforcement learning.
I am attempting to reproduce the results of QeRL-supported GRPO on Qwen 2.5-3B using an RTX 5090. I trained the base model quantized via NVFP4 using the hyperparameters from Table 4 in the paper (max_completion_length=4096, max_seq_length=5500, num_iterations=4, lora_rank=32). When evaluated on GSM8K, the accuracy was only 66.5%, which is significantly lower than the 83.7% reported in Table 1(a).
What could be causing this discrepancy? Have I overlooked any critical parameter configurations? If possible, could you share the detailed configuration file used to reproduce the main results in Table 1 and Table 2?
Thank you in advance.
Best regards, Feng

Hi, please just set larger batch size for the training.

The scripts provided in training path are for just for limited resource training. In our experiments, we used larger batch-size as we can in H100, which provided stable results. Hope this will be helpful :-) .

Thank you very much for your attention and reply. I will try that.
Looking forward to the release of both optimal setting of GRPO and DAPO scripts.

[oncleJules]

Hi, authors,
Thank you very much for your excellent work. I believe this will undoubtedly advance further research in practical reinforcement learning.
I am attempting to reproduce the results of QeRL-supported GRPO on Qwen 2.5-3B using an RTX 5090. I trained the base model quantized via NVFP4 using the hyperparameters from Table 4 in the paper (max_completion_length=4096, max_seq_length=5500, num_iterations=4, lora_rank=32). When evaluated on GSM8K, the accuracy was only 66.5%, which is significantly lower than the 83.7% reported in Table 1(a).
What could be causing this discrepancy? Have I overlooked any critical parameter configurations? If possible, could you share the detailed configuration file used to reproduce the main results in Table 1 and Table 2?
Thank you in advance.
Best regards, Feng

Hi, please just set larger batch size for the training.

The scripts provided in training path are for just for limited resource training. In our experiments, we used larger batch-size as we can in H100, which provided stable results. Hope this will be helpful :-) .

Thank you @Aaronhuang-778. I will follow your suggestion to use the largest batch-size that can be fitted into H100 (so far i found it to be 32 for qwen2.5-3B).

Meanwhile, i noticed that in your training script, "num-generations" must be equal to "per-device-train-batch-size".

I am new to this domain. Could you help me understand the reason behind this setting?

[yuxizhe]

I ran into this problem as well. In my replication, the Qwen 2.5 7B NVFP4 baseline score was 0.71, and the score only improved to 0.73 after complete QeRL training. I used the project's evaluation script.

Another question, which prompt was used for the GSM8K training and eval in the paper? I see there are three prompts here, with two commented out. I've noticed that the structure and placement order of the system prompt significantly affects the score.

https://github.com/NVlabs/QeRL/blob/main/utils/data.py#L16-L49

# uncomment middle messages for 1-shot prompting
def get_gsm8k_questions(split = "train") -> Dataset:
    data = load_dataset('openai/gsm8k', 'main')[split] # type: ignore
    # system_prompt = """
    #     Solve the following math problem step by step. The reasoning process and direct answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, i.e., <think> reasoning process here </think> <answer> answer here </answer>: 
    #     <think>
    #     ...
    #     </think>
    #     <answer>
    #     ...
    #     </answer>
    # """
    # system_prompt = """
    #     Solve the following math problem step by step. The reasoning process is enclosed within <think> </think> tags, and the final answer is provided after "####", i.e.,: 
    #     <think>
    #     ...
    #     </think>
    #     #### answer
    # """
    instruction = """
        Let\'s think step by step first within <think> </think> tags, and output the final answer after "####" tag, i.e.,: 
        #     <think>
        #     ...
        #     </think>
        #     #### number
    """
    data = data.map(lambda x: { # type: ignore
        'prompt': [
            # {'role': 'system', 'content': system_prompt},
            {'role': 'user', 'content': x['question']  + " " + instruction}
        ],
        'answer': extract_hash_answer(x['answer'])
    }) # type: ignore
    return data # type: ignore

[Aaronhuang-778]

Hi, authors,
Thank you very much for your excellent work. I believe this will undoubtedly advance further research in practical reinforcement learning.
I am attempting to reproduce the results of QeRL-supported GRPO on Qwen 2.5-3B using an RTX 5090. I trained the base model quantized via NVFP4 using the hyperparameters from Table 4 in the paper (max_completion_length=4096, max_seq_length=5500, num_iterations=4, lora_rank=32). When evaluated on GSM8K, the accuracy was only 66.5%, which is significantly lower than the 83.7% reported in Table 1(a).
What could be causing this discrepancy? Have I overlooked any critical parameter configurations? If possible, could you share the detailed configuration file used to reproduce the main results in Table 1 and Table 2?
Thank you in advance.
Best regards, Feng

Hi, please just set larger batch size for the training.
The scripts provided in training path are for just for limited resource training. In our experiments, we used larger batch-size as we can in H100, which provided stable results. Hope this will be helpful :-) .

Thank you @Aaronhuang-778. I will follow your suggestion to use the largest batch-size that can be fitted into H100 (so far i found it to be 32 for qwen2.5-3B).

Meanwhile, i noticed that in your training script, "num-generations" must be equal to "per-device-train-batch-size".

I am new to this domain. Could you help me understand the reason behind this setting?

Hi, it is the batching method in trl.GRPO, you can refer the code in grpo_trainer.py

[Aaronhuang-778]

I ran into this problem as well. In my replication, the Qwen 2.5 7B NVFP4 baseline score was 0.71, and the score only improved to 0.73 after complete QeRL training. I used the project's evaluation script.

Another question, which prompt was used for the GSM8K training and eval in the paper? I see there are three prompts here, with two commented out. I've noticed that the structure and placement order of the system prompt significantly affects the score.

https://github.com/NVlabs/QeRL/blob/main/utils/data.py#L16-L49

# uncomment middle messages for 1-shot prompting
def get_gsm8k_questions(split = "train") -> Dataset:
    data = load_dataset('openai/gsm8k', 'main')[split] # type: ignore
    # system_prompt = """
    #     Solve the following math problem step by step. The reasoning process and direct answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, i.e., <think> reasoning process here </think> <answer> answer here </answer>: 
    #     <think>
    #     ...
    #     </think>
    #     <answer>
    #     ...
    #     </answer>
    # """
    # system_prompt = """
    #     Solve the following math problem step by step. The reasoning process is enclosed within <think> </think> tags, and the final answer is provided after "####", i.e.,: 
    #     <think>
    #     ...
    #     </think>
    #     #### answer
    # """
    instruction = """
        Let\'s think step by step first within <think> </think> tags, and output the final answer after "####" tag, i.e.,: 
        #     <think>
        #     ...
        #     </think>
        #     #### number
    """
    data = data.map(lambda x: { # type: ignore
        'prompt': [
            # {'role': 'system', 'content': system_prompt},
            {'role': 'user', 'content': x['question']  + " " + instruction}
        ],
        'answer': extract_hash_answer(x['answer'])
    }) # type: ignore
    return data # type: ignore

Hi, thank you for your attention of our work. We use the first system prompt for GSM8K:

The script we provided is just used for resource-limited single GPU, it is for training reward/ efficiency comparison with BF16 + LoRA, you can also train BF16 + LoRA for comparing. In our manuscript's table, we used H100-80G and set larger batch-size (batch_size 64*8). It is because that larger batch-size brings stable and batter training results. You can change perdevice_train_batch_size and gradient_accumulation_steps on single GPU to get stable training results, and we will also release the multi-GPU training version and scripts later (currently struggling for rebuttal :-) )

[oncleJules]

I ran into this problem as well. In my replication, the Qwen 2.5 7B NVFP4 baseline score was 0.71, and the score only improved to 0.73 after complete QeRL training. I used the project's evaluation script.
Another question, which prompt was used for the GSM8K training and eval in the paper? I see there are three prompts here, with two commented out. I've noticed that the structure and placement order of the system prompt significantly affects the score.
https://github.com/NVlabs/QeRL/blob/main/utils/data.py#L16-L49

# uncomment middle messages for 1-shot prompting
def get_gsm8k_questions(split = "train") -> Dataset:
    data = load_dataset('openai/gsm8k', 'main')[split] # type: ignore
    # system_prompt = """
    #     Solve the following math problem step by step. The reasoning process and direct answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, i.e., <think> reasoning process here </think> <answer> answer here </answer>: 
    #     <think>
    #     ...
    #     </think>
    #     <answer>
    #     ...
    #     </answer>
    # """
    # system_prompt = """
    #     Solve the following math problem step by step. The reasoning process is enclosed within <think> </think> tags, and the final answer is provided after "####", i.e.,: 
    #     <think>
    #     ...
    #     </think>
    #     #### answer
    # """
    instruction = """
        Let\'s think step by step first within <think> </think> tags, and output the final answer after "####" tag, i.e.,: 
        #     <think>
        #     ...
        #     </think>
        #     #### number
    """
    data = data.map(lambda x: { # type: ignore
        'prompt': [
            # {'role': 'system', 'content': system_prompt},
            {'role': 'user', 'content': x['question']  + " " + instruction}
        ],
        'answer': extract_hash_answer(x['answer'])
    }) # type: ignore
    return data # type: ignore

Hi, thank you for your attention of our work. We use the first system prompt for GSM8K:

The script we provided is just used for resource-limited single GPU, it is for training reward/ efficiency comparison with BF16 + LoRA, you can also train BF16 + LoRA for comparing. In our manuscript's table, we used H100-80G and set larger batch-size (batch_size 64*8). It is because that larger batch-size brings stable and batter training results. You can change `perdevice_train_batch_size` and `gradient_accumulation_steps` on single GPU to get stable training results, and we will also release the multi-GPU training version and scripts later (currently struggling for rebuttal :-) )

Hi @Aaronhuang-778 , hopefully the rebuttal has been completed successfully. Would the configs be released soon?