This repository contains my implementation and experiments for Stanford CS336 Spring 2026 Assignment 5: Alignment. The original assignment focuses on reasoning RL with GRPO; the optional supplement covers SFT, safety evaluation, and DPO.
The main experiment in this repo trains allenai/OLMo-2-0425-1B on GSM8K with
GRPO and compares GSM8K accuracy before and after training.
Original handouts:
Core components:
- GRPO prompt/response tokenization and response masks
- Response log-probability and token entropy computation
- Rollout reward computation
- Group-normalized reward/advantage computation
- On-policy GRPO loss
- Off-policy GRPO/GSPO-style importance weighting
- Microbatch loss aggregation
- One-step GRPO training update
- Packed SFT dataset loading
- MMLU and GSM8K answer parsing
- Per-instance DPO loss
Experiment scripts:
related_code/eval_gsm8k.py: evaluate a causal LM on GSM8K.related_code/train_grpo_experiment.py: run GRPO training with before/after GSM8K evaluation.related_code/grpo_pipeline.py: single-file GRPO pipeline used for smoke tests and model-interface checks.related_code/summarize_grpo_results.py: summarize multiple experiment directories into JSON/Markdown tables.
cs336_alignment/ Official assignment utilities and prompts
data/ Assignment datasets
related_code/ My implementations and experiment scripts
tests/ Provided unit tests and adapters
outputs/ Small saved experiment summaries and logs
Large model checkpoints are not included. The original and trained OLMo checkpoints were kept outside the repository on AutoDL.
The assignment uses uv for dependency management:
uv sync --no-install-package flash-attn
uv syncFor a simpler local CPU-only check, the project virtual environment can also be used directly if dependencies are already installed:
./.venv/bin/python -m pytest tests/test_grpo.py tests/test_data.py tests/test_metrics.py tests/test_dpo.py -qThe implemented adapters pass the local component tests:
python -m pytest tests/test_grpo.py tests/test_data.py tests/test_metrics.py tests/test_dpo.py -qTest coverage includes GRPO components, SFT data loading, answer parsers, and DPO loss.
Experiments were run on AutoDL with an NVIDIA RTX 5090.
Common setup:
- Model:
allenai/OLMo-2-0425-1B - Dataset: GSM8K
- Prompt:
r1_zero_three_shot_gsm8k.prompt - Rollout generation: sampling with temperature 0.7 and top-p 0.95
- No oracle rollouts during reported GRPO runs
- Reward: final numeric answer match against GSM8K gold answer
Results:
| Experiment | Before Acc. | After Acc. | Delta | Steps | Avg. Reward |
|---|---|---|---|---|---|
| olmo_grpo_real_run_small | 0.2600 | 0.3200 | 0.0600 | 100 | 0.3650 |
| olmo_grpo_real_run_small_seed1 | 0.2800 | 0.3600 | 0.0800 | 100 | 0.3750 |
| olmo_grpo_real_run_small_seed2 | 0.2700 | 0.3400 | 0.0700 | 100 | 0.4800 |
| olmo_grpo_500steps_eval500_seed0 | 0.3040 | 0.3780 | 0.0740 | 500 | 0.4020 |
Average before accuracy: 0.2785
Average after accuracy: 0.3495
Average accuracy delta: 0.0710
The 100-step runs evaluate on 100 GSM8K test examples. The 500-step run evaluates on 500 GSM8K test examples.
Download allenai/OLMo-2-0425-1B from Hugging Face and place it on the machine,
for example:
/root/autodl-tmp/hf_models/OLMo-2-0425-1B
Then run:
python related_code/train_grpo_experiment.py \
--model-source pretrained \
--model-path /root/autodl-tmp/hf_models/OLMo-2-0425-1B \
--train-data-path /root/autodl-tmp/data/gsm8k/train.jsonl \
--eval-data-path /root/autodl-tmp/data/gsm8k/test.jsonl \
--prompt-path /root/autodl-tmp/prompts/r1_zero_three_shot_gsm8k.prompt \
--device cuda \
--experiment-dir /root/autodl-tmp/outputs/olmo_grpo_real_run_small_seed2 \
--num-train-steps 100 \
--train-batch-prompts 1 \
--eval-limit 100 \
--group-size 2 \
--max-new-tokens 128 \
--learning-rate 5e-7 \
--do-sample \
--temperature 0.7 \
--top-p 0.95 \
--no-mix-oracle-rollout \
--seed 2For the larger run:
python related_code/train_grpo_experiment.py \
--model-source pretrained \
--model-path /root/autodl-tmp/hf_models/OLMo-2-0425-1B \
--train-data-path /root/autodl-tmp/data/gsm8k/train.jsonl \
--eval-data-path /root/autodl-tmp/data/gsm8k/test.jsonl \
--prompt-path /root/autodl-tmp/prompts/r1_zero_three_shot_gsm8k.prompt \
--device cuda \
--experiment-dir /root/autodl-tmp/outputs/olmo_grpo_500steps_eval500_seed0 \
--num-train-steps 500 \
--train-batch-prompts 1 \
--eval-limit 500 \
--group-size 2 \
--max-new-tokens 128 \
--learning-rate 5e-7 \
--do-sample \
--temperature 0.7 \
--top-p 0.95 \
--no-mix-oracle-rollout \
--seed 0Each run writes:
config.json
train_metrics.jsonl
eval_before_summary.json
eval_after_summary.json
eval_before_results.jsonl
eval_after_results.jsonl
comparison.json
checkpoint_final/
Only small summaries/logs are kept in this repository. checkpoint_final/ is not
intended to be committed.
To summarize multiple experiment directories:
python related_code/summarize_grpo_results.py \
outputs/olmo_grpo_real_run_small \
outputs/olmo_grpo_real_run_small_seed1 \
outputs/olmo_grpo_real_run_small_seed2 \
outputs/olmo_grpo_500steps_eval500_seed0 \
--json-output outputs/grpo_summary_with_500.json \
--markdown-output outputs/grpo_summary_with_500.md- The reported experiments are small-scale and intended to validate the GRPO pipeline and direction of improvement, not to claim a saturated GSM8K result.
- GRPO loss is not directly comparable to supervised cross-entropy loss; the main task metric here is GSM8K accuracy before and after training.
- The original model directory is not modified by training. Trained models are
saved under each experiment's
checkpoint_final/directory.