Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

CS336 Assignment 5 Alignment Experiments

This repository contains my implementation and experiments for Stanford CS336 Spring 2026 Assignment 5: Alignment. The original assignment focuses on reasoning RL with GRPO; the optional supplement covers SFT, safety evaluation, and DPO.

The main experiment in this repo trains allenai/OLMo-2-0425-1B on GSM8K with GRPO and compares GSM8K accuracy before and after training.

Original handouts:

What Is Implemented

Core components:

  • GRPO prompt/response tokenization and response masks
  • Response log-probability and token entropy computation
  • Rollout reward computation
  • Group-normalized reward/advantage computation
  • On-policy GRPO loss
  • Off-policy GRPO/GSPO-style importance weighting
  • Microbatch loss aggregation
  • One-step GRPO training update
  • Packed SFT dataset loading
  • MMLU and GSM8K answer parsing
  • Per-instance DPO loss

Experiment scripts:

  • related_code/eval_gsm8k.py: evaluate a causal LM on GSM8K.
  • related_code/train_grpo_experiment.py: run GRPO training with before/after GSM8K evaluation.
  • related_code/grpo_pipeline.py: single-file GRPO pipeline used for smoke tests and model-interface checks.
  • related_code/summarize_grpo_results.py: summarize multiple experiment directories into JSON/Markdown tables.

Repository Layout

cs336_alignment/          Official assignment utilities and prompts
data/                     Assignment datasets
related_code/             My implementations and experiment scripts
tests/                    Provided unit tests and adapters
outputs/                  Small saved experiment summaries and logs

Large model checkpoints are not included. The original and trained OLMo checkpoints were kept outside the repository on AutoDL.

Setup

The assignment uses uv for dependency management:

uv sync --no-install-package flash-attn
uv sync

For a simpler local CPU-only check, the project virtual environment can also be used directly if dependencies are already installed:

./.venv/bin/python -m pytest tests/test_grpo.py tests/test_data.py tests/test_metrics.py tests/test_dpo.py -q

Tests

The implemented adapters pass the local component tests:

python -m pytest tests/test_grpo.py tests/test_data.py tests/test_metrics.py tests/test_dpo.py -q

Test coverage includes GRPO components, SFT data loading, answer parsers, and DPO loss.

GSM8K GRPO Experiments

Experiments were run on AutoDL with an NVIDIA RTX 5090.

Common setup:

  • Model: allenai/OLMo-2-0425-1B
  • Dataset: GSM8K
  • Prompt: r1_zero_three_shot_gsm8k.prompt
  • Rollout generation: sampling with temperature 0.7 and top-p 0.95
  • No oracle rollouts during reported GRPO runs
  • Reward: final numeric answer match against GSM8K gold answer

Results:

Experiment Before Acc. After Acc. Delta Steps Avg. Reward
olmo_grpo_real_run_small 0.2600 0.3200 0.0600 100 0.3650
olmo_grpo_real_run_small_seed1 0.2800 0.3600 0.0800 100 0.3750
olmo_grpo_real_run_small_seed2 0.2700 0.3400 0.0700 100 0.4800
olmo_grpo_500steps_eval500_seed0 0.3040 0.3780 0.0740 500 0.4020

Average before accuracy: 0.2785
Average after accuracy: 0.3495
Average accuracy delta: 0.0710

The 100-step runs evaluate on 100 GSM8K test examples. The 500-step run evaluates on 500 GSM8K test examples.

Reproducing A GRPO Run

Download allenai/OLMo-2-0425-1B from Hugging Face and place it on the machine, for example:

/root/autodl-tmp/hf_models/OLMo-2-0425-1B

Then run:

python related_code/train_grpo_experiment.py \
  --model-source pretrained \
  --model-path /root/autodl-tmp/hf_models/OLMo-2-0425-1B \
  --train-data-path /root/autodl-tmp/data/gsm8k/train.jsonl \
  --eval-data-path /root/autodl-tmp/data/gsm8k/test.jsonl \
  --prompt-path /root/autodl-tmp/prompts/r1_zero_three_shot_gsm8k.prompt \
  --device cuda \
  --experiment-dir /root/autodl-tmp/outputs/olmo_grpo_real_run_small_seed2 \
  --num-train-steps 100 \
  --train-batch-prompts 1 \
  --eval-limit 100 \
  --group-size 2 \
  --max-new-tokens 128 \
  --learning-rate 5e-7 \
  --do-sample \
  --temperature 0.7 \
  --top-p 0.95 \
  --no-mix-oracle-rollout \
  --seed 2

For the larger run:

python related_code/train_grpo_experiment.py \
  --model-source pretrained \
  --model-path /root/autodl-tmp/hf_models/OLMo-2-0425-1B \
  --train-data-path /root/autodl-tmp/data/gsm8k/train.jsonl \
  --eval-data-path /root/autodl-tmp/data/gsm8k/test.jsonl \
  --prompt-path /root/autodl-tmp/prompts/r1_zero_three_shot_gsm8k.prompt \
  --device cuda \
  --experiment-dir /root/autodl-tmp/outputs/olmo_grpo_500steps_eval500_seed0 \
  --num-train-steps 500 \
  --train-batch-prompts 1 \
  --eval-limit 500 \
  --group-size 2 \
  --max-new-tokens 128 \
  --learning-rate 5e-7 \
  --do-sample \
  --temperature 0.7 \
  --top-p 0.95 \
  --no-mix-oracle-rollout \
  --seed 0

Each run writes:

config.json
train_metrics.jsonl
eval_before_summary.json
eval_after_summary.json
eval_before_results.jsonl
eval_after_results.jsonl
comparison.json
checkpoint_final/

Only small summaries/logs are kept in this repository. checkpoint_final/ is not intended to be committed.

Summarizing Results

To summarize multiple experiment directories:

python related_code/summarize_grpo_results.py \
  outputs/olmo_grpo_real_run_small \
  outputs/olmo_grpo_real_run_small_seed1 \
  outputs/olmo_grpo_real_run_small_seed2 \
  outputs/olmo_grpo_500steps_eval500_seed0 \
  --json-output outputs/grpo_summary_with_500.json \
  --markdown-output outputs/grpo_summary_with_500.md

Notes

  • The reported experiments are small-scale and intended to validate the GRPO pipeline and direction of improvement, not to claim a saturated GSM8K result.
  • GRPO loss is not directly comparable to supervised cross-entropy loss; the main task metric here is GSM8K accuracy before and after training.
  • The original model directory is not modified by training. Trained models are saved under each experiment's checkpoint_final/ directory.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages