Skip to content

About

How much wall-clock training time data-selection methods actually save for GPT-2 small, with the cost of selection charged against the savings. The data axis of the NanoGPT speedrun, sized for free Kaggle and Colab GPUs.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

Repository files navigation

Data-Efficiency Speedrun

Measuring how much wall-clock training time data-selection methods actually buy, with the selection cost honestly charged against the savings.

The NanoGPT speedrun has driven GPT-2 (124M) to a fixed 3.28 validation loss from 45 minutes down to 1.353 minutes on the same 8×H100 node. Essentially all of that came from architecture, optimizer, and kernel work — the data is held constant across all 82 records. This repo measures the other axis, at a scale that fits free-tier Kaggle/Colab.

                 ┌──────────────┐
  FineWeb ──────►│  build_pool  │──► data/_val   (frozen, SHA256-pinned, hardlinked)
   (stream)      │  tokenize 1× │──► data/_pool  (tokens + offsets, built once)
                 └──────────────┘
                        │
                 ┌──────▼───────┐   edu classifier · reference-model perplexity
                 │  score_pool  │   MinHash+LSH · SemDeDup      ──► scores.npz
                 │ ledger-timed │                               ──► selection cost
                 └──────┬───────┘
                        │
                 ┌──────▼───────┐   17 policies, equal token budget,
                 │  build_arm   │   repetition factor recorded
                 └──────┬───────┘
                        │
                 ┌──────▼───────┐   resume-safe · wall-clock survives preemption
                 │  run_sweep   │   interpolated time-to-target
                 └──────┬───────┘
                        │
                 ┌──────▼───────┐   fig0 noise floor · fig1 TTT · fig2 cost-adjusted
                 │   figures    │   fig3 scale · RESULTS.md
                 └──────────────┘

Try it without a GPU or a download

pip install -r requirements.txt
make demo          # ~35 min on CPU: full pipeline over a synthetic corpus
make real          # the real-corpus run: fetch PyPI, tokenize, score, sweep

make demo runs every stage end to end — pool, reference model, all four scoring signals, 17 arms, 62 training runs across two model scales, and all four figures. It is how the repo proves itself before touching real data.

The three things that make this a measurement

1. The frozen validation set. Filtering training data shifts its distribution; evaluating on data drawn from the filtered pool makes any filter look good. So val is carved from the unfiltered pool by document-index holdout (1 doc in 200 — contamination-proof under streaming, unlike taking a prefix), written once, and hardlinked into every arm directory. Arms cannot drift apart because they are the same inodes. A SHA256 digest is pinned in configs/val_pin.txt and re-verified from disk before every run; a mismatch aborts. Flipping one token in 60,000 is caught.

2. Wall-clock accounting that survives preemption. Free-tier sessions get killed. A resumed run reporting time.time() - start has measured the wrong thing by an unbounded margin. WallClock accumulates train-only seconds and is checkpointed, so a resumed run continues the count. Eval and checkpoint time are excluded from train_seconds (eval cadence is a knob you control); the trainer also logs session_seconds for reference. Checkpoints write tmp+rename, so a kill mid-write cannot corrupt the good file.

3. The selection-cost ledger. Scoring a pool with a quality classifier, or training a reference model for perplexity pruning, costs real compute. Almost nobody reports it. speedrun/ledger.py records it where it is paid, and the analysis reports both TTT and TTT + selection — charged in full and amortised across the sweep, because those are different questions and only showing the smaller one is how selection cost gets hidden.

Plus: time-to-target detection with linear interpolation between eval points, so TTT is not quantised to the eval interval.

Experimental arms

Policies are +-separated chains applied left to right (speedrun/policy.py):

arm policy what it tests
A0_baseline baseline noise floor (5 seeds)
A1_dedup_minhash dedup:minhash how much "quality" gain is really dedup?
A1_dedup_sem dedup:sem SemDeDup: cluster, drop near-identical members
A2_edu_top{10,25,50,75} edu_top:k quality filtering, and where the k curve turns over
A3_ppl_{low,medium,high} ppl_band:band:0.5 perplexity pruning; band is dataset-dependent
A4_curr_* curriculum:sig:dir ordering only — nothing discarded, so gain is pure order
A5_len_min128 len_min:128 cheap length policy
A6_* composed do the gains stack, or overlap?

A4 and A6 are where the original results live. A1–A3 are replications — necessary for credibility, not novel. A4 discards nothing, so its selection cost is near zero and any speedup is unambiguous. A6 asks a question the literature mostly dodges: quality filtering and perplexity pruning plausibly select overlapping documents, so their gains probably do not add.

Arms are compared at identical token budgets. If a policy selects too little data it is repeated, the repetition factor is recorded, and anything above 2× is flagged CONFOUNDED — past the point where pruning gains are known to survive.

Real run

# 1. download + tokenize once; commit the val pin
python -m scripts.build_pool --source fineweb --pool-tokens 800000000
git add configs/val_pin.txt && git commit -m "pin validation set"

# 2. reference model for perplexity pruning (a real selection cost)
python -m scripts.train_reference --tokens 20000000

# 3. every signal over the pool, each ledger-timed
python -m scripts.score_pool --signals all

# 4. materialise all arms at an equal token budget
python -m scripts.build_arm --all --train-tokens 200000000

# 5. noise floor first — 5 baseline seeds, no target yet. Read L* off the curves.
python -m scripts.run_sweep --config configs/pilot.yaml --arms A0_baseline
python -m analysis.noise_floor

# 6. the sweep, with the target you just chose
python -m scripts.run_sweep --config configs/pilot.yaml --seeds 0 1 2 \
    --target-loss 3.55 --time-budget-h 8

# 7. scale ladder — each rung gets its own run dir so seeds do not collide
python -m scripts.run_sweep --config configs/main.yaml --run-dir runs_main \
    --arms A0_baseline A6_edu25_curr --target-loss 3.55

# 8. results
python -m analysis.figures --all --run-dir runs runs_main
python -m analysis.report --run-dir runs runs_main

Re-issuing any run_sweep command after a session dies resumes every incomplete run from its checkpoint. That is the intended recovery path, not a fallback.

Layout

speedrun/model.py    GPT: RoPE, RMSNorm, QK-norm, ReLU², untied embeddings
speedrun/muon.py     Muon (Newton-Schulz), fp32 iteration for Turing
speedrun/data.py     shard I/O, resumable loader, val integrity guard
speedrun/pool.py     tokenize-once document pool
speedrun/scoring.py  edu classifier · perplexity · MinHash-LSH · SemDeDup
speedrun/policy.py   selection policies, composition, repetition accounting
speedrun/train.py    trainer: WallClock, atomic ckpt, interpolated TTT
speedrun/ledger.py   selection-cost accounting
scripts/             build_pool → train_reference → score_pool → build_arm → run_sweep
analysis/            results loader, style, fig0-fig3, RESULTS.md generator
tests/               32 tests over the invariants the results depend on

make help lists every pipeline target.

Real-data results

Run on 40,000 real PyPI package descriptions (207M characters, uniformly sampled from the 876k-package index — the genuine long tail, not a curated top-N), tokenized with a corpus-trained 8k byte-level BPE into a 20M-token pool of 11,854 documents plus a 700k-token held-out split. 37 training runs across 8 arms and 2 model scales, CPU-only, ~2 hours of compute.

Every selection method lost to random sampling.

arm seeds val loss TTT (s) sel. cost TTT+cost speedup cost-adj
A0_baseline 5 5.1449 ± 0.0197 147 0 147 1.00× 1.00×
A1_dedup_minhash 3 5.3902 ± 0.0012 199 19 218 0.74× 0.67×
A4_curr_edu_asc 3 5.4329 ± 0.0097 202 17 219 0.73× 0.67×
A2_edu_top25 3 5.4621 ± 0.0143 215 17 232 0.68× 0.63×
A2_edu_top50 3 5.4686 ± 0.0043 217 17 234 0.68× 0.63×
A3_ppl_medium 3 5.2729 ± 0.0007 163 905 1068 0.90× 0.14×
A3_ppl_high 3 5.4709 ± 0.0091 219 905 1124 0.67× 0.13×
A6_edu50_ppl50 3 5.4970 ± 0.0280 216 922 1138 0.68× 0.13×

Noise floor: baseline ±1σ = 0.0197 val loss over 5 seeds, so an effect must clear ~0.039 to be real. Every arm cleared it — in the wrong direction.

Why, and why that is the expected answer

The validation set is the unfiltered distribution. A quality filter does not make the model better at that distribution; it makes the model better at the slice it selected. Re-evaluating every trained model on three slices of the same held-out documents shows exactly that:

arm all held-out high-education half low-education half
A2_edu_top50 +0.0652 −0.0290 +0.1229
A2_edu_top25 +0.2112 +0.0285 +0.3216
A3_ppl_medium −0.0222 −0.0409 −0.0096

edu_top50 beats the baseline on educational documents and loses on the full distribution, with the damage concentrated on the low-education half. Score it on the slice it selected for — the natural mistake — and it reports a speedup that is really distribution matching. The frozen, unfiltered validation set is the thing that prevented that conclusion.

This is consistent with the literature rather than against it: FineWeb-Edu is validated on downstream benchmarks (MMLU, ARC, OpenBookQA), not on held-out web loss, and aggressive quality filtering is expected to raise distribution-matched loss. At 0.9M parameters no benchmark returns signal, so this study can only measure the metric on which filtering is expected to regress. It measures it carefully, and reports the regression.

The other three findings

The signals are not independent. Before any training: educational score and reference-model perplexity correlate at ρ = +0.687; edu_top:0.5 and ppl_band:high:0.5 select 75.6% of the same documents against a 50% chance baseline; MinHash and LSA-SemDeDup agree on 72% of MinHash's duplicates. Gains that cannot be separated at selection time cannot stack at training time, and A6_edu50_ppl50 duly lands on top of its components.

Selection cost dominated at this scale. The reference model for perplexity pruning cost 585s to train and 320s to score with — 905s against a 147s baseline training run. Perplexity pruning would need a training run 6× longer before its scoring pass merely broke even, and since it is slower than the baseline even before cost, it never pays for itself here at all.

The penalty grows with scale. Every arm's speedup got worse going from 0.9M to 2.4M non-embedding parameters (A6: 0.68× → 0.58×). A larger model exploits the training distribution harder, so mismatching it costs more.

Verified

check result
Val reconstruction held-out docs rebuilt from corpus + seed match the pinned shard byte for byte
Frozen-val guard one flipped token in 60,000 → run refused to start
Preemption + resume SIGKILL at 13.5s → resumed at step 1028; train_seconds 76.8 vs session_seconds 70.6
MinHash dedup 3.9% found against a 4% planted rate (synthetic); 5.9% on real PyPI data
Edu classifier 160 LLM annotations → 5-fold CV Spearman 0.783, RMSE 0.655 vs 1.076 baseline
Noise floor 5 baseline seeds, ±1σ = 0.0197 val loss
Test suite 32 passed

Honest limitations

  • Corpus is PyPI descriptions, not FineWeb. This sandbox cannot reach huggingface.co. PyPI is real, messy, heterogeneous text with genuine quality variance and genuine near-duplicates, but it is a narrower domain than web crawl, and conclusions may not transfer.
  • Tokenizer is a corpus-trained 8k BPE, not GPT-2's. The encoding host is also unreachable, and a small vocabulary is what makes CPU training affordable.
  • The educational classifier is TF-IDF + Ridge on 160 LLM annotations, not a BERT head on 500k. Same recipe, ~3000× fewer labels. ρ = 0.783 is useful but not the real thing.
  • SemDeDup uses LSA embeddings, not a neural sentence encoder — lexical at heart, so it will miss paraphrase. Reported as LSA-SemDeDup.
  • Scale is tiny: 0.9M and 2.4M non-embedding parameters, 600k and 900k tokens. Three orders of magnitude below where data-quality effects are believed to matter.
  • The in-training eval used the first ~12k tokens of the 700k-token val shard (eval_batches 6 × batch 8 × seq 256). Identical tokens for every arm, so the comparisons are valid and add no noise — but it is a thin subsample, and the full-val numbers from eval_slices.py reorder the top two arms. Raise eval_batches to cover the whole shard on any run that matters.

Hardware notes (these will bite you)

  • The T4 is Turing (SM 7.5): no bf16. Configs use fp16 AMP. Copying a modern bf16 config will fail or silently degrade.
  • FlashAttention-2 requires SM 8.0+ and will not build for a T4. The model uses PyTorch SDPA, which selects the memory-efficient backend.
  • Muon's Newton-Schulz iteration runs in fp32 here (ns_dtype); upstream uses bf16. On Ampere+ set ns_dtype=torch.bfloat16 to recover the speed.
  • torch.compile is off by default — flaky on Turing. Benchmark first.
  • Pool sizing: to end up with T training tokens at retention k you need ~T/k pool tokens. 200M at k=0.25 → 800M pool tokens, 1.6GB as uint16.

References

About

How much wall-clock training time data-selection methods actually save for GPT-2 small, with the cost of selection charged against the savings. The data axis of the NanoGPT speedrun, sized for free Kaggle and Colab GPUs.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages