Measuring how much wall-clock training time data-selection methods actually buy, with the selection cost honestly charged against the savings.
The NanoGPT speedrun has driven GPT-2 (124M) to a fixed 3.28 validation loss from 45 minutes down to 1.353 minutes on the same 8×H100 node. Essentially all of that came from architecture, optimizer, and kernel work — the data is held constant across all 82 records. This repo measures the other axis, at a scale that fits free-tier Kaggle/Colab.
┌──────────────┐
FineWeb ──────►│ build_pool │──► data/_val (frozen, SHA256-pinned, hardlinked)
(stream) │ tokenize 1× │──► data/_pool (tokens + offsets, built once)
└──────────────┘
│
┌──────▼───────┐ edu classifier · reference-model perplexity
│ score_pool │ MinHash+LSH · SemDeDup ──► scores.npz
│ ledger-timed │ ──► selection cost
└──────┬───────┘
│
┌──────▼───────┐ 17 policies, equal token budget,
│ build_arm │ repetition factor recorded
└──────┬───────┘
│
┌──────▼───────┐ resume-safe · wall-clock survives preemption
│ run_sweep │ interpolated time-to-target
└──────┬───────┘
│
┌──────▼───────┐ fig0 noise floor · fig1 TTT · fig2 cost-adjusted
│ figures │ fig3 scale · RESULTS.md
└──────────────┘
pip install -r requirements.txt
make demo # ~35 min on CPU: full pipeline over a synthetic corpus
make real # the real-corpus run: fetch PyPI, tokenize, score, sweepmake demo runs every stage end to end — pool, reference model, all four
scoring signals, 17 arms, 62 training runs across two model scales, and all four
figures. It is how the repo proves itself before touching real data.
1. The frozen validation set. Filtering training data shifts its
distribution; evaluating on data drawn from the filtered pool makes any filter
look good. So val is carved from the unfiltered pool by document-index
holdout (1 doc in 200 — contamination-proof under streaming, unlike taking a
prefix), written once, and hardlinked into every arm directory. Arms cannot
drift apart because they are the same inodes. A SHA256 digest is pinned in
configs/val_pin.txt and re-verified from disk before every run; a mismatch
aborts. Flipping one token in 60,000 is caught.
2. Wall-clock accounting that survives preemption. Free-tier sessions get
killed. A resumed run reporting time.time() - start has measured the wrong
thing by an unbounded margin. WallClock accumulates train-only seconds and is
checkpointed, so a resumed run continues the count. Eval and checkpoint time are
excluded from train_seconds (eval cadence is a knob you control); the trainer
also logs session_seconds for reference. Checkpoints write tmp+rename, so a
kill mid-write cannot corrupt the good file.
3. The selection-cost ledger. Scoring a pool with a quality classifier, or
training a reference model for perplexity pruning, costs real compute. Almost
nobody reports it. speedrun/ledger.py records it where it is paid, and the
analysis reports both TTT and TTT + selection — charged in full and
amortised across the sweep, because those are different questions and only
showing the smaller one is how selection cost gets hidden.
Plus: time-to-target detection with linear interpolation between eval points, so TTT is not quantised to the eval interval.
Policies are +-separated chains applied left to right (speedrun/policy.py):
| arm | policy | what it tests |
|---|---|---|
A0_baseline |
baseline |
noise floor (5 seeds) |
A1_dedup_minhash |
dedup:minhash |
how much "quality" gain is really dedup? |
A1_dedup_sem |
dedup:sem |
SemDeDup: cluster, drop near-identical members |
A2_edu_top{10,25,50,75} |
edu_top:k |
quality filtering, and where the k curve turns over |
A3_ppl_{low,medium,high} |
ppl_band:band:0.5 |
perplexity pruning; band is dataset-dependent |
A4_curr_* |
curriculum:sig:dir |
ordering only — nothing discarded, so gain is pure order |
A5_len_min128 |
len_min:128 |
cheap length policy |
A6_* |
composed | do the gains stack, or overlap? |
A4 and A6 are where the original results live. A1–A3 are replications —
necessary for credibility, not novel. A4 discards nothing, so its selection cost
is near zero and any speedup is unambiguous. A6 asks a question the literature
mostly dodges: quality filtering and perplexity pruning plausibly select
overlapping documents, so their gains probably do not add.
Arms are compared at identical token budgets. If a policy selects too little
data it is repeated, the repetition factor is recorded, and anything above 2× is
flagged CONFOUNDED — past the point where pruning gains are known to survive.
# 1. download + tokenize once; commit the val pin
python -m scripts.build_pool --source fineweb --pool-tokens 800000000
git add configs/val_pin.txt && git commit -m "pin validation set"
# 2. reference model for perplexity pruning (a real selection cost)
python -m scripts.train_reference --tokens 20000000
# 3. every signal over the pool, each ledger-timed
python -m scripts.score_pool --signals all
# 4. materialise all arms at an equal token budget
python -m scripts.build_arm --all --train-tokens 200000000
# 5. noise floor first — 5 baseline seeds, no target yet. Read L* off the curves.
python -m scripts.run_sweep --config configs/pilot.yaml --arms A0_baseline
python -m analysis.noise_floor
# 6. the sweep, with the target you just chose
python -m scripts.run_sweep --config configs/pilot.yaml --seeds 0 1 2 \
--target-loss 3.55 --time-budget-h 8
# 7. scale ladder — each rung gets its own run dir so seeds do not collide
python -m scripts.run_sweep --config configs/main.yaml --run-dir runs_main \
--arms A0_baseline A6_edu25_curr --target-loss 3.55
# 8. results
python -m analysis.figures --all --run-dir runs runs_main
python -m analysis.report --run-dir runs runs_mainRe-issuing any run_sweep command after a session dies resumes every incomplete
run from its checkpoint. That is the intended recovery path, not a fallback.
speedrun/model.py GPT: RoPE, RMSNorm, QK-norm, ReLU², untied embeddings
speedrun/muon.py Muon (Newton-Schulz), fp32 iteration for Turing
speedrun/data.py shard I/O, resumable loader, val integrity guard
speedrun/pool.py tokenize-once document pool
speedrun/scoring.py edu classifier · perplexity · MinHash-LSH · SemDeDup
speedrun/policy.py selection policies, composition, repetition accounting
speedrun/train.py trainer: WallClock, atomic ckpt, interpolated TTT
speedrun/ledger.py selection-cost accounting
scripts/ build_pool → train_reference → score_pool → build_arm → run_sweep
analysis/ results loader, style, fig0-fig3, RESULTS.md generator
tests/ 32 tests over the invariants the results depend on
make help lists every pipeline target.
Run on 40,000 real PyPI package descriptions (207M characters, uniformly sampled from the 876k-package index — the genuine long tail, not a curated top-N), tokenized with a corpus-trained 8k byte-level BPE into a 20M-token pool of 11,854 documents plus a 700k-token held-out split. 37 training runs across 8 arms and 2 model scales, CPU-only, ~2 hours of compute.
Every selection method lost to random sampling.
| arm | seeds | val loss | TTT (s) | sel. cost | TTT+cost | speedup | cost-adj |
|---|---|---|---|---|---|---|---|
A0_baseline |
5 | 5.1449 ± 0.0197 | 147 | 0 | 147 | 1.00× | 1.00× |
A1_dedup_minhash |
3 | 5.3902 ± 0.0012 | 199 | 19 | 218 | 0.74× | 0.67× |
A4_curr_edu_asc |
3 | 5.4329 ± 0.0097 | 202 | 17 | 219 | 0.73× | 0.67× |
A2_edu_top25 |
3 | 5.4621 ± 0.0143 | 215 | 17 | 232 | 0.68× | 0.63× |
A2_edu_top50 |
3 | 5.4686 ± 0.0043 | 217 | 17 | 234 | 0.68× | 0.63× |
A3_ppl_medium |
3 | 5.2729 ± 0.0007 | 163 | 905 | 1068 | 0.90× | 0.14× |
A3_ppl_high |
3 | 5.4709 ± 0.0091 | 219 | 905 | 1124 | 0.67× | 0.13× |
A6_edu50_ppl50 |
3 | 5.4970 ± 0.0280 | 216 | 922 | 1138 | 0.68× | 0.13× |
Noise floor: baseline ±1σ = 0.0197 val loss over 5 seeds, so an effect must clear ~0.039 to be real. Every arm cleared it — in the wrong direction.
The validation set is the unfiltered distribution. A quality filter does not make the model better at that distribution; it makes the model better at the slice it selected. Re-evaluating every trained model on three slices of the same held-out documents shows exactly that:
| arm | all held-out | high-education half | low-education half |
|---|---|---|---|
A2_edu_top50 |
+0.0652 | −0.0290 | +0.1229 |
A2_edu_top25 |
+0.2112 | +0.0285 | +0.3216 |
A3_ppl_medium |
−0.0222 | −0.0409 | −0.0096 |
edu_top50 beats the baseline on educational documents and loses on the
full distribution, with the damage concentrated on the low-education half. Score
it on the slice it selected for — the natural mistake — and it reports a
speedup that is really distribution matching. The frozen, unfiltered validation
set is the thing that prevented that conclusion.
This is consistent with the literature rather than against it: FineWeb-Edu is validated on downstream benchmarks (MMLU, ARC, OpenBookQA), not on held-out web loss, and aggressive quality filtering is expected to raise distribution-matched loss. At 0.9M parameters no benchmark returns signal, so this study can only measure the metric on which filtering is expected to regress. It measures it carefully, and reports the regression.
The signals are not independent. Before any training: educational score and
reference-model perplexity correlate at ρ = +0.687; edu_top:0.5 and
ppl_band:high:0.5 select 75.6% of the same documents against a 50% chance
baseline; MinHash and LSA-SemDeDup agree on 72% of MinHash's duplicates.
Gains that cannot be separated at selection time cannot stack at training time,
and A6_edu50_ppl50 duly lands on top of its components.
Selection cost dominated at this scale. The reference model for perplexity pruning cost 585s to train and 320s to score with — 905s against a 147s baseline training run. Perplexity pruning would need a training run 6× longer before its scoring pass merely broke even, and since it is slower than the baseline even before cost, it never pays for itself here at all.
The penalty grows with scale. Every arm's speedup got worse going from 0.9M
to 2.4M non-embedding parameters (A6: 0.68× → 0.58×). A larger model exploits
the training distribution harder, so mismatching it costs more.
| check | result |
|---|---|
| Val reconstruction | held-out docs rebuilt from corpus + seed match the pinned shard byte for byte |
| Frozen-val guard | one flipped token in 60,000 → run refused to start |
| Preemption + resume | SIGKILL at 13.5s → resumed at step 1028; train_seconds 76.8 vs session_seconds 70.6 |
| MinHash dedup | 3.9% found against a 4% planted rate (synthetic); 5.9% on real PyPI data |
| Edu classifier | 160 LLM annotations → 5-fold CV Spearman 0.783, RMSE 0.655 vs 1.076 baseline |
| Noise floor | 5 baseline seeds, ±1σ = 0.0197 val loss |
| Test suite | 32 passed |
- Corpus is PyPI descriptions, not FineWeb. This sandbox cannot reach huggingface.co. PyPI is real, messy, heterogeneous text with genuine quality variance and genuine near-duplicates, but it is a narrower domain than web crawl, and conclusions may not transfer.
- Tokenizer is a corpus-trained 8k BPE, not GPT-2's. The encoding host is also unreachable, and a small vocabulary is what makes CPU training affordable.
- The educational classifier is TF-IDF + Ridge on 160 LLM annotations, not a BERT head on 500k. Same recipe, ~3000× fewer labels. ρ = 0.783 is useful but not the real thing.
- SemDeDup uses LSA embeddings, not a neural sentence encoder — lexical at heart, so it will miss paraphrase. Reported as LSA-SemDeDup.
- Scale is tiny: 0.9M and 2.4M non-embedding parameters, 600k and 900k tokens. Three orders of magnitude below where data-quality effects are believed to matter.
- The in-training eval used the first ~12k tokens of the 700k-token val shard
(
eval_batches6 × batch 8 × seq 256). Identical tokens for every arm, so the comparisons are valid and add no noise — but it is a thin subsample, and the full-val numbers fromeval_slices.pyreorder the top two arms. Raiseeval_batchesto cover the whole shard on any run that matters.
- The T4 is Turing (SM 7.5): no bf16. Configs use fp16 AMP. Copying a modern bf16 config will fail or silently degrade.
- FlashAttention-2 requires SM 8.0+ and will not build for a T4. The model uses PyTorch SDPA, which selects the memory-efficient backend.
- Muon's Newton-Schulz iteration runs in fp32 here (
ns_dtype); upstream uses bf16. On Ampere+ setns_dtype=torch.bfloat16to recover the speed. torch.compileis off by default — flaky on Turing. Benchmark first.- Pool sizing: to end up with T training tokens at retention k you need ~T/k pool tokens. 200M at k=0.25 → 800M pool tokens, 1.6GB as uint16.