Record: Sliding Window + FP16 Embed + 10L + Muon WD + Overtone Init (val_bpb=1.1748)

[notapplica]

Summary

Mean val_bpb: 1.1748 (3 seeds, p<0.001)

Stacks 6 orthogonal improvements over the baseline:

1. Sliding window evaluation (stride=64, seq_len=1024) — every token scored with 960+ context
2. FP16 tied embedding export — skip int8 quantization for tok_emb (errors compound in both input/output paths)
3. 10 transformer layers (up from 9) — Muon weight decay compresses enough to fit the extra layer
4. Decoupled weight decay for Muon optimizer (0.02) — Muon has no built-in regularization; adding p.mul_(1 - wd * lr) improves generalization + quantization
5. Overtone spectral embedding init — SVD power-law spectrum shaping
6. Phase-transition residual mixing — sigmoid-scheduled resid_mix initialization

Seed	val_loss	val_bpb	Steps	ms/step
1337	1.9849	1.1756	10424	57.55
42	1.9827	1.1742	10710	56.06
7	1.9830	1.1744	10498	57.18
Mean	1.9835	1.1748

Artifact: ~14.7 MB (under 16 MB limit)

[0hq]

Looks good to me!

[FI-Mihej]

@0hq , looks moltbot-ty to me. Just look to issues opened by it:

• Parameter Golf Live AI Commentary #140
• Pausing on the competition until grant comes in #138

[notapplica]

#138 is me lolol
Not moltbot but somewhat automated (i steer) (:
I have one claude working on the challenge and one claude analyzing everything in public