Seq2048 + FP16 Tied Embedding + Tuned LR (val_bpb 1.2067)

[yahya010]

Summary

Five changes stacked on the Naive Baseline, achieving mean val_bpb 1.2067 (3 seeds):

• Sequence length 2048 (from 1024): longer context per step outweighs reduced step count
• FP16 tied embedding passthrough: eliminates ~0.007 BPB quant gap (dual-duty matrix is disproportionately sensitive to int8)
• MLP hidden 960 (from 1024): trimmed to fit fp16 embedding under 16MB cap
• Lower LRs: MATRIX_LR=0.032, SCALAR_LR=0.032, TIED_EMBED_LR=0.04
• Warmdown 3600 (from 1200): ensures LR decay fires under wallclock cap

Results

Seed	Steps	val_loss	val_bpb	Artifact size
1337	10,408	2.0370	1.2064	15,632,845
42	10,403	2.0383	1.2072	15,635,682
3	10,375	2.0370	1.2064	15,633,777

Mean val_bpb: 1.2067 (std: 0.00044)
Improvement: 0.0353 nats (threshold: 0.005), t=-70.69, p << 0.01

Hardware: 8xH100 80GB HBM3, PyTorch 2.8.0+cu128, ~57.65ms/step.

Test plan

• 3 seeds on 8xH100, all under 600s wallclock
• All artifacts under 16MB cap
• Statistical significance p << 0.01
• Post-quant roundtrip validation matches