[davek44]

Train coverage r, r2 and spec are now updated on every train.train_metrics_everyth batch of each dataset (default 10; 1 restores every batch). Train loss and gene metrics still cover every batch.

Why. Train metrics are only logged, and early stopping uses them only when there is no validation data. At high track counts their per-step update is expensive. At 9-18 scale (8361 tracks × 24576 bins, batch 2), the update costs about 145 ms of a 1.61 s A100 step, roughly 9%. Spec is 55% of that and r + r2 the remaining 45%. Every 10th batch brings it under 1%, about 20 GPU-hours per 50-epoch fold. These figures were measured on a 4090 and scaled to the A100 by memory bandwidth. The in-step cost matched the standalone timing: +81 ms versus 79 ms.

Details

• The counter is per dataset, so each dataset's first batch of every epoch is always scored. The sampler shuffles batches across datasets, so the sample can't alias with dataset order.
• Gene metrics stay on every batch. They have no length axis, so they're cheap, and gene R² raises at epoch end on fewer than two observations, which a sparse sample could hit.
• Runs without validation data now select model_best from sampled train metrics.

Testing

• test_train.py and test_spec_metrics.py pass, except test_train_gene.
• test_train_gene fails its learning-floor check (valid r < 0.1) on unmodified main too, and it also passed once on this branch, so it looks flaky.

🤖 Generated with Claude Code