RQZ-Golf v1: Depth recurrence for parameter efficiency

[TheCause]

Non-record experimental submission

Approach: Replace some unique layers with a single shared recurrent layer applied K times, saving parameters while increasing effective depth.

Architecture

• 7 unique layers (encoder/decoder with U-Net skip connections)
• 1 recurrent layer applied K=3 times with learned iteration embeddings
• Effective depth: 10 layers (7 unique + 3 recurrent) vs baseline 9
• Residual scaling by 1/sqrt(K) for stability

Key ideas

1. Depth recurrence: shared weights across K passes saves ~20% parameters
2. Iteration embeddings: per-pass learned vector (psi_k) for pass-awareness
3. Test-time compute: increase K at inference (K'>K) for better BPB without changing model size

Status

• Preliminary baseline: 1.5283 BPB (1 shard, 1xA100)
• RQZ-Golf architecture implemented, not yet benchmarked on full dataset
• Requesting compute credits for full evaluation

Theoretical basis

Inspired by Universal Transformers (Dehghani 2019) and Deep Equilibrium Models (Bai 2019).