Future: compact decision model (<10M params)

[Oaklight]

Parent: #58

Idea: Purpose-built tiny decision model — 2-4 transformer layers, 128-256 width, byte-level or subword encoder.

References:

• CUA-S1: 706K params, 2-layer transformer, byte encoder → 99.7% on form-filling
• jevlike: AttentionHead with byte encoder
• Ettin-68M: 68M params, 28ms — current smallest viable backbone

Open questions:

• Can a <10M model match Ettin-68M's zero-shot 47.6% avg accuracy?
• Does a byte encoder avoid tokenizer-dependent surface-form bias?
• What's the minimum capacity for each head type (noul is simpler than choice)?

Why deferred: Ettin-68M at 28ms is already fast enough for most use cases. Building a custom architecture from scratch is high effort for marginal latency gains.

When to revisit: If there's demand for <10ms inference or edge deployment.

[Oaklight]

Update from Head-Size Sweep (Sep 26)

R1 results show Reranker-0.6B achieves 58.1% aggregate accuracy with only 402K trainable head params on a 596M frozen backbone — already a compact decision model. Key findings relevant to this issue:

• Smallest effective head: r64 mlp=0 = 467K params (sweep optimal for 0.6B)
• MLP projector doesn't help for small backbones (-2.3pp vs mlp=0 at 5 epochs)
• Full model: 596M backbone + 467K heads = ~597M total (deployable on 12GB GPU)

The "compact <10M params" goal from this issue may need reframing:

• Head-only compact: Already achieved (467K params). But needs a frozen backbone at inference time.
• Standalone compact: Would need knowledge distillation from larger models into a tiny backbone — different from our current frozen-backbone approach.

Keeping open as deferred — may revisit with distillation in a later stage.