Fold Gemma's decoder depth into a smaller, reusable block.
Quick start · Measured result · Project status · Training roadmap · Benchmarks · Research
RecGemma converts an original Gemma causal-language-model checkpoint into a parameter-sharing Transformer whose compact decoder block is executed multiple times. It implements strict and relaxed initialization from Relaxed Recursive Transformers, saves a reloadable Hugging Face checkpoint, and generates publication-ready reports.
Important
Conversion is initialization, not training. The measured checkpoint above is structurally valid and reloadable, but it does not yet retain coherent language quality. Uptraining or knowledge distillation is required before deployment.
RecGemma is an executable research preview, not a pretrained model release.
| Area | Status | Evidence |
|---|---|---|
| Strict and relaxed conversion | ✅ Implemented | 18 source layers become 9 stored layers and 18 effective evaluations |
| SVD weight reconstruction | ✅ Measured | Real matrix-level metrics in weights.json and the chart below |
| Checkpoint save and reload | ✅ Verified | Safetensors checkpoint reloads without expanding the shared layers |
| Recursive execution | ✅ Verified | Loop 2 receives the hidden state emitted by loop 1 |
| Text generation | The untrained converted checkpoint runs, but currently generates incoherent text | |
| Post-conversion training | ⬜ Required | Training command, data pipeline, and resumable optimizer state are not implemented |
| Paper-scale evaluation | ⬜ Required | The benchmark runner exists; complete trained-model scores have not been produced |
| Deployment readiness | ⛔ Not ready | Blocked on quality recovery and complete evaluation |
The chart above was produced by a real google/gemma-2b relaxed conversion on
Apple Silicon—not by sample data.
| Property | Measured result |
|---|---|
| Decoder layers | 18 → 9 stored, executed twice |
| Parameters | 2.51B → 1.83B |
| Parameter reduction | 27.0% |
| LoRA rank | 256 |
| Saved checkpoint | 3.4 GB Safetensors |
The parameter count includes embeddings and depth-specific adapters, so halving the stored decoder layers does not halve the complete checkpoint. The right side of the chart reports how much layer-difference energy each truncated-SVD adapter retained; it is a weight-reconstruction diagnostic, not a language-quality score.
Requirements: Python 3.11–3.13, uv, enough memory for the original and
converted checkpoints, and accepted access to Google's gated Gemma weights on
Hugging Face.
uv sync --all-groups
uv run recgemma convert google/gemma-2b artifacts/gemma-relaxed \
--mode relaxed \
--loops 2 \
--rank 256 \
--dtype bfloat16 \
--device mpsThe command writes a reloadable checkpoint, weights.json, and weights.png
to artifacts/gemma-relaxed/. Replace mps with cuda or cpu for the target
machine. SVD is substantially faster on CPU than MPS for this model; conversion
still needs enough host memory for the source weights and temporary matrices.
- Compact storage: register each shared decoder layer only once.
- Full effective depth: execute the compact block repeatedly during inference.
- Relaxed sharing: recover loop-specific differences with low-rank adapters.
- Hugging Face native: save and reload without expanding discarded layers.
- Visible experiments: Rich progress plus automatic JSON and PNG reports.
- Strict recursive Gemma with paper-aligned fixed-end stepwise initialization.
- Relaxed recursive Gemma with averaged shared layers.
- Independent depth-wise LoRA adapters on every attention and MLP projection.
- Truncated-SVD adapter initialization from each original layer residual.
- Hugging Face
save_pretrainedand explicitfrom_pretrainedround trips. - Cache-safe effective layer indices when the shared block is repeated.
- Original-versus-recursive evaluation through
lm-evaluation-harness. - JSON results and 2400×1400 PNG comparisons for weights and benchmark quality.
- Rich matrix-level conversion progress, benchmark progress, and terminal result tables.
The current scope is deliberately limited to the original Gemma architecture,
including google/gemma-2b. Gemma 2 and Gemma 3 use depth-dependent attention
patterns and are rejected rather than converted incorrectly.
flowchart LR
A["Gemma · 18 decoder layers"] --> B["Group corresponding depths"]
B --> C["Average 9 shared layers"]
C --> D["Recover differences with SVD LoRA"]
D --> E["Run shared block · loop 1"]
E --> F["Run shared block · loop 2"]
F --> G["18 effective layer evaluations"]
In relaxed mode, shared layer i is initialized from original layers i and
i + 9. A rank-limited adapter captures each original layer's difference from
their shared average. The two passes execute sequentially: the hidden state from
the first pass becomes the input to the second.
Strict recursion keeps half the unique decoder layers and runs them twice:
uv run recgemma convert google/gemma-2b artifacts/gemma-recursive \
--mode recursive \
--loops 2 \
--dtype bfloat16The relaxed command in the quick start uses average initialization and depth-specific rank-256 LoRA. Strict mode has fewer parameters; relaxed mode uses the adapters to approximate each source depth more closely.
Use --device cuda or --device mps when appropriate. Conversion temporarily
needs the original checkpoint plus working memory for averaging and SVD.
The earlier recgemma SOURCE OUTPUT spelling remains accepted for compatibility.
During strict conversion, Rich shows every source layer selected for recursive
reuse. During relaxed conversion, it shows every attention/MLP weight matrix as
its depth residuals are approximated. Every successful conversion automatically
writes weights.json and weights.png inside the checkpoint directory. The
report contains the layer mapping, original and converted parameter counts, and,
for relaxed mode, residual norm, reconstruction error, and retained residual
energy for every truncated-SVD adapter. These are weight-approximation metrics,
not downstream model accuracy.
Conversion only initializes a smaller shared model; it does not train it. The checkpoint can run immediately, but quality is expected to fall until the paper's uptraining or distillation stage is reproduced.
Training is the next required milestone. It must update all student parameters—the shared decoder weights, RMSNorm weights, embeddings, output head, and depth-specific adapters. Training only the LoRA adapters would not reproduce the paper's method.
| Phase | Deliverable | Completion gate | Status |
|---|---|---|---|
| 0. Conversion | Saved relaxed Gemma checkpoint and weight report | Reload and execute both recurrent passes | ✅ Complete |
| 1. Training engine | recgemma train with streaming data, validation, checkpointing, and deterministic resume |
One batch produces finite loss and gradients on every intended parameter group | ⬜ Next |
| 2. Distillation | Frozen full-size Gemma teacher with causal-LM loss plus forward-KL loss | Student loss, teacher KL, gradient norm, throughput, and memory are logged | ⬜ Planned |
| 3. Smoke run | Short CPU/MPS run followed by a small GPU pilot | Loss decreases, checkpoint resumes exactly, and sample text improves | ⬜ Planned |
| 4. Scale run | Distributed mixed-precision training | Complete the chosen token budget without divergence | ⬜ Planned |
| 5. Evaluation | Perplexity, seven few-shot tasks, and generation samples | Publish measured JSON/PNG results against identical baselines | ⬜ Planned |
| 6. Release | CI, contribution/citation files, model card, and trained checkpoint | Quality limits, hardware, data, and reproducibility metadata documented | ⬜ Planned |
The ICLR experiments used 15B-token ablations and a 60B-token final run on
SlimPajama. Their setup used context length 2048, an approximately 2M-token
batch, AdamW with learning rate 2e-4, cosine decay, mixed precision,
DeepSpeed ZeRO, and eight H100 GPUs. The strongest configuration used online
forward-KL distillation from a full-size teacher.
That is a reproduction target—not the first development run. The practical order is one-batch correctness, a resumable small pilot, then distributed scale. Dataset quality and distribution must be treated as experimental variables: the paper reports that SlimPajama uptraining can reduce full-size Gemma benchmark quality because its distribution differs from Gemma's original training data.
RecGemma compares both checkpoints with the same few-shot protocol on the seven downstream tasks reported for the paper's Gemma experiments. It leaves the shot count unset so the pinned harness definitions use their standard value, matching the paper's evaluation description:
| Task | Reported metric |
|---|---|
| LAMBADA | accuracy |
| HellaSwag | normalized accuracy |
| PIQA | normalized accuracy |
| WinoGrande | accuracy |
| ARC-Easy | normalized accuracy |
| ARC-Challenge | normalized accuracy |
| OpenBookQA | normalized accuracy |
First run a short pipeline smoke test. Limited scores are useful for checking the setup, but are not statistically comparable final results:
uv run recgemma benchmark \
google/gemma-2b \
artifacts/gemma-recursive \
--output runs/smoke \
--limit 20 \
--device mps \
--dtype bfloat16Then omit --limit for the complete comparison:
uv run recgemma benchmark \
google/gemma-2b \
artifacts/gemma-recursive \
--output runs/full \
--device mps \
--dtype bfloat16Each run writes results.json with the raw comparison data and quality.png
with per-task accuracy, macro-average quality retention, quality delta, and
parameter reduction. The PNG is created automatically; the final Rich summary
shows clickable paths to both artifacts. No benchmark figure is presented in
this README until it comes from a trained checkpoint and a complete measured
run.
RecGemma checkpoints use Gemma's configuration with additional conversion metadata. Load them through the explicit model class:
from typing import Any
import torch
from transformers import AutoTokenizer
from recgemma import RecursiveGemmaForCausalLM
checkpoint: str = "artifacts/gemma-recursive"
tokenizer: Any = AutoTokenizer.from_pretrained(checkpoint)
model: RecursiveGemmaForCausalLM = RecursiveGemmaForCausalLM.from_pretrained(
checkpoint,
dtype="auto",
)
inputs: dict[str, torch.Tensor] = tokenizer(
"The recursive Transformer is",
return_tensors="pt",
)
outputs: torch.Tensor = model.generate(**inputs, max_new_tokens=32)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))Standard training loops can update every shared weight and LoRA adapter after conversion. Sharing reduces stored decoder parameters, not the number of layer evaluations in a complete forward pass.
RecGemma is an independent implementation inspired by:
- Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA defines the conversion, SVD initialization, uptraining, and distillation target.
- A Mechanistic Analysis of Looped Reasoning Language Models motivates future recurrent-state diagnostics, input injection, and non-recurrent prelude/coda experiments.
The repository currently implements conversion, persistence, reporting, and evaluation—not uptraining, continuous depth-wise batching, early exit, or the mechanistic architecture variants. Gemma is a Google model; this project is not affiliated with or endorsed by Google or the paper authors.
These experiments should remain separate ablations until the paper-aligned training baseline works:
- Plot hidden-state deltas, cosine convergence, attention-map stability, and per-loop logit KL.
- Compare the current full-cycle architecture with prelude/recurrent/coda layouts and input injection.
- Support projection-specific ranks: Gemma's K/V matrices cap rank at 256, while Q/O and MLP matrices can test rank 512.
- Evaluate early exits and continuous depth-wise batching only after final-loop quality is established.
recgemma/
├── src/
│ └── recgemma/
│ ├── engine/ # conversion, evaluation, metrics, plots and reports
│ ├── models/
│ │ └── gemma/ # Gemma recipe and reloadable recursive model
│ ├── nn/ # recursive layers and depth-wise LoRA
│ ├── utils/ # structural helpers and Rich progress reporting
│ └── cli.py # compact convert and benchmark commands
├── docs/ # architecture notes and benchmark assets
└── tests/ # flat focused test modules
See architecture for ownership and dependency rules.
uv run ruff check .
uv run ruff format --check .
uv run pyright
uv run ty check
uv run pytestRecGemma is available under the Apache License 2.0. Model weights and datasets remain subject to their respective licenses and access terms.
