Skip to content
RoniLeorPublic

About

Convert Gemma checkpoints into recursive and relaxed-recursive Transformers.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

Repository files navigation

RecGemma

Fold Gemma's decoder depth into a smaller, reusable block.

Python 3.11–3.13 uv Apache 2.0 license research preview

Quick start · Measured result · Project status · Training roadmap · Benchmarks · Research

RecGemma converts an original Gemma causal-language-model checkpoint into a parameter-sharing Transformer whose compact decoder block is executed multiple times. It implements strict and relaxed initialization from Relaxed Recursive Transformers, saves a reloadable Hugging Face checkpoint, and generates publication-ready reports.

Measured RecGemma weight conversion

Important

Conversion is initialization, not training. The measured checkpoint above is structurally valid and reloadable, but it does not yet retain coherent language quality. Uptraining or knowledge distillation is required before deployment.

Project status

RecGemma is an executable research preview, not a pretrained model release.

Area Status Evidence
Strict and relaxed conversion ✅ Implemented 18 source layers become 9 stored layers and 18 effective evaluations
SVD weight reconstruction ✅ Measured Real matrix-level metrics in weights.json and the chart below
Checkpoint save and reload ✅ Verified Safetensors checkpoint reloads without expanding the shared layers
Recursive execution ✅ Verified Loop 2 receives the hidden state emitted by loop 1
Text generation ⚠️ Structural only The untrained converted checkpoint runs, but currently generates incoherent text
Post-conversion training ⬜ Required Training command, data pipeline, and resumable optimizer state are not implemented
Paper-scale evaluation ⬜ Required The benchmark runner exists; complete trained-model scores have not been produced
Deployment readiness ⛔ Not ready Blocked on quality recovery and complete evaluation

Measured conversion

The chart above was produced by a real google/gemma-2b relaxed conversion on Apple Silicon—not by sample data.

Property Measured result
Decoder layers 18 → 9 stored, executed twice
Parameters 2.51B → 1.83B
Parameter reduction 27.0%
LoRA rank 256
Saved checkpoint 3.4 GB Safetensors

The parameter count includes embeddings and depth-specific adapters, so halving the stored decoder layers does not halve the complete checkpoint. The right side of the chart reports how much layer-difference energy each truncated-SVD adapter retained; it is a weight-reconstruction diagnostic, not a language-quality score.

Quick start

Requirements: Python 3.11–3.13, uv, enough memory for the original and converted checkpoints, and accepted access to Google's gated Gemma weights on Hugging Face.

uv sync --all-groups

uv run recgemma convert google/gemma-2b artifacts/gemma-relaxed \
  --mode relaxed \
  --loops 2 \
  --rank 256 \
  --dtype bfloat16 \
  --device mps

The command writes a reloadable checkpoint, weights.json, and weights.png to artifacts/gemma-relaxed/. Replace mps with cuda or cpu for the target machine. SVD is substantially faster on CPU than MPS for this model; conversion still needs enough host memory for the source weights and temporary matrices.

Why RecGemma

  • Compact storage: register each shared decoder layer only once.
  • Full effective depth: execute the compact block repeatedly during inference.
  • Relaxed sharing: recover loop-specific differences with low-rank adapters.
  • Hugging Face native: save and reload without expanding discarded layers.
  • Visible experiments: Rich progress plus automatic JSON and PNG reports.

What is implemented

  • Strict recursive Gemma with paper-aligned fixed-end stepwise initialization.
  • Relaxed recursive Gemma with averaged shared layers.
  • Independent depth-wise LoRA adapters on every attention and MLP projection.
  • Truncated-SVD adapter initialization from each original layer residual.
  • Hugging Face save_pretrained and explicit from_pretrained round trips.
  • Cache-safe effective layer indices when the shared block is repeated.
  • Original-versus-recursive evaluation through lm-evaluation-harness.
  • JSON results and 2400×1400 PNG comparisons for weights and benchmark quality.
  • Rich matrix-level conversion progress, benchmark progress, and terminal result tables.

The current scope is deliberately limited to the original Gemma architecture, including google/gemma-2b. Gemma 2 and Gemma 3 use depth-dependent attention patterns and are rejected rather than converted incorrectly.

How it works

flowchart LR
    A["Gemma · 18 decoder layers"] --> B["Group corresponding depths"]
    B --> C["Average 9 shared layers"]
    C --> D["Recover differences with SVD LoRA"]
    D --> E["Run shared block · loop 1"]
    E --> F["Run shared block · loop 2"]
    F --> G["18 effective layer evaluations"]
Loading

In relaxed mode, shared layer i is initialized from original layers i and i + 9. A rank-limited adapter captures each original layer's difference from their shared average. The two passes execute sequentially: the hidden state from the first pass becomes the input to the second.

Conversion modes

Strict recursion keeps half the unique decoder layers and runs them twice:

uv run recgemma convert google/gemma-2b artifacts/gemma-recursive \
  --mode recursive \
  --loops 2 \
  --dtype bfloat16

The relaxed command in the quick start uses average initialization and depth-specific rank-256 LoRA. Strict mode has fewer parameters; relaxed mode uses the adapters to approximate each source depth more closely.

Use --device cuda or --device mps when appropriate. Conversion temporarily needs the original checkpoint plus working memory for averaging and SVD. The earlier recgemma SOURCE OUTPUT spelling remains accepted for compatibility.

During strict conversion, Rich shows every source layer selected for recursive reuse. During relaxed conversion, it shows every attention/MLP weight matrix as its depth residuals are approximated. Every successful conversion automatically writes weights.json and weights.png inside the checkpoint directory. The report contains the layer mapping, original and converted parameter counts, and, for relaxed mode, residual norm, reconstruction error, and retained residual energy for every truncated-SVD adapter. These are weight-approximation metrics, not downstream model accuracy.

Conversion only initializes a smaller shared model; it does not train it. The checkpoint can run immediately, but quality is expected to fall until the paper's uptraining or distillation stage is reproduced.

Training roadmap

Training is the next required milestone. It must update all student parameters—the shared decoder weights, RMSNorm weights, embeddings, output head, and depth-specific adapters. Training only the LoRA adapters would not reproduce the paper's method.

Phase Deliverable Completion gate Status
0. Conversion Saved relaxed Gemma checkpoint and weight report Reload and execute both recurrent passes ✅ Complete
1. Training engine recgemma train with streaming data, validation, checkpointing, and deterministic resume One batch produces finite loss and gradients on every intended parameter group ⬜ Next
2. Distillation Frozen full-size Gemma teacher with causal-LM loss plus forward-KL loss Student loss, teacher KL, gradient norm, throughput, and memory are logged ⬜ Planned
3. Smoke run Short CPU/MPS run followed by a small GPU pilot Loss decreases, checkpoint resumes exactly, and sample text improves ⬜ Planned
4. Scale run Distributed mixed-precision training Complete the chosen token budget without divergence ⬜ Planned
5. Evaluation Perplexity, seven few-shot tasks, and generation samples Publish measured JSON/PNG results against identical baselines ⬜ Planned
6. Release CI, contribution/citation files, model card, and trained checkpoint Quality limits, hardware, data, and reproducibility metadata documented ⬜ Planned

Paper reproduction target

The ICLR experiments used 15B-token ablations and a 60B-token final run on SlimPajama. Their setup used context length 2048, an approximately 2M-token batch, AdamW with learning rate 2e-4, cosine decay, mixed precision, DeepSpeed ZeRO, and eight H100 GPUs. The strongest configuration used online forward-KL distillation from a full-size teacher.

That is a reproduction target—not the first development run. The practical order is one-batch correctness, a resumable small pilot, then distributed scale. Dataset quality and distribution must be treated as experimental variables: the paper reports that SlimPajama uptraining can reduce full-size Gemma benchmark quality because its distribution differs from Gemma's original training data.

Benchmark quality

RecGemma compares both checkpoints with the same few-shot protocol on the seven downstream tasks reported for the paper's Gemma experiments. It leaves the shot count unset so the pinned harness definitions use their standard value, matching the paper's evaluation description:

Task Reported metric
LAMBADA accuracy
HellaSwag normalized accuracy
PIQA normalized accuracy
WinoGrande accuracy
ARC-Easy normalized accuracy
ARC-Challenge normalized accuracy
OpenBookQA normalized accuracy

First run a short pipeline smoke test. Limited scores are useful for checking the setup, but are not statistically comparable final results:

uv run recgemma benchmark \
  google/gemma-2b \
  artifacts/gemma-recursive \
  --output runs/smoke \
  --limit 20 \
  --device mps \
  --dtype bfloat16

Then omit --limit for the complete comparison:

uv run recgemma benchmark \
  google/gemma-2b \
  artifacts/gemma-recursive \
  --output runs/full \
  --device mps \
  --dtype bfloat16

Each run writes results.json with the raw comparison data and quality.png with per-task accuracy, macro-average quality retention, quality delta, and parameter reduction. The PNG is created automatically; the final Rich summary shows clickable paths to both artifacts. No benchmark figure is presented in this README until it comes from a trained checkpoint and a complete measured run.

Load and run

RecGemma checkpoints use Gemma's configuration with additional conversion metadata. Load them through the explicit model class:

from typing import Any

import torch
from transformers import AutoTokenizer

from recgemma import RecursiveGemmaForCausalLM

checkpoint: str = "artifacts/gemma-recursive"
tokenizer: Any = AutoTokenizer.from_pretrained(checkpoint)
model: RecursiveGemmaForCausalLM = RecursiveGemmaForCausalLM.from_pretrained(
    checkpoint,
    dtype="auto",
)
inputs: dict[str, torch.Tensor] = tokenizer(
    "The recursive Transformer is",
    return_tensors="pt",
)
outputs: torch.Tensor = model.generate(**inputs, max_new_tokens=32)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Standard training loops can update every shared weight and LoRA adapter after conversion. Sharing reduces stored decoder parameters, not the number of layer evaluations in a complete forward pass.

Research basis

RecGemma is an independent implementation inspired by:

The repository currently implements conversion, persistence, reporting, and evaluation—not uptraining, continuous depth-wise batching, early exit, or the mechanistic architecture variants. Gemma is a Google model; this project is not affiliated with or endorsed by Google or the paper authors.

Research after quality recovery

These experiments should remain separate ablations until the paper-aligned training baseline works:

  • Plot hidden-state deltas, cosine convergence, attention-map stability, and per-loop logit KL.
  • Compare the current full-cycle architecture with prelude/recurrent/coda layouts and input injection.
  • Support projection-specific ranks: Gemma's K/V matrices cap rank at 256, while Q/O and MLP matrices can test rank 512.
  • Evaluate early exits and continuous depth-wise batching only after final-loop quality is established.

Layout

recgemma/
├── src/
│   └── recgemma/
│       ├── engine/       # conversion, evaluation, metrics, plots and reports
│       ├── models/
│       │   └── gemma/    # Gemma recipe and reloadable recursive model
│       ├── nn/           # recursive layers and depth-wise LoRA
│       ├── utils/        # structural helpers and Rich progress reporting
│       └── cli.py        # compact convert and benchmark commands
├── docs/                 # architecture notes and benchmark assets
└── tests/                # flat focused test modules

See architecture for ownership and dependency rules.

Verify

uv run ruff check .
uv run ruff format --check .
uv run pyright
uv run ty check
uv run pytest

License

RecGemma is available under the Apache License 2.0. Model weights and datasets remain subject to their respective licenses and access terms.

About

Convert Gemma checkpoints into recursive and relaxed-recursive Transformers.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages