A curated collection of papers, code, and benchmarks for weight-space programming.
Website ↗ · Start here · Browse the library · 简体中文 · Suggest a resource ↗
⭐ Find this useful? Star the repo to bookmark the collection and help others discover it.
Four starting points, from the first ideas to hands-on exploration.
-
Sparks of WSP · Discover
Read behavior from weights; explore network repair and construction. -
WSP from Scratch · Understand
Explore neural circuits for arithmetic, sorting, and parity. -
Tracr · Build
Compile readable programs into transformer weights. -
Community Experiments · Explore
Find replications, weight-reading tests, and evaluation tools.
Begin with direct experiments and community projects, then explore the foundations.
Direct work · Community · Compilers · Weight generation · Weight-space models · Model editing · Foundations
Rikiya Takehi, Jayden Teoh, Alex Zhang, Omar Khattab · 2026 · Blog + released experiment artifacts
Article · Code, logs and networks
Explores identifying network behavior from weights, repairing a control policy, writing game-playing networks, and editing a language model.
Useful for studying the difference between reading weights, constructing a network, and repairing an existing one.
Experimental conditions & evidence
- Access conditions vary: some experiments use a single response without tools; others allow an external Python sandbox and bounded feedback. The repository documents these conditions per experiment.
- Evidence note: the release contains reported runs, not all earlier prompt trials. Its README lists omitted artifacts and reconstruction instructions; do not interpret the release as an exhaustive search history.
Chandan Singh · 2026 · Blog + autonomous coding experiments
Article and interactive circuits · Code and snapshots · Agent instructions
An agent assigns small transformer weights and may change the architecture, then iterates using evaluation feedback.
Covers arithmetic, sorting, parity, counting, Boolean expressions, and other bounded tasks; the saved programs make concrete circuit inspection possible.
Experimental conditions & evidence
- Evidence note: reported algorithmic scores use finite evaluation samples. The article acknowledges evaluation leakage in language tasks, fitting in the SNLI run, and parameter-count gaming through buffers. These are not valid demonstrations of general training-free language understanding.
Community work, listed separately from the original reports. Summarized from public sources; not independently rerun here.
-
Weight-Space Claude
A Claude-based harness, scorers, and side-by-side results for selected Sparks prompts. -
Weight-Space Programming experiments — Notes by Lex
Cross-model game identification and a separate DQN checkpoint blind test with a restricted tensor reader. -
WSPBench
A research preview separating unaided, weight-analysis, and instrumented conditions; public CPU replay of saved submissions. -
Beyond Math
Audit of released editing traces and paired non-math probes of base and edited checkpoints. -
Weight-Space Programming for the Social Safety Net
Construction of networks implementing stipulated piecewise-linear formulas, with protocols and trained comparators.
Project scope & reproduction notes
Early community projects are listed separately from the originating reports. Descriptions below summarize public READMEs checked on the curation date; we have not rerun their experiments. A replay of saved outputs, a checkpoint evaluation, and a fresh replication of an editing procedure are different forms of evidence.
- Weight-Space Claude — Covers game identification, diffusion-shape reading, and MinAtar construction; not a replication of every original experiment.
- Weight-Space Programming experiments — Notes by Lex — Includes shape-only controls; small checkpoint samples do not establish a general success rate.
- WSPBench — The published pilot covers the unaided condition on small constructed circuits. Replay does not rerun model inference.
- Beyond Math — Separates author benchmark records from small local probes; does not reproduce the original editing search.
- Weight-Space Programming for the Social Safety Net — Simplified formula experiments, not operational benefit calculators or evidence about full eligibility decisions.
Foundational and closely related work. Compilation provides a useful baseline for asking what a general-purpose LLM adds.
-
Thinking Like Transformers
Gail Weiss, Yoav Goldberg, Eran Yahav. ICML 2021.Introduces RASP, a programming language expressing attention-style computations such as histograms and sorting. A conceptual basis for writing transformer algorithms.
-
Tracr: Compiled Transformers as a Laboratory for Interpretability
David Lindner et al. NeurIPS 2023.Compiles readable programs into transformer weights with known structure, enabling controlled interpretability experiments. Code.
-
Learning Transformer Programs
Dan Friedman, Alexander Wettig, Danqi Chen. NeurIPS 2023.Trains constrained transformers that can be converted into discrete, human-readable programs. This method uses gradient training; it is a related alternative, not a no-training construction result. Code.
These methods train a weight generator. They should not be conflated with a general-purpose LLM reasoning directly over weights without task-specific generator training.
-
HyperNetworks
David Ha, Andrew Dai, Quoc V. Le. 2016 preprint.Uses one network to generate the weights of another, trained end-to-end. A foundational comparison for any claim about generating networks rather than optimizing each target independently.
-
Parameter Prediction for Unseen Deep Architectures
Boris Knyazev et al. NeurIPS 2021.Predicts parameters from architecture graphs and introduces DeepNets-1M, providing a learned alternative to direct construction. Code and dataset tools.
Neurons can often be permuted without changing a network's function. These works help explain why raw tensor coordinates are not a unique semantic representation.
-
Permutation Equivariant Neural Functionals
Allan Zhou et al. NeurIPS 2023.Builds layers that process weights while respecting neuron-permutation symmetries; applications include prediction and editing of implicit neural representations. Code.
-
Neural Functional Transformers
Allan Zhou et al. NeurIPS 2023.Uses attention to construct permutation-equivariant models over weight spaces. Useful for comparing learned weight readers with general-purpose LLM readers. Code.
-
Equivariant Deep Weight Space Alignment
Aviv Navon et al. ICML 2024.Learns to align networks under permutation symmetries, relevant to comparing and combining parameter representations. Code.
Selected comparison points for changing behavior through parameter updates; this is not a comprehensive knowledge-editing or model-merging bibliography.
-
Locating and Editing Factual Associations in GPT
Kevin Meng et al. NeurIPS 2022.Introduces ROME, which localizes and edits factual associations through targeted feed-forward weight updates. It is an editing algorithm with an optimization component, not an LLM directly writing a solution from scratch. Code.
-
Editing Models with Task Arithmetic
Gabriel Ilharco et al. ICLR 2023.Constructs task vectors from fine-tuned weight differences and combines them arithmetically to steer behavior. Useful for distinguishing weight composition from direct circuit synthesis. Code.
-
A Mathematical Framework for Transformer Circuits
Nelson Elhage et al. 2021 research article.Develops residual-stream, QK, and OV views of transformer computation. Supplies vocabulary for both reading trained circuits and designing explicit ones.
Reproduction checklist · Contributing · Acknowledgments · License
Awesome Weight-Space Programming is an independent, open-source resource list for researchers, engineers, and anyone exploring neural networks as programmable artifacts. Contributions are welcome from the community.
Weight-space programming explores how an agent can inspect, construct, or modify neural network parameters to implement a specified behavior. This collection connects direct LLM-driven experiments to program compilation, learned weight generation, weight-space representations, and targeted model editing.
Scope. Direct weight reasoning and construction are the focus. Related papers are labeled as foundations; they need not use the term “weight-space programming.” General fine-tuning, prompt optimization, and architecture search without weight construction are outside the main scope.
Curation. Last curated November 3, 2026. This is a selective collection, not an exhaustive survey. Inclusion means a resource is useful to inspect; it does not mean its results have been independently reproduced here. The two originating reports are blog posts, not verified peer-reviewed publications.
Compare five research routes
| Research route | How weights are produced or used | Why it matters |
|---|---|---|
| Direct LLM programming | An LLM reasons about tensors or writes code that assigns them. | The central topic of this list. |
| Program compilation | A compiler maps a formal program to a network. | Provides explicit mechanisms and useful correctness baselines. |
| Learned weight generation | A trained hypernetwork predicts another network's parameters. | A distinct route to generating weights. |
| Weight-space modeling | A model reads or transforms another model's parameters. | Addresses symmetries and representations of networks. |
| Targeted editing | An algorithm updates selected weights or combines task vectors. | Offers comparison points for behavior modification. |
Open the reproduction checklist
When comparing results or proposing a new entry, make the experimental contract visible:
- Target and architecture: fixed or editable architecture; parameter, buffer, and constant storage; numerical precision.
- Access: weights only, tensor tools, forward passes, activations, examples, labels, or environment feedback.
- Construction: direct assignment, compilation, lookup tables, learned generation, or fitted coefficients.
- Evaluation: unseen seeds or splits, exhaustive finite-domain checks where feasible, boundary cases, and length generalization.
- Search budget: attempts, model identity, token usage, execution time, and selection procedure.
- Evidence: runnable artifacts and exact configurations; separate reported scores, independent regrading, checkpoint reruns, and fresh procedure replications.
“No gradients” alone does not imply no fitting, no feedback, or no search. A low parameter count does not imply low total storage when constants are moved into buffers or code. Constructed explanations are valuable hypotheses that still benefit from causal or equivalence checks.
New papers, working projects, replications, and corrections are welcome. Please read CONTRIBUTING.md, then open a pull request or suggest a resource.
Prefer primary sources and a short explanation of why the item belongs. Keep direct work separate from related foundations, and label experimental status accurately.
Design references & acknowledgments
List structure follows the curation principles of the Awesome manifesto, with organizational inspiration from Awesome Node.js and Awesome AI Agents. The visual introduction also draws on the banner-first layouts of Dify and n8n, and the reading hierarchy of Awesome-LLM. The cover artwork is original to this list. All research and software credit belongs to the linked authors and maintainers. The Awesome badge does not imply inclusion in the central Awesome index.
This list's original text is dedicated to the public domain under CC0 1.0. Linked papers, code, models, and datasets retain their respective licenses.
Also known as Awesome-WSP / awesome-wsp.