Skip to content

About

Awesome Weight-Space Programming: curated papers, code, benchmarks, and open experiments for reading, writing, and editing neural network weights.

Topics

Resources

Contributing

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

8 Commits

Folders and files

Repository files navigation

Awesome Weight-Space Programming — neural weights, readable and writable

Awesome Weight-Space Programming

A curated collection of papers, code, and benchmarks for weight-space programming.

Awesome License: CC0 1.0 Contributions welcome

Website ↗   ·   Start here   ·   Browse the library   ·   简体中文   ·   Suggest a resource ↗

⭐ Find this useful? Star the repo to bookmark the collection and help others discover it.


Start here

Four starting points, from the first ideas to hands-on exploration.

  1. Sparks of WSP · Discover
    Read behavior from weights; explore network repair and construction.

  2. WSP from Scratch · Understand
    Explore neural circuits for arithmetic, sorting, and parity.

  3. Tracr · Build
    Compile readable programs into transformer weights.

  4. Community Experiments · Explore
    Find replications, weight-reading tests, and evaluation tools.

Research library

Begin with direct experiments and community projects, then explore the foundations.

Direct work · Community · Compilers · Weight generation · Weight-space models · Model editing · Foundations

Direct weight-space programming

Sparks of Weight-Space Programming

Rikiya Takehi, Jayden Teoh, Alex Zhang, Omar Khattab · 2026 · Blog + released experiment artifacts

Article · Code, logs and networks

Explores identifying network behavior from weights, repairing a control policy, writing game-playing networks, and editing a language model.

Useful for studying the difference between reading weights, constructing a network, and repairing an existing one.

Experimental conditions & evidence
  • Access conditions vary: some experiments use a single response without tools; others allow an external Python sandbox and bounded feedback. The repository documents these conditions per experiment.
  • Evidence note: the release contains reported runs, not all earlier prompt trials. Its README lists omitted artifacts and reconstruction instructions; do not interpret the release as an exhaustive search history.

Weight-Space Programming from scratch on simple tasks

Chandan Singh · 2026 · Blog + autonomous coding experiments

Article and interactive circuits · Code and snapshots · Agent instructions

An agent assigns small transformer weights and may change the architecture, then iterates using evaluation feedback.

Covers arithmetic, sorting, parity, counting, Boolean expressions, and other bounded tasks; the saved programs make concrete circuit inspection possible.

Experimental conditions & evidence
  • Evidence note: reported algorithmic scores use finite evaluation samples. The article acknowledges evaluation leakage in language tasks, fitting in the SNLI run, and parameter-count gaming through buffers. These are not valid demonstrations of general training-free language understanding.

Community experiments and benchmarks

Community work, listed separately from the original reports. Summarized from public sources; not independently rerun here.

Project scope & reproduction notes

Early community projects are listed separately from the originating reports. Descriptions below summarize public READMEs checked on the curation date; we have not rerun their experiments. A replay of saved outputs, a checkpoint evaluation, and a fresh replication of an editing procedure are different forms of evidence.

  • Weight-Space Claude — Covers game identification, diffusion-shape reading, and MinAtar construction; not a replication of every original experiment.
  • Weight-Space Programming experiments — Notes by Lex — Includes shape-only controls; small checkpoint samples do not establish a general success rate.
  • WSPBench — The published pilot covers the unaided condition on small constructed circuits. Replay does not rerun model inference.
  • Beyond Math — Separates author benchmark records from small local probes; does not reproduce the original editing search.
  • Weight-Space Programming for the Social Safety Net — Simplified formula experiments, not operational benefit calculators or evidence about full eligibility decisions.

Programs to transformer weights

Foundational and closely related work. Compilation provides a useful baseline for asking what a general-purpose LLM adds.

  • Thinking Like Transformers
    Gail Weiss, Yoav Goldberg, Eran Yahav. ICML 2021.

    Introduces RASP, a programming language expressing attention-style computations such as histograms and sorting. A conceptual basis for writing transformer algorithms.

  • Tracr: Compiled Transformers as a Laboratory for Interpretability
    David Lindner et al. NeurIPS 2023.

    Compiles readable programs into transformer weights with known structure, enabling controlled interpretability experiments. Code.

  • Learning Transformer Programs
    Dan Friedman, Alexander Wettig, Danqi Chen. NeurIPS 2023.

    Trains constrained transformers that can be converted into discrete, human-readable programs. This method uses gradient training; it is a related alternative, not a no-training construction result. Code.

Learned weight generation

These methods train a weight generator. They should not be conflated with a general-purpose LLM reasoning directly over weights without task-specific generator training.

  • HyperNetworks
    David Ha, Andrew Dai, Quoc V. Le. 2016 preprint.

    Uses one network to generate the weights of another, trained end-to-end. A foundational comparison for any claim about generating networks rather than optimizing each target independently.

  • Parameter Prediction for Unseen Deep Architectures
    Boris Knyazev et al. NeurIPS 2021.

    Predicts parameters from architecture graphs and introduces DeepNets-1M, providing a learned alternative to direct construction. Code and dataset tools.

Networks that process weights

Neurons can often be permuted without changing a network's function. These works help explain why raw tensor coordinates are not a unique semantic representation.

  • Permutation Equivariant Neural Functionals
    Allan Zhou et al. NeurIPS 2023.

    Builds layers that process weights while respecting neuron-permutation symmetries; applications include prediction and editing of implicit neural representations. Code.

  • Neural Functional Transformers
    Allan Zhou et al. NeurIPS 2023.

    Uses attention to construct permutation-equivariant models over weight spaces. Useful for comparing learned weight readers with general-purpose LLM readers. Code.

  • Equivariant Deep Weight Space Alignment
    Aviv Navon et al. ICML 2024.

    Learns to align networks under permutation symmetries, relevant to comparing and combining parameter representations. Code.

Related model editing

Selected comparison points for changing behavior through parameter updates; this is not a comprehensive knowledge-editing or model-merging bibliography.

  • Locating and Editing Factual Associations in GPT
    Kevin Meng et al. NeurIPS 2022.

    Introduces ROME, which localizes and edits factual associations through targeted feed-forward weight updates. It is an editing algorithm with an optimization component, not an LLM directly writing a solution from scratch. Code.

  • Editing Models with Task Arithmetic
    Gabriel Ilharco et al. ICLR 2023.

    Constructs task vectors from fine-tuned weight differences and combines them arithmetically to steer behavior. Useful for distinguishing weight composition from direct circuit synthesis. Code.

Mechanistic foundations

  • A Mathematical Framework for Transformer Circuits
    Nelson Elhage et al. 2021 research article.

    Develops residual-stream, QK, and OV views of transformer computation. Supplies vocabulary for both reading trained circuits and designing explicit ones.

About this list

Reproduction checklist · Contributing · Acknowledgments · License

Awesome Weight-Space Programming is an independent, open-source resource list for researchers, engineers, and anyone exploring neural networks as programmable artifacts. Contributions are welcome from the community.

Weight-space programming explores how an agent can inspect, construct, or modify neural network parameters to implement a specified behavior. This collection connects direct LLM-driven experiments to program compilation, learned weight generation, weight-space representations, and targeted model editing.

Scope. Direct weight reasoning and construction are the focus. Related papers are labeled as foundations; they need not use the term “weight-space programming.” General fine-tuning, prompt optimization, and architecture search without weight construction are outside the main scope.

Curation. Last curated November 3, 2026. This is a selective collection, not an exhaustive survey. Inclusion means a resource is useful to inspect; it does not mean its results have been independently reproduced here. The two originating reports are blog posts, not verified peer-reviewed publications.

Compare five research routes

Research routes

Research route How weights are produced or used Why it matters
Direct LLM programming An LLM reasons about tensors or writes code that assigns them. The central topic of this list.
Program compilation A compiler maps a formal program to a network. Provides explicit mechanisms and useful correctness baselines.
Learned weight generation A trained hypernetwork predicts another network's parameters. A distinct route to generating weights.
Weight-space modeling A model reads or transforms another model's parameters. Addresses symmetries and representations of networks.
Targeted editing An algorithm updates selected weights or combines task vectors. Offers comparison points for behavior modification.

Reproduction checklist

Open the reproduction checklist

When comparing results or proposing a new entry, make the experimental contract visible:

  • Target and architecture: fixed or editable architecture; parameter, buffer, and constant storage; numerical precision.
  • Access: weights only, tensor tools, forward passes, activations, examples, labels, or environment feedback.
  • Construction: direct assignment, compilation, lookup tables, learned generation, or fitted coefficients.
  • Evaluation: unseen seeds or splits, exhaustive finite-domain checks where feasible, boundary cases, and length generalization.
  • Search budget: attempts, model identity, token usage, execution time, and selection procedure.
  • Evidence: runnable artifacts and exact configurations; separate reported scores, independent regrading, checkpoint reruns, and fresh procedure replications.

“No gradients” alone does not imply no fitting, no feedback, or no search. A low parameter count does not imply low total storage when constants are moved into buffers or code. Constructed explanations are valuable hypotheses that still benefit from causal or equivalence checks.

Contributing

New papers, working projects, replications, and corrections are welcome. Please read CONTRIBUTING.md, then open a pull request or suggest a resource.

Prefer primary sources and a short explanation of why the item belongs. Keep direct work separate from related foundations, and label experimental status accurately.

Acknowledgments

Design references & acknowledgments

List structure follows the curation principles of the Awesome manifesto, with organizational inspiration from Awesome Node.js and Awesome AI Agents. The visual introduction also draws on the banner-first layouts of Dify and n8n, and the reading hierarchy of Awesome-LLM. The cover artwork is original to this list. All research and software credit belongs to the linked authors and maintainers. The Awesome badge does not imply inclusion in the central Awesome index.

License

This list's original text is dedicated to the public domain under CC0 1.0. Linked papers, code, models, and datasets retain their respective licenses.

Also known as Awesome-WSP / awesome-wsp.

About

Awesome Weight-Space Programming: curated papers, code, benchmarks, and open experiments for reading, writing, and editing neural network weights.

Topics

Resources

Contributing

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors