Skip to content

Constant folding: run load-time weight precompute as graph ops on the GPU #118

Open

Description

@chmjkb

Area

Core (engine, ops, kernels)

Problem description

A model that precomputes something from its weights has to do it by hand, on the CPU, in a transformStateDict hook.

The Higgs Audio v2 codec decoder (#102) is the case. For each codebook, the decoder input is embed_k · project_out_kᵀ · fc2ᵀ plus a shared bias, so foldCodebooks in src/zoo/higgs/decoder.ts multiplies these out once at load into a [codebooks · codebookSize, latent] table, and the forward pass gathers rows from it. That fold is about 60 lines of f64 JS loops over derivedTensor.

Doing it by hand costs us in three places:

Proposed solution

A load-time fold step written with the normal graph ops. A module gets a hook that runs once after its weights load. The fold inside it builds a graph from parameters like forward does, the executor runs it on the GPU, and the result is kept as a constant Value.

override fold(): void {
  // embed_k · project_outᵀ · fc2ᵀ for every codebook, once at load
  this.table.bind(cat(books.map((b) => matmul(matmul(b.embed, b.projectOut), fc2)), 0));
}

What it needs:

  1. The hook on Module, called from loadStateDict after the weights are bound.
  2. The folded result kept outside the buffer pool and freed in dispose().
  3. The parameters the fold reads freed after it, when nothing else reads them.
  4. The Higgs decoder moved onto it, with a test that checks the GPU table against today's f64 CPU table.

Quantized weights need nothing extra: the fold runs through the same quantW matmul as forward.

Alternative solutions

  • Keep the CPU fold. It works and folds in f64. The cost is the hand-written loops, the dequantize step and the load time, paid again by each model that folds.
  • Fully automatic folding. The executor marks parameter values as constant and caches any op whose inputs are all constant, with no hook. The graph is rebuilt on every forward call, so it would have to recognize the same op on the same weights across calls (hash op, attrs and input buffers), and decide when cached results are freed. That is more work and harder to get right than an explicit hook, for the same result on this use case.
  • Fold in the Python quantizer and ship the table in the checkpoint. That makes the quantizer model-specific, it writes non-torch layouts, and the folded table gets quantized, which changes accuracy.

Benefits to ruNNtime

  • Load-time precompute written as ordinary ops instead of CPU loops.
  • Quantized checkpoints work in folds with no extra code.
  • Faster loads for models that fold weights.
  • One place for this pattern instead of one per model.

Additional context

Activity

  1. added
    enhancementNew feature or request
    coreEngine: graph ops, dispatch, buffer pool, weight loading
    performanceSpeed or memory improvements
    on Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    coreEngine: graph ops, dispatch, buffer pool, weight loadingenhancementNew feature or requestperformanceSpeed or memory improvements

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions