Repository navigation
Description
Area
Core (engine, ops, kernels)
Problem description
A model that precomputes something from its weights has to do it by hand, on the CPU, in a transformStateDict hook.
The Higgs Audio v2 codec decoder (#102) is the case. For each codebook, the decoder input is embed_k · project_out_kᵀ · fc2ᵀ plus a shared bias, so foldCodebooks in src/zoo/higgs/decoder.ts multiplies these out once at load into a [codebooks · codebookSize, latent] table, and the forward pass gathers rows from it. That fold is about 60 lines of f64 JS loops over derivedTensor.
Doing it by hand costs us in three places:
- Quantized weights. The fold needs float values, so a quantized
project_outorfc2has to be dequantized first. feat(zoo): load the Higgs Audio v2 codec decoder from int8 and int4 checkpoints #102 addedweightValuesfor this, and feat(zoo): load the Higgs Audio v2 codec encoder from int8 and int4 checkpoints #104 moved it onto the shareddequantize()helper. Every hook that does math on weights has to remember this. - Load time. The fold is a CPU triple loop, run on every load.
- Repetition. Each model that folds weights writes its own loops for math the GPU ops already do:
matmulalready runs on f32, f16 and quantW weights.
Proposed solution
A load-time fold step written with the normal graph ops. A module gets a hook that runs once after its weights load. The fold inside it builds a graph from parameters like forward does, the executor runs it on the GPU, and the result is kept as a constant Value.
override fold(): void {
// embed_k · project_outᵀ · fc2ᵀ for every codebook, once at load
this.table.bind(cat(books.map((b) => matmul(matmul(b.embed, b.projectOut), fc2)), 0));
}What it needs:
- The hook on
Module, called fromloadStateDictafter the weights are bound. - The folded result kept outside the buffer pool and freed in
dispose(). - The parameters the fold reads freed after it, when nothing else reads them.
- The Higgs decoder moved onto it, with a test that checks the GPU table against today's f64 CPU table.
Quantized weights need nothing extra: the fold runs through the same quantW matmul as forward.
Alternative solutions
- Keep the CPU fold. It works and folds in f64. The cost is the hand-written loops, the dequantize step and the load time, paid again by each model that folds.
- Fully automatic folding. The executor marks parameter values as constant and caches any op whose inputs are all constant, with no hook. The graph is rebuilt on every
forwardcall, so it would have to recognize the same op on the same weights across calls (hash op, attrs and input buffers), and decide when cached results are freed. That is more work and harder to get right than an explicit hook, for the same result on this use case. - Fold in the Python quantizer and ship the table in the checkpoint. That makes the quantizer model-specific, it writes non-torch layouts, and the folded table gets quantized, which changes accuracy.
Benefits to ruNNtime
- Load-time precompute written as ordinary ops instead of CPU loops.
- Quantized checkpoints work in folds with no extra code.
- Faster loads for models that fold weights.
- One place for this pattern instead of one per model.
Additional context
- Use case:
foldCodebooksinpackages/runntime/src/zoo/higgs/decoder.ts(feat(zoo): load the Higgs Audio v2 codec decoder from int8 and int4 checkpoints #102), dequantize step moved ontodequantize()in feat(zoo): load the Higgs Audio v2 codec encoder from int8 and int4 checkpoints #104. - Precision drops from f64 (CPU) to f32 (GPU). The test in step 4 should show whether that matters for the codebook table.
- Out of scope: the decoder's
conv_t1hook inBlock.transformStateDict. It reorders a weight's layout and computes nothing, so folding does not apply.
Activity