Skip to content

feat(tools): quantize any float weight in checkpoint layout, and load what no quantized kernel runs as float - #123

Draft
chmjkb wants to merge 3 commits into
mainfrom
@chmjkb/quantizer-kernel-agnostic
Draft

chmjkb wants to merge 3 commits into
mainfrom
@chmjkb/quantizer-kernel-agnostic

Conversation

@chmjkb

@chmjkb chmjkb commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Description

The principle. Exporting weights should be as simple as MLX. Float weights are the PyTorch / Hugging Face checkpoint saved as safetensors. Quantized weights come from runntime quantize, which writes ints and scales under the checkpoint's own names and knows nothing about kernels or layer types. The runtime fits each weight to its kernels at load, so a file never needs re-exporting because a kernel changed. A slowdown on minor layers is an acceptable price; the hot quantized paths must stay as they are.

What changes.

  • tools/python: runntime quantize quantizes any float tensor with 2 or more dims that matches --include, stored as rows [dim0, prod(rest)] in groups of --group-size along the row. The kernel rules and the default excludes are gone.
  • packages/runntime: a quantized tensor that no quantized kernel runs loads as its dequantized floats instead of throwing CHECKPOINT_MISMATCH, and one console.warn per load lists those tensors.
  • Docs: core/exporting-weights.mdx and its file figure, tools/python/README.md.

What the tool writes (format version 1, as main reads it):

ints scales (<name>_scale, F16)
header runntime.quant = {"version":1,"scheme":"symmetric"}
Linear [out, in] I8 [out, in], int4 U8 [out, in/2] [out, in/g]
Conv1d [out, in, k] I8 [out, in·k], int4 U8 [out, in·k/2] [out, in·k/g]
ConvTranspose1d [in, out, k] I8 [in, out·k], int4 U8 [in, out·k/2] [in, out·k/g]

A selected tensor stays as stored only when the format can't hold it: not float, fewer than 2 dims, the group size doesn't divide the row, or (int4) the row is odd.

Decisions, why, and what each costs

1. Store every quantized tensor flattened to [dim0, prod(rest)] (chosen over keeping torch N-D shapes under a new format version).

  • Why: it is one rule with no layer knowledge, it is exactly what the reader and the Conv1d / ConvTranspose1d loaders on main already read, and it needs no format, reader or kernel change. Hosted files, Higgs included, keep loading.
  • Cost: the file holds the flattened shape, not torch's. A conv's group size must divide in·k, not in. An int4 row must be even. A hook that reshapes, transposes or splits an N-D tensor has to dequantize it first (see known issues).

2. Kernel rules removed from the tool (in % 8, out % 4 or % 8, group size % 4).

  • Why: they describe today's quantized matmul, not the file format.
  • Cost: a layer the kernel can't take used to stay float in the file. Now it is quantized in the file and dequantized at load, so that layer uses float memory and takes a small load-time cost.

3. Header without bits / groupSize.

4. Default excludes removed (*embed*, *pos_emb*, lm_head.*). Pending the user's decision: keep excluding embeddings until more is measured, or quantize them now.

  • Why: they existed because the runtime could not load a quantized lookup table. It can now, and int8 costs little. Worst cosine against the float model over 20 texts, int8, group size 32:
model tables float tables int8 drop
all-MiniLM-L6-v2 0.999736 0.999719 1.7e-5
EmbeddingGemma 2 text 0.999723 0.999688 3.5e-5
  • The EmbeddingGemma 2 text file goes from 414 MB to 288 MB. The old *embed* glob also kept its embedding_projection.weight float by accident; that is a plain Linear.
  • Cost: an embedding table saves download size only, because it is dequantized at load. Not measured: int4 beyond one MiniLM run (worst cosine 0.932 to 0.925 on 8 texts, which is over the 1e-3 bar), Moonshine's tied embedding and output head, privacy-filter.

5. Runtime float fallbacks.

  • QuantTensor.f32() returns the dequantized values instead of throwing, and readFloats follows. The readers of stored bits (u16, halfWords, words) still throw.
  • Linear: a weight off the quantized matmul's limits loads as float in the model's dtype, instead of throwing.
  • Conv1d / ConvTranspose1d: same limits and same fallback as before.
  • loadStateDict: a quantized tensor bound to any float parameter (Embedding on GPU or CPU, a norm, a plain Parameter, a Conv2d weight) dequantizes on the CPU. A raw quantW slot still throws.
  • Cost: float memory for those tensors. A CPU table that would have kept raw BF16 bits becomes f32.

6. The warning. Pending the user's decision: warn on every fallback, or only where a quantized kernel exists and the shape missed it.

  • Now: one console.warn per load, loadStateDict: quantized tensors loaded as float, no quantized kernel runs them: a, b, … (+N), first 8 names plus a count, file names. It is tracked through an onFloatRead callback on QuantTensor that concatRows, splitRows and splitCols pass on. A hook's own dequantize() is left out, so the hosted Higgs files stay silent.
  • Cost, raised in review: with embedding tables quantized by default, it fires on almost every model made with the tool's defaults.

Assumptions

  • Flattening is row-major, so value (o, c, tap) of a Conv1d is value c·k + tap of row o, which is what the conv loaders read. A unit test loads the tool's golden conv weight into Conv1d(16, 6, 4).
  • The generic bind checks element count only, so a flattened tensor fills a parameter of another shape with the same size.
  • The quantized conv scale readers take any group size that divides the row.
  • Run end to end through createEmbedder: all-MiniLM-L6-v2 and the EmbeddingGemma 2 text tower. No other model was loaded from a fresh export in this PR.

Introduces a breaking change?

  • Yes
  • No

The public API is unchanged and existing files load as before. runntime quantize now quantizes embedding tables and conv weights by default, and its output needs a runtime with #97.

Type of change

  • Bug fix (change which fixes an issue)
  • New feature (change which adds functionality)
  • Documentation update (improves or adds clarity to existing documentation)
  • Performance (kernel tuning, graph building, anything related to performance)
  • Other (chores, tests, code style improvements etc.)

Testing instructions

  1. pnpm typecheck && pnpm lint && pnpm format:check
  2. pnpm --filter runntime test:unit (1158 pass)
  3. pnpm --filter runntime exec vitest run --project gpu test/core test/zoo (1910 pass, 15 expected fail)
  4. pnpm --filter docs... run build
  5. In tools/python: uv run pytest (50 pass), uv run ruff check, uv run ruff format --check, uv run pyright (0 errors)
  6. pnpm --filter runntime bench:ab kernels -b 8de7374 -k matmul,conv: 93 of 93 cases same, max abs diff 0. The sha is the base of this branch; a stale local main ref compares against the wrong commit.

New tests: Linear, Conv1d and ConvTranspose1d just inside and just outside each kernel limit with route comments; an Embedding on GPU and CPU, int4 and int8, f32 and f16, matching the dequantized floats; the warning (names listed, silent cases, hook-fused sources, a hook's own dequantize left out); the tool's skip reasons, 3D and odd shapes, the header, the golden files read from both Python and TypeScript.

End to end: the hosted all-MiniLM-L6-v2 f32 file, quantized with this tool at its defaults and loaded through createEmbedder({ modelPath }), against the float model:

file worst cosine loaded as float
int8 0.999719 the 3 embedding tables
int8, --group-size 6 0.999887 every Linear and the tables, all in the warning
int4 0.925 the 3 embedding tables

Screenshots

Related issues

Part of #70.

Checklist

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly
  • My changes generate no new warnings

Additional notes

Known issues from review, to be fixed in this PR before it leaves draft. QA, which runs every hosted model through its runner, is still running.

  • EmbeddingGemma 2 audio: layer1.conv.weight throws at load from a fresh default export (stateDictHooks.ts:235).
  • Hooks that reshape, transpose or column-split an N-D tensor do not handle its flattened quantized form:
    • privacy-filter experts go through splitCols, which gives silently wrong values under a custom --include;
    • EmbeddingGemma 2's depthwise transpose;
    • EmbeddingGemma 2's position table;
    • the Higgs encoder's shape check.
  • The int4 numbers in the docs' "Picking a setting" are stale: they were measured with float embedding tables.
  • An assertion in loadStateDict.test.ts was loosened and needs tightening.
  • The docs sentence about memory is wrong for BF16 CPU tables.
  • The cosine wording in tools/python/README.md needs correcting.
  • The docs page is draft: true, so the production build skips it; the edited figure was not checked in a browser.

https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK

…equantized floats, with one warning per load

A quantized Linear weight off the quantized matmul's limits, and a quantized
tensor bound to an embedding table, a CpuParameter, a norm or a plain
parameter, used to throw CHECKPOINT_MISMATCH. They load as float now, in the
model's dtype.

QuantTensor.f32() gives the dequantized values, so a hook that reads a tensor
as floats works on a quantized one. loadStateDict names every tensor read that
way in one console.warn, Conv1d and ConvTranspose1d's float fallback included.
A hook's own dequantize() is left out.
…ith no kernel or layer rules

runntime quantize knew the quantized matmul: it took 2D tensors only, wanted
in features a multiple of 8, out features of 4 or 8 and a group size of 4, and
kept embeddings and lm_head float because only Linear loaded quantized.

It knows the file format only now. A selected float tensor with 2 or more dims
is stored as [first dim, everything else] in groups of --group-size along that
row, so a Conv1d [out, in, k] is [out, in·k], the layout the conv loaders read.
It stays as stored only when it is not float, has fewer than 2 dims, the group
size does not divide the row, or an int4 row is odd.

The header leaves out bits and groupSize: the reader takes them from each
tensor. No default excludes: int8 embedding tables move an embedding's cosine
by under 1e-4 on all-MiniLM-L6-v2 and EmbeddingGemma 2.
… rows, embedding tables too, float fallback at load

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant