Repository navigation
Conversation
…equantized floats, with one warning per load A quantized Linear weight off the quantized matmul's limits, and a quantized tensor bound to an embedding table, a CpuParameter, a norm or a plain parameter, used to throw CHECKPOINT_MISMATCH. They load as float now, in the model's dtype. QuantTensor.f32() gives the dequantized values, so a hook that reads a tensor as floats works on a quantized one. loadStateDict names every tensor read that way in one console.warn, Conv1d and ConvTranspose1d's float fallback included. A hook's own dequantize() is left out.
…ith no kernel or layer rules runntime quantize knew the quantized matmul: it took 2D tensors only, wanted in features a multiple of 8, out features of 4 or 8 and a group size of 4, and kept embeddings and lm_head float because only Linear loaded quantized. It knows the file format only now. A selected float tensor with 2 or more dims is stored as [first dim, everything else] in groups of --group-size along that row, so a Conv1d [out, in, k] is [out, in·k], the layout the conv loaders read. It stays as stored only when it is not float, has fewer than 2 dims, the group size does not divide the row, or an int4 row is odd. The header leaves out bits and groupSize: the reader takes them from each tensor. No default excludes: int8 embedding tables move an embedding's cosine by under 1e-4 on all-MiniLM-L6-v2 and EmbeddingGemma 2.
… rows, embedding tables too, float fallback at load
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The principle. Exporting weights should be as simple as MLX. Float weights are the PyTorch / Hugging Face checkpoint saved as safetensors. Quantized weights come from
runntime quantize, which writes ints and scales under the checkpoint's own names and knows nothing about kernels or layer types. The runtime fits each weight to its kernels at load, so a file never needs re-exporting because a kernel changed. A slowdown on minor layers is an acceptable price; the hot quantized paths must stay as they are.What changes.
tools/python:runntime quantizequantizes any float tensor with 2 or more dims that matches--include, stored as rows[dim0, prod(rest)]in groups of--group-sizealong the row. The kernel rules and the default excludes are gone.packages/runntime: a quantized tensor that no quantized kernel runs loads as its dequantized floats instead of throwingCHECKPOINT_MISMATCH, and oneconsole.warnper load lists those tensors.core/exporting-weights.mdxand its file figure,tools/python/README.md.What the tool writes (format version 1, as
mainreads it):<name>_scale, F16)runntime.quant = {"version":1,"scheme":"symmetric"}[out, in]I8 [out, in], int4U8 [out, in/2][out, in/g][out, in, k]I8 [out, in·k], int4U8 [out, in·k/2][out, in·k/g][in, out, k]I8 [in, out·k], int4U8 [in, out·k/2][in, out·k/g]A selected tensor stays as stored only when the format can't hold it: not float, fewer than 2 dims, the group size doesn't divide the row, or (int4) the row is odd.
Decisions, why, and what each costs
1. Store every quantized tensor flattened to
[dim0, prod(rest)](chosen over keeping torch N-D shapes under a new format version).mainalready read, and it needs no format, reader or kernel change. Hosted files, Higgs included, keep loading.in·k, notin. An int4 row must be even. A hook that reshapes, transposes or splits an N-D tensor has to dequantize it first (see known issues).2. Kernel rules removed from the tool (
in % 8,out % 4or% 8, group size% 4).3. Header without
bits/groupSize.4. Default excludes removed (
*embed*,*pos_emb*,lm_head.*). Pending the user's decision: keep excluding embeddings until more is measured, or quantize them now.*embed*glob also kept itsembedding_projection.weightfloat by accident; that is a plain Linear.5. Runtime float fallbacks.
QuantTensor.f32()returns the dequantized values instead of throwing, andreadFloatsfollows. The readers of stored bits (u16,halfWords,words) still throw.Linear: a weight off the quantized matmul's limits loads as float in the model's dtype, instead of throwing.Conv1d/ConvTranspose1d: same limits and same fallback as before.loadStateDict: a quantized tensor bound to any float parameter (Embedding on GPU or CPU, a norm, a plainParameter, a Conv2d weight) dequantizes on the CPU. A rawquantWslot still throws.6. The warning. Pending the user's decision: warn on every fallback, or only where a quantized kernel exists and the shape missed it.
console.warnper load,loadStateDict: quantized tensors loaded as float, no quantized kernel runs them: a, b, … (+N), first 8 names plus a count, file names. It is tracked through anonFloatReadcallback onQuantTensorthatconcatRows,splitRowsandsplitColspass on. A hook's owndequantize()is left out, so the hosted Higgs files stay silent.Assumptions
(o, c, tap)of a Conv1d is valuec·k + tapof rowo, which is what the conv loaders read. A unit test loads the tool's golden conv weight intoConv1d(16, 6, 4).createEmbedder: all-MiniLM-L6-v2 and the EmbeddingGemma 2 text tower. No other model was loaded from a fresh export in this PR.Introduces a breaking change?
The public API is unchanged and existing files load as before.
runntime quantizenow quantizes embedding tables and conv weights by default, and its output needs a runtime with #97.Type of change
Testing instructions
pnpm typecheck && pnpm lint && pnpm format:checkpnpm --filter runntime test:unit(1158 pass)pnpm --filter runntime exec vitest run --project gpu test/core test/zoo(1910 pass, 15 expected fail)pnpm --filter docs... run buildtools/python:uv run pytest(50 pass),uv run ruff check,uv run ruff format --check,uv run pyright(0 errors)pnpm --filter runntime bench:ab kernels -b 8de7374 -k matmul,conv: 93 of 93 cases same, max abs diff 0. The sha is the base of this branch; a stale localmainref compares against the wrong commit.New tests: Linear, Conv1d and ConvTranspose1d just inside and just outside each kernel limit with route comments; an Embedding on GPU and CPU, int4 and int8, f32 and f16, matching the dequantized floats; the warning (names listed, silent cases, hook-fused sources, a hook's own dequantize left out); the tool's skip reasons, 3D and odd shapes, the header, the golden files read from both Python and TypeScript.
End to end: the hosted all-MiniLM-L6-v2 f32 file, quantized with this tool at its defaults and loaded through
createEmbedder({ modelPath }), against the float model:--group-size 6Screenshots
Related issues
Part of #70.
Checklist
Additional notes
Known issues from review, to be fixed in this PR before it leaves draft. QA, which runs every hosted model through its runner, is still running.
layer1.conv.weightthrows at load from a fresh default export (stateDictHooks.ts:235).splitCols, which gives silently wrong values under a custom--include;loadStateDict.test.tswas loosened and needs tightening.tools/python/README.mdneeds correcting.draft: true, so the production build skips it; the edited figure was not checked in a browser.https://claude.ai/code/session_019uvkwmTDn1DksCJwKRLHTK