Repository navigation
test(zoo): load MiniLM from int8 and int4 checkpoints and check it against the dequantized and float weights - #82
Merged
Merged
Conversation
chmjkb
force-pushed
the
@chmjkb/quant-minilm-e2e
branch
from
October 28, 2026 14:00
7ace4b0 to
f510084
Compare
7 of 11 tasks
chmjkb
force-pushed
the
@chmjkb/quant-linear
branch
from
October 30, 2026 11:26
1ac6b2c to
ea235c4
Compare
chmjkb
force-pushed
the
@chmjkb/quant-minilm-e2e
branch
from
October 30, 2026 11:26
f510084 to
9513176
Compare
Closed
chmjkb
force-pushed
the
@chmjkb/quant-linear
branch
from
October 30, 2026 12:47
ea235c4 to
1fa57a3
Compare
chmjkb
force-pushed
the
@chmjkb/quant-minilm-e2e
branch
from
October 30, 2026 12:47
9513176 to
83bbd9e
Compare
chmjkb
force-pushed
the
@chmjkb/quant-linear
branch
from
October 30, 2026 13:26
1fa57a3 to
2bf66d1
Compare
chmjkb
force-pushed
the
@chmjkb/quant-minilm-e2e
branch
from
October 30, 2026 13:26
83bbd9e to
8306f44
Compare
chmjkb
marked this pull request as ready for review
November 2, 2026 09:01
chmjkb
force-pushed
the
@chmjkb/quant-minilm-e2e
branch
from
November 2, 2026 09:11
8306f44 to
d7432dd
Compare
5 of 11 tasks
chmjkb
force-pushed
the
@chmjkb/quant-minilm-e2e
branch
from
November 2, 2026 17:32
d7432dd to
8d1853d
Compare
chmjkb
force-pushed
the
@chmjkb/quant-linear
branch
from
November 2, 2026 17:32
09f1ce5 to
1c61170
Compare
xxpopielxx
reviewed
Nov 3, 2026
…kQuantW, replacing packQuantColSlice
…words and f32 weightScale
… bias and activation in the epilogue
…fuse a quantized tensor outside a Linear weight
…ights on the plain and tiled routes
…, in] order, without a transposed host copy
…tized weights, with addend cases
chmjkb
force-pushed
the
@chmjkb/quant-linear
branch
from
November 4, 2026 09:26
1c61170 to
a8a599f
Compare
…the quantized checkpoint helper
…against the dequantized and float weights
…ogether or not at all
…64 and 128 as examples
chmjkb
force-pushed
the
@chmjkb/quant-minilm-e2e
branch
from
November 4, 2026 09:32
8d1853d to
506da9c
Compare
xxpopielxx
approved these changes
Nov 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Phase 2c of #70: MiniLM loads and runs from int8 and int4 checkpoints with no source change.
quantizeWeight(values, [out, in], bits, groupSize)does symmetric per-group quantization (max abs / 127 or 7, scale rounded to f16).quantFiletakes optional float entries so one file holds quantized and F32 tensors.test/zoo/minilm/model.gpu.test.ts: a MiniLM checkpoint in HF names with the real widths (384 hidden, 12 heads), 2 layers, seeded random weights. It loads throughfromSafetensors→loadStateDict(real hooks: q/k scale, q|k|vconcatRows) →createMinilmEmbedCore, on the replay and batch paths. 8 cases: f32/f16 × int8/int4 × groupSize 32/128. Each checks that everyLineargot aweightScale, that the output matches the same ints dequantized and loaded as F32 (main assertion), and the cosine to the float weights.Introduces a breaking change?
Type of change
Testing instructions
pnpm typecheck && pnpm lint && pnpm --filter runntime test:unit(1032 pass)pnpm --filter runntime test:gpu test/zoo/minilm/model.gpu.test.ts(8 pass). Measured worst case vs dequantized: 6e-7 max abs on f32, one f16 step on f16. Cosine vs float: 0.999993 int8, 0.998 int4, on the test's random weights. On the real MiniLM weights int4 drops to 0.94; that and the user docs are in docs(docs): add the Exporting weights guide, from a checkpoint to a float or quantized file #98.weightScaletranspose, fails 8 of 8.Related issues
Part of #70. Stacked on #81.
Checklist
Additional notes
Speed, measured locally (6 layers, f16): quantized is about 20% slower on a 16-token sentence, 7–8% slower on a batch of 32, and level or up to 6% faster at 256 tokens. Weights shrink from 22.5 MB to 12–13 MB (int8) and 7–8 MB (int4). The short-input loss is because the plain quantW kernel has no bias epilogue, so every biased quantized
Lineartakes the 64-row tiled kernel. A few-row quantized kernel with bias is a separate follow-up.