Skip to content

test(zoo): load MiniLM from int8 and int4 checkpoints and check it against the dequantized and float weights - #82

Merged
chmjkb merged 16 commits into
mainfrom
@chmjkb/quant-minilm-e2e
Nov 5, 2026
Merged

chmjkb merged 16 commits into
mainfrom
@chmjkb/quant-minilm-e2e

Conversation

@chmjkb

@chmjkb chmjkb commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Phase 2c of #70: MiniLM loads and runs from int8 and int4 checkpoints with no source change.

  • Test helper: quantizeWeight(values, [out, in], bits, groupSize) does symmetric per-group quantization (max abs / 127 or 7, scale rounded to f16). quantFile takes optional float entries so one file holds quantized and F32 tensors.
  • test/zoo/minilm/model.gpu.test.ts: a MiniLM checkpoint in HF names with the real widths (384 hidden, 12 heads), 2 layers, seeded random weights. It loads through fromSafetensors → loadStateDict (real hooks: q/k scale, q|k|v concatRows) → createMinilmEmbedCore, on the replay and batch paths. 8 cases: f32/f16 × int8/int4 × groupSize 32/128. Each checks that every Linear got a weightScale, that the output matches the same ints dequantized and loaded as F32 (main assertion), and the cosine to the float weights.

Introduces a breaking change?

  • Yes
  • No

Type of change

  • Bug fix (change which fixes an issue)
  • New feature (change which adds functionality)
  • Documentation update (improves or adds clarity to existing documentation)
  • Performance (kernel tuning, graph building, anything related to performance)
  • Other (chores, tests, code style improvements etc.)

Testing instructions

  1. pnpm typecheck && pnpm lint && pnpm --filter runntime test:unit (1032 pass)
  2. pnpm --filter runntime test:gpu test/zoo/minilm/model.gpu.test.ts (8 pass). Measured worst case vs dequantized: 6e-7 max abs on f32, one f16 step on f16. Cosine vs float: 0.999993 int8, 0.998 int4, on the test's random weights. On the real MiniLM weights int4 drops to 0.94; that and the user docs are in docs(docs): add the Exporting weights guide, from a checkpoint to a float or quantized file #98.
  3. Skipping the q/k scale on the quantized path, or swapping the weightScale transpose, fails 8 of 8.

Related issues

Part of #70. Stacked on #81.

Checklist

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly
  • My changes generate no new warnings

Additional notes

Speed, measured locally (6 layers, f16): quantized is about 20% slower on a 16-token sentence, 7–8% slower on a batch of 32, and level or up to 6% faster at 256 tokens. Weights shrink from 22.5 MB to 12–13 MB (int8) and 7–8 MB (int4). The short-input loss is because the plain quantW kernel has no bias epilogue, so every biased quantized Linear takes the 64-row tiled kernel. A few-row quantized kernel with bias is a separate follow-up.

@chmjkb
chmjkb force-pushed the @chmjkb/quant-minilm-e2e branch from 7ace4b0 to f510084 Compare October 28, 2026 14:00
@chmjkb
chmjkb force-pushed the @chmjkb/quant-linear branch from 1ac6b2c to ea235c4 Compare October 30, 2026 11:26
@chmjkb
chmjkb force-pushed the @chmjkb/quant-minilm-e2e branch from f510084 to 9513176 Compare October 30, 2026 11:26
@xxpopielxx xxpopielxx mentioned this pull request Oct 30, 2026
Closed
@chmjkb
chmjkb force-pushed the @chmjkb/quant-linear branch from ea235c4 to 1fa57a3 Compare October 30, 2026 12:47
@chmjkb
chmjkb force-pushed the @chmjkb/quant-minilm-e2e branch from 9513176 to 83bbd9e Compare October 30, 2026 12:47
@chmjkb
chmjkb force-pushed the @chmjkb/quant-linear branch from 1fa57a3 to 2bf66d1 Compare October 30, 2026 13:26
@chmjkb
chmjkb force-pushed the @chmjkb/quant-minilm-e2e branch from 83bbd9e to 8306f44 Compare October 30, 2026 13:26
@chmjkb
chmjkb marked this pull request as ready for review November 2, 2026 09:01
@chmjkb
chmjkb force-pushed the @chmjkb/quant-minilm-e2e branch from 8306f44 to d7432dd Compare November 2, 2026 09:11
@chmjkb
chmjkb force-pushed the @chmjkb/quant-minilm-e2e branch from d7432dd to 8d1853d Compare November 2, 2026 17:32
@chmjkb
chmjkb force-pushed the @chmjkb/quant-linear branch from 09f1ce5 to 1c61170 Compare November 2, 2026 17:32
@chmjkb
chmjkb force-pushed the @chmjkb/quant-linear branch from 1c61170 to a8a599f Compare November 4, 2026 09:26
Base automatically changed from @chmjkb/quant-linear to main November 4, 2026 09:27
@chmjkb
chmjkb force-pushed the @chmjkb/quant-minilm-e2e branch from 8d1853d to 506da9c Compare November 4, 2026 09:32
@chmjkb chmjkb changed the title test(zoo): load MiniLM from int8 and int4 checkpoints and document the quantized weight layout test(zoo): load MiniLM from int8 and int4 checkpoints and check it against the dequantized and float weights Nov 4, 2026
@chmjkb
chmjkb merged commit 598c24f into main Nov 5, 2026
2 checks passed
@chmjkb
chmjkb deleted the @chmjkb/quant-minilm-e2e branch November 5, 2026 09:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants