Skip to content

docs(docs): add the Exporting weights guide, from a checkpoint to a float or quantized file - #98

Merged
chmjkb merged 14 commits into
mainfrom
@chmjkb/docs-exporting-weights
Nov 13, 2026
Merged

chmjkb merged 14 commits into
mainfrom
@chmjkb/docs-exporting-weights

Conversation

@chmjkb

@chmjkb chmjkb commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator

Description

Part of #70: a new docs page, Zoo → Guides → Exporting weights, on how to get your own weights into runntime.

  • The weights file: one safetensors file with the names the model's loader expects, how a runner gets it (modelPath, spreading a models config).
  • How a file loads: header and name check before any download, 8 MB range chunks, browser cache keyed by the URL, file dtype vs the model's dtype.
  • Exporting float weights from a Hugging Face checkpoint, and checking a file (CHECKPOINT_MISMATCH and what it reports).
  • Quantizing with runntime quantize (feat(tools): add runntime quantize, a uv Python CLI that writes int8/int4 checkpoints in format v1 #93): command, defaults, --f16, --drop, --include/--exclude, reading the report, measured MiniLM file sizes.
  • The quantized format and shape rules, now in one place; the Embedding page's quantized section keeps only MiniLM's own rule (q/k/v together) and links here.
  • Picking the setting: int8 in groups of 32 is the safe pick; int4 costs quality on the real MiniLM weights (worst cosine 0.94, top-3 changes on 8 of 20 queries).
  • QAT: a float QAT checkpoint quantizes like any other; the tool's exact rule; no zero points or int8 activations.
  • Three diagrams as Astro components on the site's theme tokens: the path from checkpoint to load, what a quantized file holds, how one group of 32 weights is quantized.

Introduces a breaking change?

  • Yes
  • No

Type of change

  • Bug fix (change which fixes an issue)
  • New feature (change which adds functionality)
  • Documentation update (improves or adds clarity to existing documentation)
  • Performance (kernel tuning, graph building, anything related to performance)
  • Other (chores, tests, code style improvements etc.)

Testing instructions

  1. pnpm typecheck && pnpm lint && pnpm format:check
  2. pnpm --filter docs... run build, then pnpm dev:docs and open /zoo/exporting-weights/ in light and dark theme, desktop and phone width.
  3. Every bash and Python snippet on the page was run as written; the file sizes and report lines come from quantizing the hosted MiniLM f32 file.

Related issues

Part of #70. Stacked on #93.

Checklist

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly
  • My changes generate no new warnings

Additional notes

  • Only all-MiniLM-L6-v2 loads a quantized file on main so far; the page says so in one place, to extend as Moonshine and EmbeddingGemma 2 land.
  • No speed numbers on the page: the benchmarks page gets measured ones separately.

@chmjkb
chmjkb force-pushed the @chmjkb/python-quantizer branch from 5e92e44 to 3c4e31a Compare November 4, 2026 10:16
@chmjkb
chmjkb force-pushed the @chmjkb/docs-exporting-weights branch from 6215422 to b8dc1e2 Compare November 4, 2026 12:31
@chmjkb
chmjkb force-pushed the @chmjkb/python-quantizer branch from d284954 to 0aaf88d Compare November 5, 2026 11:42
@chmjkb
chmjkb force-pushed the @chmjkb/docs-exporting-weights branch from b8dc1e2 to e8146f1 Compare November 5, 2026 11:42
Base automatically changed from @chmjkb/python-quantizer to main November 6, 2026 15:54
@chmjkb
chmjkb force-pushed the @chmjkb/docs-exporting-weights branch from e8146f1 to b511130 Compare November 10, 2026 13:59
@chmjkb
chmjkb force-pushed the @chmjkb/docs-exporting-weights branch from b511130 to a4ca278 Compare November 12, 2026 14:33
@chmjkb
chmjkb marked this pull request as ready for review November 12, 2026 14:34
of `F32` and doesn't change how the model computes.

If a weight is missing or has the wrong shape, loading stops with a
`CHECKPOINT_MISMATCH` error that names it (see

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe also say that an extra tensor in the file fails the same way (unexpected: pooler.dense.weight)? A plain HF checkpoint hits this. --drop could be listed with the other flags, since it removes them when quantizing.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

once #101 lands unexpected tensors no longer throw unless strict: true is passed, but I'll add a small mention

@chmjkb
chmjkb merged commit 43053ee into main Nov 13, 2026
2 checks passed
@chmjkb
chmjkb deleted the @chmjkb/docs-exporting-weights branch November 13, 2026 15:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants