Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 3 additions & 5 deletions apps/docs/src/components/QuantFile.astro
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,13 @@
* tensors around them. Names are all-MiniLM-L6-v2's, shortened; the floats
* are F16 as `--f16` stores them. */
const rows = [
{ name: '…word_embeddings.weight', dtype: 'F16', shape: '[30522, 384]', loads: 'float' },
{ name: '…query.weight', dtype: 'I8', shape: '[384, 384]', loads: 'int8 weight', quant: true },
{ name: '…query.weight_scale', dtype: 'F16', shape: '[384, 12]', loads: 'its scales', quant: true },
{ name: '…query.bias', dtype: 'F16', shape: '[384]', loads: 'float' },
{ name: '…LayerNorm.weight', dtype: 'F16', shape: '[384]', loads: 'float' },
];
const rowHeight = 30;
const top = 84;
const top = 66;
const height = top + rows.length * rowHeight + 14;
---

Expand All @@ -23,9 +22,8 @@ const height = top + rows.length * rowHeight + 14;
>
<rect class="file" x="0.5" y="0.5" width="399" height={height - 1} rx="6"></rect>
<text class="header" x="12" y="22">__metadata__ runntime.quant =</text>
<text class="header strong" x="12" y="42">{'{'}"version":1,"bits":8,</text>
<text class="header strong" x="12" y="60">"groupSize":32,"scheme":"symmetric"{'}'}</text>
<line class="rule" x1="0" y1="72" x2="400" y2="72"></line>
<text class="header strong" x="12" y="42">{'{'}"version":1,"scheme":"symmetric"{'}'}</text>
<line class="rule" x1="0" y1="54" x2="400" y2="54"></line>
{
rows.map((row, i) => {
const y = top + i * rowHeight + 12;
Expand Down
45 changes: 32 additions & 13 deletions apps/docs/src/content/docs/core/exporting-weights.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -124,16 +124,31 @@ Full / half-precision files work as is. Optionally, you can quantize your model
**How the runtime recognizes it.** A quantized file has a `runntime.quant`
entry in its header metadata. Each quantized weight is stored as `I8`
(int8) or `U8` (two int4 values per byte), with its scales next to it under
the same name plus `_scale`. The entry can set the bits and group size for
the whole file; without them, each weight takes its bits from its dtype and
the same name plus `_scale`. Each weight takes its bits from its dtype and
its group size from the shape of its scales. Every other tensor
is a normal float tensor. An int tensor in a file without `runntime.quant`
fails to load.

**Which layers are quantized.** `Linear`, `Conv1d` and `ConvTranspose1d`
weights. Embedding tables, biases and norms stay float, because the model
reads them as float. `runntime quantize` only quantizes 2D tensors, so
today it covers Linear weights and leaves conv weights float.
**What is in the file.** `runntime quantize` quantizes every float tensor
named `*.weight` that has 2 or more dims: Linear and conv weights. Biases
and norms stay float, and so do embedding tables and `lm_head`, because
quantizing them usually costs too much quality. It stores each quantized
tensor as rows, `[first dim, everything else]`, so a `Conv1d` weight
`[out, in, k]` becomes `[out, in·k]`, and groups of `--group-size` values
run along that row. The group size has to divide the row: `in` for a
Linear, `in·k` for a conv. A tensor it doesn't divide stays float. The
runtime loads each tensor back into its layer's shape.

**Which layers run quantized.** `Linear`, `Conv1d` and `ConvTranspose1d`
run on the int weights. Any other quantized tensor, like a `Conv2d` weight
or an embedding table you chose to quantize, is turned back into floats at
load. It keeps the rounding of its ints, so the download is smaller but
the model is no faster and uses no less memory than the float one. A
table kept on the CPU takes 4 bytes per value.

A `Linear` or conv weight whose shape the quantized kernels can't take
also loads as float, and a console warning names it. To stay on the
quantized kernels, keep `--group-size` a multiple of 4.

**Making a file.** `runntime quantize` lives in the repository under
`tools/python` and runs with [uv](https://docs.astral.sh/uv/):
Expand All @@ -146,20 +161,24 @@ uv run --no-dev runntime quantize model.safetensors model.int8.safetensors --bit

- `--bits 8` or `--bits 4`. Required.
- `--f16` stores the tensors that stay float as `F16`.
- `--group-size N`: how many inputs share one scale. Default 32.
- `--include` / `--exclude GLOB`: which tensors to quantize. The defaults
quantize `*.weight` and skip embeddings and `lm_head`.
- `--group-size N`: how many values of a row share one scale. Default 32.
- `--include GLOB`: which tensors to quantize. Default `*.weight`.
- `--exclude GLOB`: tensors to keep float, like a layer that loses too
much. Replaces the default `*embed*`, `*pos_emb*` and `lm_head.*`.
- `--drop GLOB`: tensors to leave out of the file, like a pooler the model
never reads.

The tool prints one line per tensor with its error against the float
weights, or why it was skipped.

**Picking a setting.** On all-MiniLM-L6-v2, int8 keeps every embedding at
cosine similarity 0.9998 or higher against the float model; int4 drops to
0.94 and changes the top 3 search results for 8 of 20 queries. Use int8.
Try int4 only when the download size matters more, and check it on your
own data.
cosine similarity 0.9997 or higher against the float model. int4 drops as
low as 0.93 and reorders the top 3 search results for 12 of 20 queries,
where int8 reorders them for 2. Use int8. Try int4 only when the download
size matters more, and check it on your own data.

Use int8 for models built on 2D convolutions. int4 breaks the outputs of
MobileNetV4, DepthART and YOLO26.

A quantization-aware trained checkpoint quantizes the same way if it was
trained on symmetric int8 or int4 with one scale per group of inputs.
1 change: 1 addition & 0 deletions packages/runntime/src/core/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,7 @@ export { foldBnIntoConv } from './weights/foldBn.ts';
export {
concatRows,
dequantize,
inTorchShape,
permuteRows,
reshapeWeight,
scaleWeight,
Expand Down
61 changes: 32 additions & 29 deletions packages/runntime/src/core/nn/layers.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,7 @@ import {
snake,
} from '../graph/ops.ts';
import { matrix, type Value } from '../graph/value.ts';
import {
MATMUL_QUANT_TILED_BK,
matmulQuantTiledEligible,
} from '../kernels/matmul/matmulQuantTiled.ts';
import { matmulQuantTiledEligible } from '../kernels/matmul/matmulQuantTiled.ts';
import {
convToTapMajor,
f16ToF32,
Expand All @@ -35,7 +32,12 @@ import {
} from '../weights/convert.ts';
import { foldBnIntoConv } from '../weights/foldBn.ts';
import { packQuantW } from '../weights/quantPack.ts';
import { derivedTensor, type LazyStateDict, type QuantTensor } from '../weights/safetensors.ts';
import {
derivedTensor,
type LazyStateDict,
type LazyTensor,
type QuantTensor,
} from '../weights/safetensors.ts';
import { dequantize, readFloats, transposeWeight } from '../weights/transforms.ts';
import { tensor } from '../gpu/buffers.ts';
import { CpuParameter, Module, Parameter, type AnyModule } from './module.ts';
Expand Down Expand Up @@ -138,6 +140,14 @@ function loadQuantWeight(
return scale;
}

/** A quantized weight its layer's quantized kernel cannot run, as its
* dequantized f32 values in torch's `shape`, and noted on `sd` for
* loadStateDict's warning. */
function asFloat(sd: LazyStateDict, t: QuantTensor, shape: readonly number[]): LazyTensor {
(sd.quantizedAsFloat ??= []).push(t.name);
return dequantize(t, shape);
}

/** `t`'s scales in f32, as stored. */
async function scalesF32(t: QuantTensor): Promise<Float32Array> {
const data = await t.scales();
Expand All @@ -161,7 +171,8 @@ async function scaleColumns(t: QuantTensor, groups: number, n: number): Promise<
* A quantized checkpoint weight loads as quantW words with its f32 scales in
* `weightScale` [in / groupSize, out], and forward() runs the quantized
* matmul. Only the checkpoint says which kind it is, so transformStateDict
* retypes the slot at load.
* retypes the slot at load. A weight the quantized matmul cannot run loads
* as its dequantized floats.
*
* `addend` and `activation` fold a residual add and a following tanh or GELU
* into the matmul epilogue, as act(x·W + bias) + addend. */
Expand Down Expand Up @@ -202,33 +213,25 @@ export class Linear extends Module {
`expected torch layout [${this.outFeatures}, ${this.inFeatures}]`,
);
}
if (tensor.kind === 'quant') {
this.loadQuantized(sd, prefix, name, tensor);
if (tensor.kind !== 'quant') {
sd.tensors.set(name, transposeWeight(tensor));
return;
}
sd.tensors.set(name, transposeWeight(tensor));
}
/** Loads `name` as quantW words [in, out] with its scales
const { bits, groupSize } = tensor;
// The quantized matmul reads whole tiles of 4 columns and 8 rows, groups
// of whole 4-row chunks, and packs int4 columns 8 to a word.
const fits =
matmulQuantTiledEligible(this.inFeatures, this.outFeatures, groupSize) &&
this.outFeatures % (32 / bits) === 0;
if (fits) this.loadQuantized(sd, prefix, tensor);
else sd.tensors.set(name, transposeWeight(asFloat(sd, tensor, tensor.shape)));
}
/** Loads the weight as quantW words [in, out] with its scales
* [in / groupSize, out]. */
private loadQuantized(
sd: LazyStateDict,
prefix: string,
name: string,
tensor: QuantTensor,
): void {
private loadQuantized(sd: LazyStateDict, prefix: string, tensor: QuantTensor): void {
const { bits, groupSize } = tensor;
const k = this.inFeatures;
const n = this.outFeatures;
// The quantized matmul reads whole tiles of 4 columns and 8 rows and
// packs int4 columns 8 to a word.
if (!matmulQuantTiledEligible(k, n, groupSize) || n % (32 / bits) !== 0) {
throw new RunntimeError(
'CHECKPOINT_MISMATCH',
`Linear: ${name} is int${bits} in groups of ${groupSize}, which needs in features ` +
`a multiple of ${MATMUL_QUANT_TILED_BK} and of the group size, out features a ` +
`multiple of ${32 / bits} and a group size a multiple of 4, got [out, in] = [${n}, ${k}]`,
);
}
this.quantization = { bits, groupSize };
const groups = k / groupSize;
this.weightScale = loadQuantWeight(
Expand Down Expand Up @@ -350,7 +353,7 @@ export class Conv1d extends Module {
this.loadQuantized(sd, prefix, name, tensor, n);
return;
}
tensor = dequantize(tensor, [cOut, cIn, kernel]);
tensor = asFloat(sd, tensor, [cOut, cIn, kernel]);
}
// Without this a weight with equal in/out channels loads scrambled.
if (
Expand Down Expand Up @@ -528,7 +531,7 @@ export class ConvTranspose1d extends Module {
if (cOut % 4 === 0 && n % (32 / w.bits) === 0 && quantConvFits(cIn, taps, n)) {
this.loadQuantized(sd, prefix, w);
} else {
w = dequantize(w, [cIn, cOut, k]);
w = asFloat(sd, w, [cIn, cOut, k]);
}
}
if (w && w.kind !== 'quant') {
Expand Down
33 changes: 25 additions & 8 deletions packages/runntime/src/core/nn/loadStateDict.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,12 @@ import { defaultRoot } from '../gpu/context.ts';
import { inGpuErrorScopes } from '../gpu/errorScopes.ts';
import { materialized, type Value, type ValueMeta } from '../graph/value.ts';
import { bf16ToF16Bits, f32ArrayToF16Bits } from '../weights/convert.ts';
import type { DerivedTensor, FileTensor, LazyStateDict } from '../weights/safetensors.ts';
import {
derivedTensor,
type DerivedTensor,
type FileTensor,
type LazyStateDict,
} from '../weights/safetensors.ts';
import { CpuParameter, type AnyModule } from './module.ts';

export type UploadFn = (data: Float32Array | Uint32Array | Uint16Array, shape: ValueMeta) => Value;
Expand Down Expand Up @@ -120,6 +125,13 @@ async function bindStateDict(
}
};
visit(model, '');
if (sd.quantizedAsFloat?.length) {
console.warn(
'loadStateDict: quantized weights loaded as float, the quantized kernel of their layer ' +
`cannot run their shape or group size: ${sd.quantizedAsFloat.join(', ')}`,
);
sd.quantizedAsFloat = undefined;
}

const params = [...model.namedParameters()];
const paramNames = new Set(params.map(([n]) => n));
Expand All @@ -143,15 +155,20 @@ async function bindStateDict(
const total = sd.totalBytes();
let done = 0;
for (const [name, p] of params) {
const t = sd.tensors.get(name)!;
let t = sd.tensors.get(name)!;
// Linear's and the convs' hooks turn a quantized weight into quantW
// words, so a QuantTensor still here is bound to something else.
// words, so a QuantTensor still here is bound to something no quantized
// kernel reads: an embedding table, a norm, a plain parameter.
if (t.kind === 'quant') {
throw new RunntimeError(
'CHECKPOINT_MISMATCH',
`loadStateDict: '${name}' is quantized (int${t.bits}), but only a Linear, Conv1d or ` +
`ConvTranspose1d weight loads quantized and this parameter holds ${p.shape.dtype}`,
);
if (p.shape.dtype === 'quantW') {
throw new RunntimeError(
'CHECKPOINT_MISMATCH',
`loadStateDict: '${name}' is quantized (int${t.bits}), but this quantW parameter ` +
'takes packed words, which only its layer makes',
);
}
const quant = t;
t = derivedTensor(quant.shape, () => quant.f32(), quant.byteLength);
}
const numel = t.shape.reduce((a, b) => a * b, 1);
// A quantW slot counts u32 words under logical dims. Privacy-filter's
Expand Down
30 changes: 26 additions & 4 deletions packages/runntime/src/core/weights/safetensors.ts
Original file line number Diff line number Diff line change
Expand Up @@ -317,8 +317,9 @@ export interface DerivedTensor extends LazyTensorBase {

/** Symmetric int weights with one scale per `groupSize` values along the last
* dim: value = int · scale of its group. `shape` is logical, one element per
* int even for int4, and `byteLength` covers the ints and the scales. The
* float accessors reject with CHECKPOINT_MISMATCH. */
* int even for int4, and `byteLength` covers the ints and the scales. f32()
* gives the dequantized values, for a reader that only takes floats; the
* readers of stored bits reject with CHECKPOINT_MISMATCH. */
export interface QuantTensor extends LazyTensorBase {
kind: 'quant';
/** What the tensor is, for error messages: its name in the file, or the
Expand All @@ -338,6 +339,10 @@ export type LazyTensor = FileTensor | DerivedTensor | QuantTensor;
export interface LazyStateDict {
metadata: Record<string, string>;
tensors: Map<string, LazyTensor>;
/** Names of the quantized weights a Linear, Conv1d or ConvTranspose1d
* loaded as float because its quantized kernel cannot run them. The
* layers' hooks fill it and loadStateDict warns about them. */
quantizedAsFloat?: string[];
totalBytes(): number;
}

Expand Down Expand Up @@ -410,6 +415,19 @@ function fileTensor(name: string, info: SafeTensorInfo, source: RangeSource): Fi
};
}

/** The values `ints` stand for in f32: each int times the scale of its group
* of `groupSize`. */
export function dequantizeInts(
ints: Int8Array,
scales: Float32Array | Uint16Array,
groupSize: number,
): Float32Array {
const f32 = scales instanceof Float32Array ? scales : f16ToF32(scales);
const out = new Float32Array(ints.length);
for (let i = 0; i < out.length; i++) out[i] = ints[i]! * f32[Math.floor(i / groupSize)]!;
return out;
}

/** A QuantTensor over `ints` and `scales`. Like derivedTensor it holds
* nothing between reads. */
export function quantTensor(
Expand All @@ -422,10 +440,14 @@ export function quantTensor(
Promise.reject(
new RunntimeError(
'CHECKPOINT_MISMATCH',
`tensor ${parts.name} is quantized (int${parts.bits}, groups of ${parts.groupSize}) and has no float reading`,
`tensor ${parts.name} is quantized (int${parts.bits}, groups of ${parts.groupSize}) and stores no float bits`,
),
);
return { kind: 'quant', ...parts, f32: refuse, halfWords: refuse, words: refuse, u16: refuse };
const f32 = async () => {
const [ints, scales] = await Promise.all([parts.ints(), parts.scales()]);
return dequantizeInts(ints, scales, parts.groupSize);
};
return { kind: 'quant', ...parts, f32, halfWords: refuse, words: refuse, u16: refuse };
}

/** Unpacks int4 bytes, two values each with the low nibble first, into one
Expand Down
29 changes: 23 additions & 6 deletions packages/runntime/src/core/weights/transforms.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,17 @@
* A QuantTensor source gives a QuantTensor, except in `dequantize`: the
* helpers move its ints and scales together. Whatever would split a scale
* group (a column split off the group grid, a reshape of the last dim, a
* transpose) throws CHECKPOINT_MISMATCH. */
* transpose) throws CHECKPOINT_MISMATCH; a hook that needs one calls
* `inTorchShape` first. */
import { RunntimeError } from '../error.ts';
import { f16ToF32 } from './convert.ts';
import { derivedTensor, quantTensor, type LazyTensor, type QuantTensor } from './safetensors.ts';
import {
dequantizeInts,
derivedTensor,
quantTensor,
type LazyTensor,
type QuantTensor,
} from './safetensors.ts';

/** f32 values, or f16 halfwords. */
type Floats = Float32Array | Uint16Array;
Expand All @@ -27,12 +34,13 @@ const numel = (shape: readonly number[]): number => shape.reduce((a, b) => a * b
const mismatch = (helper: string, why: string) =>
new RunntimeError('CHECKPOINT_MISMATCH', `${helper}: ${why}`);

/** Reads `t` as f16 halfwords when it holds F16 and as f32 otherwise. Throws
/** Reads `t` as f16 halfwords when it holds F16 and as f32 otherwise, a
* QuantTensor as its dequantized values. Throws
* CHECKPOINT_MISMATCH, naming `helper`, on packed u32 words or on a derived
* tensor whose data does not fill its shape. A file tensor's length is
* checked by the header parser. */
export async function readFloats(helper: string, t: LazyTensor): Promise<Floats> {
if (t.kind === 'quant') throw mismatch(helper, `${t.name} is quantized, not float`);
if (t.kind === 'quant') return t.f32();
if (t.kind === 'file') return t.dtype === 'F16' ? t.u16() : t.f32();
const data = await t.data();
if (data instanceof Uint32Array) throw mismatch(helper, 'source holds packed u32 words');
Expand Down Expand Up @@ -395,9 +403,18 @@ export function dequantize(t: QuantTensor, shape: readonly number[] = t.shape):
[...shape],
async () => {
const [ints, scales] = await Promise.all([t.ints(), t.scales()]);
const f32 = toF32(scales);
return Float32Array.from(ints, (q, i) => q * f32[Math.floor(i / t.groupSize)]!);
return dequantizeInts(ints, scales, t.groupSize);
},
t.byteLength,
);
}

/** `t` in torch's `shape`, for a hook that reshapes, transposes or splits it
* past dim 0. A file stores a quantized tensor as rows, [shape[0], everything
* else], with its scale groups along the row, so those steps would cut
* through the groups: it comes back as its dequantized floats in `shape`.
* Any other tensor comes back as it is. Throws CHECKPOINT_MISMATCH when a
* quantized tensor's element count is not `shape`'s. */
export function inTorchShape(t: LazyTensor, shape: readonly number[]): LazyTensor {
return t.kind === 'quant' ? dequantize(t, shape) : t;
}
Loading
Loading