Skip to content

feat(core): run Conv1d and ConvTranspose1d on quantized weights - #97

Merged
xxpopielxx merged 4 commits into
mainfrom
feat/quant-conv
Nov 12, 2026
Merged

xxpopielxx merged 4 commits into
mainfrom
feat/quant-conv

Conversation

@xxpopielxx

Copy link
Copy Markdown
Member

Description

Quantized Conv1d and ConvTranspose1d, on top of #96.

  • runntime.quant version 2: each weight takes bits from its dtype and groupSize from its scales' shape, so one file can mix them. Version 1 loads as before.
  • matmulQuantTiled takes a loader for a and a scale period, so conv1d runs on it with one scale per group of input channels in every tap.
  • conv1d takes a quantW weight. Conv1d and ConvTranspose1d load quantized weights; Linear shares their loader.

Introduces a breaking change?

  • Yes
  • No

Type of change

  • Bug fix (change which fixes an issue)
  • New feature (change which adds functionality)
  • Documentation update (improves or adds clarity to existing documentation)
  • Performance (kernel tuning, graph building, anything related to performance)
  • Other (chores, tests, code style improvements etc.)

Testing instructions

  • pnpm --filter runntime test: new GPU cases for quantW conv1d and for Conv1d / ConvTranspose1d loaded from a quantized file against the dequantized float layer, plus unit tests for the reader, op and layer loaders.
  • pnpm bench:ab kernels -b @chmjkb/quant-small-m -k matmul,conv: all 79 existing cases same, max abs diff 0. New quantW conv cases skip on the base.

Screenshots

Related issues

#96

Checklist

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly
  • My changes generate no new warnings

Additional notes

Version 2 changes #96's file format, so it is its own commit. Int8 convs run at about float speed (compute-bound); they save weight memory.

@xxpopielxx xxpopielxx changed the title Feat/quant conv feat(core): run Conv1d and ConvTranspose1d on quantized weights Nov 3, 2026
@xxpopielxx xxpopielxx added enhancement New feature or request core Engine: graph ops, dispatch, buffer pool, weight loading kernels GPU kernels and kernel routes labels Nov 3, 2026

@chmjkb chmjkb left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rebase please 🙏🏻 🥺

Base automatically changed from @chmjkb/quant-small-m to main November 5, 2026 09:11
…t version 2

A version 2 file sets bits by dtype and groupSize by scale shape per weight,
so one file can mix them. Version 1 loads as before.
conv1d takes a quantW weight and runs on matmulQuantTiled, which gains
the loader slot and a scale period so a group of input channels shares
a scale in every tap. Linear, Conv1d and ConvTranspose1d share the loader.
…ional bits and groupSize

A file without bits or groupSize takes them from each tensor's dtype and scale shape. A file
that sets them is checked against every tensor, as before.

@chmjkb chmjkb left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With this setup, the quantizer has to know about convs. It needs to know the kernel size to pick the group size. It also needs to swap the axes of ConvTranspose1d weights before quantizing.

The reason is that the kernel only reads scales in one fixed order. You can't regroup the scales after quantizing, so the file has to match the kernel.

Could we flip this around? The kernel could let each layer say where its scales are, like loadA does for the input. Then the quantizer wouldn't care about layers at all, and the files would stay in torch layout like the rest of runntime.

It might make the kernel a bit slower, so it needs a benchmark.

…s them

matmulQuantTiled takes a scale loader next to loadA. Conv1d and ConvTranspose1d load
quantized weights as torch's [out, in·kernel] and [in, out·kernel], one scale per group
of values along each row, so a quantizer needs no layer knowledge.
@xxpopielxx
xxpopielxx added this pull request to stack #114 November 12, 2026 13:21
@xxpopielxx
xxpopielxx merged commit 0110ca4 into main Nov 12, 2026
2 checks passed
@xxpopielxx
xxpopielxx deleted the feat/quant-conv branch November 12, 2026 13:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Engine: graph ops, dispatch, buffer pool, weight loading enhancement New feature or request kernels GPU kernels and kernel routes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants