Skip to content

fix(zoo): run EmbeddingGemma 2's vision and audio towers from a quantized f16 checkpoint - #99

Merged
chmjkb merged 5 commits into
mainfrom
@chmjkb/quant-embeddinggemma2-models
Nov 16, 2026
Merged

chmjkb merged 5 commits into
mainfrom
@chmjkb/quant-embeddinggemma2-models

Conversation

@chmjkb

@chmjkb chmjkb commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Part of #70, phase 4: EmbeddingGemma 2's vision and audio towers run in f16 from int8 and int4 checkpoints.

visionEncoder.ts and audioEncoder.ts picked the input dtype from input_proj.weight. A quantized weight is quantW, so a quantized f16 tower uploaded f32 inputs: vision threw matmul: addend must be f32, audio threw matmul: both inputs must be f32 or both f16. Both now read the dtype from embedding_pre_projection_norm.scale, as the text tower reads norm.scale.

This pull request carries no test. The comparison of a quantized tower against the float model comes with the parity tests (started in #115), so the GPU test that was here is removed.

Measured on the real Hub files, int8 group size 32 from the f16 files (this Mac, f16 math):

tower f16 int8 g32 worst cosine vs f16
text 542.1 MB 413.9 MB (−24%) 0.9997
vision 335.5 MB 193.1 MB (−42%) 0.9999
audio 611.3 MB 336.8 MB (−45%) 0.9999

Short text (12 tokens) is 28% faster on int8; long text, images and audio are 4–6% slower. Same top-1 retrieval as float on 48 queries. int4 text drops to 0.94 cosine on its worst sentence and is not faster, so int4 is not worth publishing.

Commands (from tools/python, #93):

  • text: uv run runntime quantize text.f16.safetensors text.int8.g32.safetensors --bits 8 --group-size 32 --f16 --exclude language_model.embed_tokens.weight
  • vision: ... --bits 8 --group-size 32 --f16 --exclude none (*embed* would skip embed_vision.embedding_projection and patch_embedder.input_proj, which are Linears)
  • audio: ... --bits 8 --group-size 32 --f16 --exclude "*relative_k_proj.weight"

Introduces a breaking change?

  • Yes
  • No

Type of change

  • Bug fix (change which fixes an issue)
  • New feature (change which adds functionality)
  • Documentation update (improves or adds clarity to existing documentation)
  • Performance (kernel tuning, graph building, anything related to performance)
  • Other (chores, tests, code style improvements etc.)

Testing instructions

  1. pnpm typecheck && pnpm lint && pnpm format:check && pnpm --filter runntime test:unit

Related issues

Part of #70.

Checklist

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly
  • My changes generate no new warnings

Additional notes

  • The quantized files are not on the Hub yet; models.ts entries come with the upload.
  • Caveats from QA: batched text with degenerate inputs ("a", "Yes.") has a worst cosine of 0.99915 under int8; image → nearest sentence flipped on 1 of 14 images (a near-tie), caption → image stayed 14/14.

@chmjkb
chmjkb force-pushed the @chmjkb/quant-embeddinggemma2-models branch from a52a5ad to f40b702 Compare November 4, 2026 12:31
Base automatically changed from @chmjkb/quant-small-m to main November 5, 2026 09:11
@chmjkb
chmjkb force-pushed the @chmjkb/quant-embeddinggemma2-models branch from f40b702 to 67c7588 Compare November 5, 2026 11:42
@chmjkb
chmjkb marked this pull request as ready for review November 13, 2026 14:19
@chmjkb
chmjkb added this pull request to stack #119 November 13, 2026 14:19
@chmjkb
chmjkb force-pushed the @chmjkb/quant-embeddinggemma2-models branch from 67c7588 to d4f7e34 Compare November 13, 2026 14:20
@chmjkb chmjkb changed the title fix(zoo): run EmbeddingGemma 2's vision and audio towers from a quantized checkpoint, with a GPU test fix(zoo): run EmbeddingGemma 2's vision and audio towers from a quantized f16 checkpoint Nov 13, 2026
@chmjkb
chmjkb force-pushed the @chmjkb/quant-embeddinggemma2-models branch from fd3e34d to ed331af Compare November 16, 2026 10:11
@chmjkb
chmjkb merged commit 8de7374 into main Nov 16, 2026
2 checks passed
@chmjkb
chmjkb deleted the @chmjkb/quant-embeddinggemma2-models branch November 16, 2026 10:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants