Repository navigation
fix(zoo): run EmbeddingGemma 2's vision and audio towers from a quantized f16 checkpoint - #99
Merged
Merged
Conversation
chmjkb
force-pushed
the
@chmjkb/quant-embeddinggemma2-models
branch
from
November 4, 2026 12:31
a52a5ad to
f40b702
Compare
chmjkb
force-pushed
the
@chmjkb/quant-embeddinggemma2-models
branch
from
November 5, 2026 11:42
f40b702 to
67c7588
Compare
6 of 11 tasks
chmjkb
marked this pull request as ready for review
November 13, 2026 14:19
chmjkb
force-pushed
the
@chmjkb/quant-embeddinggemma2-models
branch
from
November 13, 2026 14:20
67c7588 to
d4f7e34
Compare
xxpopielxx
approved these changes
Nov 16, 2026
…Gemma 2 encoders, so a quantized tower runs f16
…an int8 and int4 checkpoint on the GPU
…o feed-forward for tight bounds and short cases
…Gemma 2 test's K_NORM_SCALE comment
… the parity tests take over quantized against float
chmjkb
force-pushed
the
@chmjkb/quant-embeddinggemma2-models
branch
from
November 16, 2026 10:11
fd3e34d to
ed331af
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Part of #70, phase 4: EmbeddingGemma 2's vision and audio towers run in f16 from int8 and int4 checkpoints.
visionEncoder.tsandaudioEncoder.tspicked the input dtype frominput_proj.weight. A quantized weight isquantW, so a quantized f16 tower uploaded f32 inputs: vision threwmatmul: addend must be f32, audio threwmatmul: both inputs must be f32 or both f16. Both now read the dtype fromembedding_pre_projection_norm.scale, as the text tower readsnorm.scale.This pull request carries no test. The comparison of a quantized tower against the float model comes with the parity tests (started in #115), so the GPU test that was here is removed.
Measured on the real Hub files, int8 group size 32 from the f16 files (this Mac, f16 math):
Short text (12 tokens) is 28% faster on int8; long text, images and audio are 4–6% slower. Same top-1 retrieval as float on 48 queries. int4 text drops to 0.94 cosine on its worst sentence and is not faster, so int4 is not worth publishing.
Commands (from
tools/python, #93):uv run runntime quantize text.f16.safetensors text.int8.g32.safetensors --bits 8 --group-size 32 --f16 --exclude language_model.embed_tokens.weight... --bits 8 --group-size 32 --f16 --exclude none(*embed*would skipembed_vision.embedding_projectionandpatch_embedder.input_proj, which are Linears)... --bits 8 --group-size 32 --f16 --exclude "*relative_k_proj.weight"Introduces a breaking change?
Type of change
Testing instructions
pnpm typecheck && pnpm lint && pnpm format:check && pnpm --filter runntime test:unitRelated issues
Part of #70.
Checklist
Additional notes
models.tsentries come with the upload.