We're currently experimenting with this methodology — testing whether low-bit quantized vision-language models can match higher-bit models on a narrow classification task once their noisy output is repeatedly sampled and normalized. This repo is a working lab notebook, not a finished tool or a settled conclusion.
Can a 1-3 bit quantized Qwen2.5-VL-3B-Instruct, after repeated sampling and post-hoc normalization, match a 6-bit quantized version on a simple color-naming task? If so, a much smaller/cheaper model could be usable in practice wherever this kind of squeeze-and-normalize post-processing is acceptable.
- Split each color's images into calibration/eval sets
- Build a separate normalization dictionary per quantization level —
one model's phrasing never leaks into another model's scoring
(
build_mapping.py) - For each eval image, sample repeatedly starting at the lowest
temperature; if no new normalized answer appears, raise the
temperature and keep going ("squeezing") —
run_squeeze.py - Normalize every raw response into a canonical color /
EMPTY(model produced nothing — collapse) /UNK(non-empty but unrecognized), then compute hit-rate, confusion matrices, and category breakdowns (analyze.py)
See SRS.md for the full design rationale (Korean).
| Model | Raw hit-rate | Majority-vote hit-rate | Canonical rate | UNK rate |
|---|---|---|---|---|
| IQ1_M | 12.3% | 16.7% | 13.0% | 86.8% |
| IQ2_M | 79.0% | 80.0% | 96.3% | 3.7% |
| IQ3_M | 94.8% | 100.0% | 99.7% | 0.3% |
| Q4_K_M | 80.8% | 80.0% | 98.5% | 1.5% |
| Q5_K_M | 81.8% | 80.0% | 99.5% | 0.5% |
| Q6_K | 82.5% | 80.0% | 99.8% | 0.2% |
Notable
- IQ1_M (~1-bit) scores below random chance for 5 classes (20%) — its vocabulary space has collapsed and no amount of normalization recovers it
- There's a sharp jump between IQ2_M and IQ3_M — this looks like the boundary where the model stops collapsing
- IQ3_M through Q6_K cluster closely together (80-95%)
Limitations: n=30, single run, synthetic HSV solid-color swatches only (256×256). Not yet verified on real photos. Purple showed a consistent red/blue confusion across every quantization level (including Q6_K) — it's unclear whether this generalizes or is specific to the Hue (275°) chosen for this pilot.
pip install llama-cpp-python pillow
cp experiment_config.example.json experiment_config.json
# edit model_dir in experiment_config.json to point at your local GGUF files
python generate_synthetic_dataset.py # or place your own images under dataset/<Color>/
python build_mapping.py # build per-model normalization dictionaries (interactive)
python run_squeeze.py # adaptive sampling across all configured quant levels
python analyze.py # writes results/report.mdRequires Qwen2.5-VL-3B-Instruct GGUF files at whatever bit-widths you want to compare, plus a shared vision projector (mmproj, F16 recommended). Imatrix-calibrated quants (e.g. from mradermacher) are recommended.
flag_suspicious_mapping.py: flags normalization-dictionary entries that look like a different color's name landed in the wrong bucket (e.g. "blue" registered as a Purple synonym)remove_suspicious_entries.py: removes reviewed/confirmed bad entries (backs up the dictionary first)
MIT — see LICENSE.