Are sub-billion-parameter vision–language models a reliable foundation for document understanding — and where do they break?
A systematic, reproducible evaluation of six open-source sub-1B VLMs across five document benchmarks, followed by an evidence-driven knowledge-gap analysis and a targeted improvement study on the best model. Built on VLMEvalKit as a pinned dependency.
🌐 Project page: forall-labs.github.io/MiniVLMDocEval · 📝 Blog post: Your Sign to Build on Sub-1B VLMs (Medium) · 🐦 On X: thread
📄 Technical report: technical_report/report.pdf ·
🧪 Reproduce: Colab notebook or scripts/reproduce.sh ·
🧭 Design & decisions: research_plan.md · 📋 Brief: task.md
🤗 Dataset: savoji/minivlm-tablevqa-sft · 🤗 LoRA adapter: savoji/qwen3.5-0.8b-tablevqa-lora (the Part-2 experimental adapter — published for transparency; it failed the gate, see below)
📂 Full result artifacts (predictions, per-pair scores, logs, summary tables): Google Drive
Six generalist VLMs (0.45–0.9B params) under a matched protocol — greedy decoding, uniform 16-bit precision, a 128-token generation cap, and fixed N=1000 seed-42 subsets. Scores are each benchmark's native metric, normalized to 0–100.
| Rank | Model | Params | ChartQA | DocVQA | InfoVQA | OCRBench | TableVQA | Mean |
|---|---|---|---|---|---|---|---|---|
| 1 | Qwen3.5-0.8B | 0.8B | 70.8 | 89.3 | 62.3 | 79.2 | 50.1 | 70.3 |
| 2 | InternVL3-1B | 0.9B | 68.8 | 80.8 | 54.2 | 79.4 | 33.5 | 63.3 |
| 3 | LFM2.5-VL-450M | 0.45B | 73.1 | 77.2 | 41.3 | 67.8 | 40.2 | 59.9 |
| 4 | LLaVA-OneVision-0.5B | 0.9B | 60.0 | 70.4 | 39.0 | 60.2 | 34.4 | 52.8 |
| 5 | SmolVLM2-500M | 0.5B | 60.4 | 67.8 | 27.4 | 61.0 | 30.3 | 49.4 |
| 6 | FastVLM-0.5B | 0.8B | 44.3 | 63.6 | 32.5 | 26.2 | 17.5 | 36.8 |
Findings. (1) Qwen3.5-0.8B is the strongest foundation (mean 70.3, wins 3/5). (2) Architecture beats parameter count — the smallest model (0.45B LFM2.5-VL) outranks both 0.9B models and wins ChartQA. (3) The image-resolution strategy — not scale — is the dominant driver of both accuracy and latency. (4) Visual table QA is the frontier: no model exceeds 50.1.
The gap, localized. Decomposing TableVQA into its sub-domains shows the best model's weakness is Wikipedia-style visual lookup (VWTQ 27.8 / VWTQ-Syn 33.1), not financial or numeric tables (FinTabNetQA 84.0) — correcting a common assumption in the table-QA literature.
Part 2 (improvement, reported faithfully). A targeted LoRA fine-tune to close the lookup gap failed a pre-registered no-regression gate (catastrophic forgetting: TableVQA 50.1→19.6, DocVQA 89.3→50.8). The diagnosis is a train/eval answer-style mismatch (verbose training answers vs. short-answer scoring); the report gives a concrete corrected recipe. The gate behaving as designed — rejecting a degraded model — is itself the intended outcome of the protocol. See report §8.
Six models, two incompatible transformers versions → reproduction runs in two phases
(a runtime restart between them), then a light figures phase. Results persist under a single
OUT_DIR and merge into one comparison table.
- Env A (
transformers<4.57): SmolVLM2, InternVL3, LLaVA-OneVision, FastVLM - Env B (
transformers@main): Qwen3.5-0.8B, LFM2.5-VL-450M, and the Part-2 study
Open notebooks/colab.ipynb and run top to bottom: Phase A →
restart runtime (banner in the notebook) → Phase B → Phase C. Point outputs at Google
Drive so they persist. T4 suffices for Phase A; prefer A100/L4 for Phase B.
git clone https://github.com/forall-labs/MiniVLMDocEval.git && cd MiniVLMDocEval
OUT_DIR=outputs bash scripts/reproduce.sh env-a # Phase A
# --> start a FRESH shell / environment (new transformers) <--
OUT_DIR=outputs bash scripts/reproduce.sh env-b # Phase B (+ Part 2)
OUT_DIR=outputs bash scripts/reproduce.sh figures # report plotsOutputs land in outputs/: summary/comparison.{md,csv} (the table), predictions/<model>/…
(per-pair scores), and logs/. Everything is seeded and pinned; a re-run reproduces the numbers.
Reproducibility knobs. Fixed seed (42), fixed N (1000), greedy decoding, pinned VLMEvalKit commit, per-model 16-bit precision, and pinned training deps. All configurable in
minivlmdoceval/config.py.
- Evaluation engine: VLMEvalKit at a pinned
commit (cloned by
setup.sh, never forked). Adding a model = onegenerate_innermethod. - Runner:
scripts/run_eval.py— a single config-driven entry point scores any registered model on a fixed-seed N-sample subset, with per-pair caching/resume, and refreshes the comparison table after each pair. - Model registry:
minivlmdoceval/custom_models.py— we contribute custom wrappers that add three models the harness lacks natively (FastVLM-0.5B, Qwen3.5-0.8B, LFM2.5-VL-450M) plus the fine-tunedQwen3.5-0.8B-TableLoRA. - Benchmarks (a diagnostic ladder): OCRBench → DocVQA → ChartQA → InfoVQA → TableVQABench (recognition → extraction → numeric → layout → tabular).
setup.sh Env bootstrap: clone VLMEvalKit @ pinned commit; Env A / --bleeding-edge (Env B) / --train
scripts/
reproduce.sh One-command reproduction driver (env-a | env-b | figures)
run_eval.py Config-driven eval runner (scores any registered model)
smoke_test.py Fast end-to-end sanity check
efficiency_ablation.py fp16 + generation-cap ablations
tablevqa_subdomain_report.py TableVQA per-sub-domain breakdown (the gap diagnosis)
build_table_sft.py Part-2 training mixture (contamination-guarded)
train_lora_qwen.py Part-2 bf16 LoRA fine-tune (+ --smoke de-risk)
regression_gate.py Pre-registered no-regression gate
export_examples.py Qualitative examples for the report
make_report_figures.py Figures: Pareto, per-benchmark, sub-domains
make_examples_figure.py Figure: qualitative-example gallery
push_dataset_hf.py Publish the training set to the HuggingFace Hub
minivlmdoceval/
config.py Models, benchmarks, N, seed, output tree
custom_models.py Custom VLMEvalKit wrappers + LoRA variant
notebooks/colab.ipynb Reproducible two-phase runner
technical_report/ NeurIPS-style report (report.tex → report.pdf) + figures + references.bib
research_plan.md Strategy, model/benchmark/metric rationale, and Part-2 results
task.md The assignment brief
external/VLMEvalKit/ The engine — cloned by setup.sh at a pinned commit (gitignored)
setup.sh targets Linux/Colab and requires a CUDA GPU (VLMEvalKit's wrappers are
CUDA-bound). It clones VLMEvalKit at a pinned commit and installs it editable. Flags:
--bleeding-edge (Env B, transformers@main) and --train (adds peft/trl/accelerate
for Part 2). The mac is for development only; the eval runs on GPU.
Author: Sajjad Pakdamansavoji · sj.pakdaman.edu@gmail.com