Do VLMs See or Guess?

Code for the paper Do Vision–Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark.

We hold a 540-image set fixed and generate four question variants of the same scene, then measure how much VLMs rely on textual priors rather than the image. The hardest variant is written from the image alone. A no-image ablation, difficulty/similarity ratings, in-context exemplars, and GRPO post-training complete the study.

Authors: Pratham Singla, Shivank Garg, Vihan Singh, Paras Chopra

Repository layout

dataset_generation/   four question-variant generators
evaluation/           11-model with-image and no-image inference + LLM judge
in_context/           matched-exemplar in-context evaluation
analysis/
  difficulty/         Claude 1–5 difficulty ratings
  similarity/         Claude 1–5 base-to-final textual similarity
post_training/        GRPO training, in-distribution eval, OOD eval

Setup

pip install -r requirements.txt

Local open-weight inference needs CUDA. GRPO training uses a separate conda env:

bash post_training/env_setup.sh

Credentials

Set only the keys for the stages you run. Nothing is hardcoded.

Variable	Used by
OPENROUTER_API_KEY	dataset generation (Question-Guided, Subquestion-Guided), OpenRouter eval, direct similarity
GEMINI_API_KEY_1 … GEMINI_API_KEY_12	Multi-Signal generation (at least one required)
ANANNAS_API_KEY	Vision-Grounded generation
HF_TOKEN	Hugging Face dataset / adapter push and load
VLLM_API_KEY	optional; local vLLM in-context Qwen eval (defaults to EMPTY)

Hugging Face artifacts

Resource	ID
Benchmark (540 images, all variants)	yobro4619/vlm_dataset
Train / test split for GRPO	yobro4619/final_common_train
OOD set (200 samples)	yobro4619/vlm_ood_eval

Images come from the original public benchmarks; we release generated questions, answers, and source identifiers.

1. Dataset generation

Four scripts, one per variant. Each loads a Hugging Face image dataset, calls Gemini-2.5-Pro, and can push the result back to the Hub.

Variant	Script	Backend
Question-Guided	dataset_generation/question_guided.py	OpenRouter
Subquestion-Guided	dataset_generation/subquestion_guided.py	OpenRouter
Multi-Signal	dataset_generation/multi_signal.py	Google Gemini API
Vision-Grounded	dataset_generation/vision_grounded.py	Anannas

Edit the DATASET_NAME / OUTPUT_DATASET_NAME constants at the bottom of the first three scripts, then:

export OPENROUTER_API_KEY=...
python dataset_generation/question_guided.py
python dataset_generation/subquestion_guided.py

export GEMINI_API_KEY_1=...
export HF_TOKEN=...
python dataset_generation/multi_signal.py

python dataset_generation/vision_grounded.py \
  --anannas_api_key "$ANANNAS_API_KEY" \
  --hf_token "$HF_TOKEN" \
  --dataset_name yobro4619/final_common_image_dataset \
  --output_dataset_name yobro4619/direct-difficult-questions

2. Evaluation (with image / no image)

Open-weight models have one script each. Proprietary models and Qwen3.5-397B go through OpenRouter. All five question types are scored. Pass --no_image to withhold the image.

cd evaluation

python eval_qwen3_5_9b.py
python eval_qwen3_5_9b.py --no_image

python eval_internvl3_5_8b.py
python eval_llava_onevision_8b.py
python eval_llava_onevision_4b.py
python eval_llama3_2_11b_vision.py
python eval_phi4_multimodal.py

python eval_openrouter.py --model anthropic/claude-sonnet-4.6
python eval_openrouter.py --model google/gemini-2.5-pro
python eval_openrouter.py --model google/gemini-3.1-flash-lite-preview
python eval_openrouter.py --model openai/gpt-5-mini
python eval_openrouter.py --model qwen/qwen3.5-397b-a17b

Then judge with Qwen3-14B and aggregate:

python llm_judge.py --directory results/ --exclude-evaluated
python accuracy_summary.py --directory results/

Optional flags: --max_samples N, --output_dir PATH.

3. In-context evaluation

Same models, but each prompt is prepended with a matched exemplar (base question, single-pass sub-questions, or chat sub-questions) for the same image. Vision-Grounded is excluded here because it has no source-text exemplar.

cd in_context

python eval_qwen3_5_9b_vllm.py          # start a local vLLM server first
python eval_internvl3_5_8b.py
python eval_llava_onevision_8b.py
python eval_llava_onevision_4b.py
python eval_llama3_2_11b_vision.py
python eval_phi4_multimodal.py

python eval_openrouter_concurrent.py --model anthropic/claude-sonnet-4.6

Restrict scenarios with --scenarios base,subquestions,new_subquestions. Then:

python llm_judge.py --directory results/
python accuracy_summary.py

4. Difficulty and similarity

Claude rates question difficulty (1–5) and how similar each generated question is to the base question (1–5; lower = more distinct).

# Bedrock (AWS profile `default`, or pass --profile)
python analysis/difficulty/claude_difficulty.py
python analysis/difficulty/claude_difficulty_analysis.py

python analysis/similarity/similarity_analysis.py
python analysis/similarity/subquestions_similarity_analysis.py
python analysis/similarity/new_subquestions_similarity_analysis.py

# OpenRouter
python analysis/similarity/direct_similarity_analysis.py
python analysis/similarity/calculate_mean_scores.py

Put the similarity JSON files in analysis/similarity/results/ (or set SIMILARITY_RESULTS_DIR) before running the aggregator.

5. GRPO post-training and OOD

One LoRA adapter per variant on Qwen3.5-4B (vision tower frozen). Training data: 440-image train split. In-distribution eval: 100-sample held-out test split. OOD: 200 provenance-clean questions.

bash post_training/env_setup.sh
conda activate vlm_rl_grpo
cd post_training

python grpo_train.py --variant base      # also: subques, chat, direct
python grpo_train.py --variant direct --push_to_hub

bash run_base_evals.sh --hf_token "$HF_TOKEN"
bash run_grpo_evals.sh --hf_token "$HF_TOKEN"
python llm_judge.py --directory eval_results/

cd ood_eval
python build_ood_dataset.py --hf_token "$HF_TOKEN"
python verify_ood_dataset.py
python push_ood_dataset.py --hub_id yobro4619/vlm_ood_eval

bash run_ood_evals_a.sh --hf_token "$HF_TOKEN"   # base model
bash run_ood_evals_b.sh --hf_token "$HF_TOKEN"   # base + subques adapters
bash run_ood_evals_c.sh --hf_token "$HF_TOKEN"   # chat + direct adapters

Judge OOD JSONs with post_training/llm_judge.py pointed at ood_eval/results/.

Citation

@article{singla2026vision,
  title={Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark},
  author={Singla, Pratham and Garg, Shivank and Singh, Vihan and Chopra, Paras},
  journal={arXiv preprint arXiv:2606.10400},
  year={2026}
}