Object Commitment Verification (OCV)

OCV is a training-free, post-decoding framework for reducing object hallucination in vision-language model captions. It treats generated object mentions as commitments, verifies whether each commitment is supported, and applies bounded edits only to high-risk unsupported mentions.

This repository is the public research package for OCV. It contains the method interfaces, example configurations, lightweight evaluation wrappers, and sanitized aggregate results. It is not a complete end-to-end benchmark reproduction package.

For Quick Review

• Method overview: the OCV pipeline, evidence families, and bounded-edit policy.
• Main public results: audited aggregate rows with provenance and caveats.
• Results overview: interpretation of released, pending, and intentionally excluded artifacts.
• Limitations: scope boundaries and explicit non-claims.
• Qwen2.5 boundary results: an additional cross-model setting with both positive and mixed outcomes.

Method In 30 Seconds

generated caption
    -> extract object commitments
    -> collect available evidence signals
    -> fuse commitment-level risk
    -> preserve or bounded-edit each commitment
    -> corrected caption

OCV runs after caption generation. It does not retrain the base model, update model parameters, or modify the decoder. The released aggregate rows cover Regular and FLB source captions; compatibility with another decoder is an interface property, not a claim that every decoder combination was benchmarked.

Terminology

• Regular: original caption generation without post-decoding intervention.
• FLB: first-logit baseline, a decoding-stage baseline used in the source experimental workspace.
• OCV: Object Commitment Verification, the bounded post-decoding verifier.
• CHAIRs / Hal / OpenCHAIR: hallucination-oriented metrics where lower values are better.

Best Supported Public Result

On COCO/CHAIR-500, InstructBLIP-7B CHAIRs decreases from 0.2360 under the first-logit decoding baseline to 0.1340 with OCV-on-FLB, a 43.2% relative reduction.

• Lower is better for CHAIRs.
• This is an aggregate public result migrated from the sanitized public experimental record.
• This is not a SOTA claim or an official benchmark reproduction claim.
• Raw generations and datasets are not redistributed.

See the main result CSV, the results overview, and the source headline audit.

Selected Released Results

Lower is better for every metric below. Values and displayed reductions are migrated without recomputation from the audited source record.

Model	Benchmark	Metric	FLB	OCV on FLB	Reduction vs FLB	Boundary
InstructBLIP-7B	COCO/CHAIR-500	CHAIRs	0.2360	0.1340	0.1020 (43.2%)	Strongest supported headline; OCV on Regular selected zero surviving edits in this cell.
LLaVA-1.5-7B	COCO/CHAIR-500	CHAIRs	0.1560	0.1100	0.0460 (29.5%)	Aggregate caption-side row.
LLaVA-1.5-7B	AMBER-1004	Hal	31.6	25.0	6.6 (20.9%)	Cover decreases from 50.1 to 47.8.
InstructBLIP-7B	AMBER-1004	Hal	52.5	48.4	4.1 (7.8%)	Cover decreases from 54.5 to 52.5.
LLaVA-1.5-7B	OpenCHAIR	OpenCHAIR	0.162093	0.158447	0.003646 (2.2%)	Small gain; local offline evaluator.

The Qwen2.5-VL-7B-Instruct rows are reported separately as boundary evidence, not as a headline.

Benchmarks

Released caption-side aggregate result families:

• COCO/CHAIR-500
• AMBER-1004
• OpenCHAIR

The repository also includes lightweight interfaces for FOIL-COCO and M-HalDetect mechanism evaluation. Their public CSV rows remain schema-only or pending and are not presented as released numeric results.

Repository Layout

• src/ocv/: dependency-light OCV interfaces for commitments, signals, risk scoring, and bounded editing
• configs/: user-editable example configurations with placeholder paths
• scripts/: validation wrappers and compact table utilities
• docs/: method, setup, reproduction, result, and limitation notes
• results/: released aggregate summaries and clearly marked pending schemas
• tests/: lightweight API checks for the public package

Model weights, datasets, raw generations, private manuscript files, and large evaluator outputs are not included.

Install

git clone https://github.com/Seren666/OCV.git
cd OCV
python -m venv .venv
python -m pip install -U pip
python -m pip install -e .

For a minimal syntax and API check:

PYTHONPATH=src python -m compileall src scripts
PYTHONPATH=src python tests/test_core.py

On Windows PowerShell:

$env:PYTHONPATH = "src"
python -m compileall src scripts
python .\tests\test_core.py

Reproduction Boundary

Edit the configs under configs/ so /path/to/... values point to local datasets, model files, source captions, and output directories.

python scripts/run_coco_chair.py --config configs/llava15_7b_coco.yaml
python scripts/run_amber.py --config configs/instructblip_7b_coco.yaml
python scripts/run_openchair.py --config configs/internvl35_8b_coco.yaml

The wrappers validate configuration structure and emit manifests describing required inputs and expected outputs. They do not download external resources or run a complete large-scale experiment by default. See reproduction.md, data_preparation.md, and model_setup.md.

To render compact Markdown tables from the public summary CSV files:

python scripts/make_tables.py --output results/paper_tables.md

Scope And Non-Claims

OCV is presented here as a research method with released aggregate evidence, not as SOTA, an accepted or submitted paper, an official benchmark reproduction, or a complete end-to-end release. See docs/limitations.md and the public release scope for the full boundary statement.