Develop VLM detection and training

README.zh-CN.md

@@ -118,7 +118,7 @@ libreyolo predict --model yolo9-t --source screen            # 屏幕捕获
 | **视频嵌入** | V-JEPA 2（片段级嵌入，另有可训练注意力探针的视频分类） |
 | **人体网格** | SAM 3D Body |
 | **图像复原** | DDColor、HVI-CIDNet、LaMa、NAFNet、QuickSRNet、Real-ESRGAN、SwinIR |
-| **背景移除** | BiRefNet、FeyNobg、ViTMatte |
+| **背景移除** | BiRefNet、FeyNobg、BEN2、ViTMatte |
 | **OCR** | PP-OCR |
 | **点检测** | FOMO、LocateAnything |
 | **视线估计** | L2CS |

THIRD_PARTY_NOTICES.txt

@@ -351,6 +351,7 @@ learned parameters unchanged). See libreyolo/models/pe/NOTICE.md.
 YOLOX (Megvii-BaseDetection)
 --------------------------------------------------------------------
 Source: https://github.com/Megvii-BaseDetection/YOLOX
+Commit: 6d30efe8f9f277e8788fb2d9b17d08238a5ea292
 License: Apache License 2.0
 Copyright (c) 2021-2022 Megvii Inc. All rights reserved.
 Used for: YOLOX model family (libreyolo/models/yolox/), EMA helper
@@ -359,7 +360,12 @@ Used for: YOLOX model family (libreyolo/models/yolox/), EMA helper
           adapted for the YOLOv7 family (libreyolo/models/yolo7/loss.py:
           imports bboxes_iou/IoULoss from the yolox modules and adapts the
           get_assignments/get_geometry_constraint/simota_matching/get_losses
-          assignment logic to the v7 anchor head).
+          assignment logic to the v7 anchor head). The shared COCO bounding-box
+          clipping helper (libreyolo/utils/coco_geometry.py) is used by the COCO
+          dataset loader (libreyolo/data/dataset.py), native COCO VLM training
+          dataset (libreyolo/models/vlm/training/data.py), and VLM confidence
+          gate writer/report reconstruction (libreyolo/validation/
+          vlm_confidence_validator.py and vlm_confidence_report.py).
 
 --------------------------------------------------------------------
 YOLO (MultimediaTechLab/YOLO)
@@ -1921,6 +1927,31 @@ Used for: the Transformers-backed LibreEdgeTAM adapter in
 
 The full Apache License 2.0 text is included at licenses/Apache-2.0.txt.
 
+--------------------------------------------------------------------
+Qwen3-VL fine-tuning recipe (QwenLM)
+--------------------------------------------------------------------
+Source: https://github.com/QwenLM/Qwen3-VL
+Commit: 96588727e44c78b25ba03ea03b8e12f7e64fd0da
+Source paths:
+  qwen-vl-finetune/qwenvl/train/argument.py
+  qwen-vl-finetune/qwenvl/train/train_qwen.py
+  qwen-vl-finetune/README.md
+License: Apache License 2.0 (see licenses/Apache-2.0.txt)
+Copyright 2023 Rohan Taori, Ishaan Gulrajani, Tianyi Zhang,
+               Yann Dubois, Xuechen Li (retained by train_qwen.py, which
+               identifies FastChat and Stanford Alpaca as earlier sources)
+Used for: the Qwen3-VL recipe reference behind
+          libreyolo/models/vlm/training/recipes.py,
+          libreyolo/models/vlm/training/trainer.py, and the trainability
+          declaration in libreyolo/models/vlm/qwen3vl.py: PEFT LoRA on
+          language-model projections while the vision tower remains frozen,
+          plus the 0.03 warmup-ratio and cosine-schedule reference. LibreYOLO
+          adds its own MLP projection scope and detection-specific pipeline,
+          and uses different rank, alpha, dropout, learning-rate, and
+          weight-decay values. No upstream source file is bundled.
+
+The full Apache License 2.0 text is included at licenses/Apache-2.0.txt.
+
 --------------------------------------------------------------------
 SenseNova-Vision / Bagel (SenseTime; ByteDance; Hugging Face; BFL)
 --------------------------------------------------------------------

docs/adr/0002-librevlm-contract.md

@@ -1,7 +1,7 @@
 # ADR 0002: LibreVLM Contract For Vision-Language Detectors
 
 - Status: Accepted
-- Date: 2026-06-05 (updated 2026-06-06)
+- Date: 2026-06-05 (updated 2026-08-16)
 - Scope: New model tier (vision-language models used as open-vocab detectors)
 
 ## Context
@@ -75,6 +75,49 @@ text = model.chat("image.jpg", "How many cars are pink?")  # raw escape hatch
   from a fixed task / grounding token plus the class list, so `prompt=` is ignored
   for those two.
 
+The supported VLM CLI surface is intentionally smaller than the detector CLI:
+
+```bash
+libreyolo predict --model qwen3-vl-4b --source image.jpg \
+  --names '["pink car", "wheel"]'
+libreyolo train --model qwen3-vl-2b --data dataset.yaml
+```
+
+- `predict` resolves VLM aliases, schema-valid checkpoint directories, and
+  immutable `hf+vlm://owner/repo@<commit>` artifacts. `--names` is a VLM-only
+  JSON vocabulary; `--classes` remains a numeric output filter. Explicit
+  `imgsz=` is rejected because the family processor owns image resizing.
+  Directory chunks still generate one image at a time.
+- `train` is verified only for the Qwen3-VL 2B and 4B base aliases. It uses
+  VLM-native defaults rather than detector defaults, and selects `best` by
+  validation loss when a validation split exists, otherwise training loss.
+  Continue an adapter with the base alias plus `resume=<checkpoint directory>`;
+  training an inference-loaded checkpoint wrapper is rejected.
+- Detector-only training options, unsupported families/sizes, and standalone
+  `val`, `export`, or `quantize` requests fail before loading VLM weights.
+
+### Publication artifacts
+
+Local training checkpoints and published artifacts are different contracts.
+The v1 publication builder accepts only Qwen3-VL 2B/4B detection LoRA output
+written with `peft==0.19.1` and `transformers==5.12.1`. It requires external,
+human-approved evidence that binds the adapter, checkpoint contract, processor,
+complete immutable base snapshot, training-data manifest, evaluation report,
+an exact-zero two-run repeatability receipt, clean code revision, and fixed
+recipe. Publication evidence v2 binds the primary and repeat run identities,
+the raw receipt, and its canonical comparison. The library can generate an
+unapproved template with derived hashes, but it cannot manufacture an
+approval.
+
+The artifact includes the adapter and exact Qwen processor, tokenizer, and
+chat-template assets under Apache-2.0. Base weights remain reference-only.
+`push_vlm_artifact()` refuses existing repositories, starts private, creates
+one commit, verifies that immutable tree through a fresh download, and returns
+`hf+vlm://owner/repo@<40-character-commit>`. The detector `hf://` transport and
+generic Hub logger are not used. Hashes are integrity bindings, not signatures
+or proof that human claims are true. See
+[`../vlm_hub_artifact.md`](../vlm_hub_artifact.md).
+
 ## Internal Contract
 
 `LibreVLMModel(BaseModel)` is the shared base. It does NOT define `can_load`, so
@@ -103,8 +146,10 @@ shared `InferenceRunner` drives:
 - `_preprocess(image, ...)` builds the chat-template inputs from the image plus
   the detection prompt; returns `(inputs, pil_image, (W, H), ratio=1.0)`. Boxes
   come back normalized to the image, so there is no letterbox/unpad math.
-- `_forward(inputs)` runs `model.generate(...)` greedily and returns only the
-  newly generated tokens.
+- `_forward(inputs)` runs `model.generate(...)` greedily and returns the newly
+  generated tokens. A family-gated scoring path can also attach one
+  selected-token log-probability per step without retaining vocabulary-sized
+  score tensors.
 - `_postprocess(output, conf, ...)` decodes, tolerantly parses the JSON, scales
   the coordinates per `BBOX_KEY`/`COORD_DIVISOR`, and returns the standard
   detection dict `{boxes, scores, classes, num_detections}` that
@@ -126,25 +171,56 @@ in [`../librevlm_design.md`](../librevlm_design.md).
 
 ## Confidence
 
-Generated detections carry no calibrated per-box score. The tier assigns a
-constant placeholder (`DEFAULT_SCORE = 1.0`), so `predict`/draw/`track` behave
-normally and `conf=` filtering still functions mechanically. Consequences:
-
-- `conf=` thresholds and ranking are soft, not calibrated.
-- `track()` runs, but because every box is scored 1.0, ByteTrack's two-stage,
-  score-stratified association is inert (no separate low-confidence recovery
-  stage and `new_track_thresh` never bites) until a real score lands.
-- `val()` (mAP) is intentionally unsupported; it would be misleading.
-
-`_score_detections(items)` is the documented override point for a real signal
-(decoder token log-probs or self-consistency) in a later iteration.
+Generated detections carry no calibrated per-box score. The generic VLM families
+currently assign a constant placeholder (`DEFAULT_SCORE = 1.0`). A bounded-memory
+candidate for Qwen3-VL can derive a ranking signal from the geometric mean of
+generated label-token and coordinate-token probabilities. It records one
+selected-token log-probability after the configured generation processors per
+step, rather than retaining a vocabulary-sized score tensor for every token.
+The candidate remains disabled until its real-data gate passes, so ordinary
+`predict()` keeps the established constant-score behavior. Candidate scores are
+available only to the internal validator, not public prediction, the CLI, or
+integrations. LibreMODUS separately uses the minimum constrained-token
+probability for each detection.
+
+`model.confidence_method` reports the configured source (`constant` today for
+Qwen3-VL, and `constrained_token_min` for LibreMODUS).
+
+Consequences:
+
+- On constant-score families, `conf=` filtering is mechanical and ByteTrack's
+  score-stratified association remains inert (no separate low-confidence
+  recovery stage).
+- `val()` (mAP) remains unsupported until the candidate score orders correct
+  detections better than the constant baseline, behaves safely with the public
+  confidence threshold, and is reproducible. Unit tests establish plumbing, not
+  score quality.
+- Candidate-only Brier error, fixed-bin ECE, and reliability bins are diagnostic
+  outputs of the internal gate, not a calibration claim. Missing scores never
+  inherit the constant fallback in those statistics; fallback safety and score
+  coverage are reported separately.
+- Persisted gate reports use schema v2 and can be compared across processes only
+  after strict reconstruction validates their hashes and duplicated metrics.
+  Timing and plot artifacts do not define reproducibility. The hashes provide
+  consistency checks, not report authentication.
+- A canonical repeatability receipt uses two distinct run and process
+  identifiers and binds both report/envelope byte identities, zero tolerances,
+  and the complete comparison. Publication treats it as structural integrity
+  evidence, not publisher or reviewer authentication or proof that reports are
+  truthful.
+
+`_score_detections(items)` remains the scalar fallback for custom generation
+paths. Scored greedy generations use the additive per-item scoring path.
 
 ## Licensing
 
-LibreYOLO ships only its own VLM adapter code: families either load through the
-Apache-2.0 `transformers` API or, when a model genuinely requires Hugging Face
-remote code, download that upstream model-repository code at runtime under the
-upstream model repo's terms. LibreYOLO does not redistribute VLM weights.
+Family aliases either load through the Apache-2.0 `transformers` API or, when a
+model genuinely requires Hugging Face remote code, download that upstream
+repository at runtime under its terms. LibreYOLO does not mirror those base
+weights or remote-code repositories. The strict Qwen LoRA artifact is the
+narrow exception for non-weight assets: it redistributes the exact processor,
+tokenizer, and chat-template files under Apache-2.0 while retaining the base
+weights as an immutable external reference.
 
 The default model (Qwen3-VL-4B) is Apache-2.0, so it needs no notice. When a
 model's weights or required model-repository code are under a non-permissive
@@ -160,15 +236,15 @@ executing mutable upstream model-repository code under the same alias.
 
 ## Out Of Scope (v1)
 
-- Training / fine-tuning (`train()` raises; fine-tune upstream). Superseded
-  for Qwen3-VL: LoRA detection fine-tuning shipped later behind the same
-  `train()` surface, with untrainable families keeping documented refusal
-  messages. See `docs/vlm_training.md`.
-- Dataset validation / mAP (`val()` raises; see "Confidence").
-- Export to ONNX/TensorRT/etc. (`export()` raises; generative decode).
-- CLI: the `libreyolo` command does not resolve VLM aliases in v1. The tier is a
-  Python-API surface (`LibreVLM(...)`); `predict`/`track` parity is at the API
-  level, not the CLI.
+- Training outside the verified Qwen3-VL 2B/4B detection cohort. Supported
+  fine-tuning is documented in [`../vlm_training.md`](../vlm_training.md).
+- Dataset validation / mAP (`val()` and the CLI `val` command reject VLMs; see
+  "Confidence").
+- Export to ONNX/TensorRT/etc. and post-training quantization. The CLI `export`
+  and `quantize` commands reject VLM references before loading weights.
+- Broad CLI parity. The documented VLM CLI contract is prediction plus the
+  verified Qwen training cohort; tier-specific methods such as `chat()` remain
+  Python API surfaces.
 
 ## Consequences
 
@@ -180,7 +256,8 @@ executing mutable upstream model-repository code under the same alias.
 
 ### Negative
 
-- Confidence is synthetic until the log-prob path lands.
+- Generic-family confidence remains constant until each score path passes its
+  real-data quality gate; LibreMODUS's constrained-token score is uncalibrated.
 - Generation is slower and less deterministic than a detector forward.
 - Adds `transformers` (already an optional extra) to the `vlm` extra.
 
@@ -191,4 +268,9 @@ executing mutable upstream model-repository code under the same alias.
   SmolVLM2, Gemma 4). Florence-2 and Kosmos-2 use task / grounding tokens.
   Moondream uses native detect/point skills. See the Available-models table in
   [`../librevlm_design.md`](../librevlm_design.md).
+- CLI alias/checkpoint/immutable-artifact routing for VLM prediction and
+  verified Qwen3-VL 2B/4B training, with pre-load guards for remote training
+  and unsupported validation, export, and quantization.
+- Strict, reviewed Qwen3-VL LoRA artifact construction and immutable Hub
+  transport, separate from detector checkpoint publication.
 - Offline parser unit tests plus a `vlm`-marked end-to-end smoke test.

docs/checkpoint_schema.md

@@ -328,6 +328,36 @@ distributed as training checkpoints.
 For release compatibility, readers accept legacy best-metric aliases such as
 `best_mAP50_95`, `best_mAP50`, `best_metric`, and `best_metric_name`.
 
+## VLM directory checkpoints and Hub artifacts
+
+LibreVLM training outputs directories rather than native detector `.pt` files.
+Each local directory carries `libreyolo_vlm.json` schema 1, which records the
+family, size, task, ordered vocabulary, prompt and box conventions, metrics,
+LibreYOLO version, and exact base repository revision. A LoRA checkpoint also
+contains `adapter_model.safetensors`, `adapter_config.json`, and its processor
+files. A full fine-tune instead contains a self-contained model directory.
+Both are local training/reload formats, not publication approval.
+
+`libreyolo.vlm-artifact.v1` is a separate, flat publication format restricted
+to the verified Qwen3-VL 2B/4B detection LoRA cohort. Its
+`libreyolo_vlm_artifact.json` manifest binds the exact payload inventory and
+the adapter, contract, processor, evidence, and immutable base-snapshot
+identities. Base weights are referenced and verified but are not included;
+the exact Qwen processor, tokenizer, and chat-template assets are redistributed
+under Apache-2.0 with generated license and notice files.
+
+The artifact carries reviewed `libreyolo.vlm-publication-evidence.v2` evidence.
+Its evaluation record binds one primary confidence report and envelope plus a
+canonical `libreyolo.vlm-confidence-repeatability-receipt.v1` comparison of
+that primary run with a second fresh-process run. Both the raw receipt and its
+canonical comparison object are SHA-256 bound. These are structural integrity
+records, not publisher or reviewer authentication.
+
+Published artifacts are addressed as
+`hf+vlm://owner/repo@<40-character-commit>`. This is distinct from the
+single-file detector `hf://` transport. The evidence and Hub workflow are
+defined in [`vlm_hub_artifact.md`](vlm_hub_artifact.md).
+
 ## External Snapshot Exception
 
 The schema above governs LibreYOLO-authored `.pt` checkpoints. It does not

docs/fiftyone.md

@@ -2,8 +2,8 @@
 
 [FiftyOne](https://github.com/voxel51/fiftyone) (Apache-2.0) is a dataset
 curation and prediction-analysis tool. `libreyolo.integrations.fiftyone` sends
-LibreYOLO predictions into a FiftyOne dataset and moves datasets in both
-directions between a LibreYOLO dataset yaml and a FiftyOne dataset.
+LibreYOLO and LibreVLM predictions into a FiftyOne dataset and moves datasets
+in both directions between a LibreYOLO dataset yaml and a FiftyOne dataset.
 
 Nothing is vendored and nothing is imported at `import libreyolo` time. The
 module imports `fiftyone` lazily and raises an install hint when it is absent.
@@ -41,11 +41,35 @@ apply_model(dataset, model, label_field="predictions", conf=0.25, batch_size=8)
 session = fo.launch_app(dataset)
 ```
 
-`apply_model` accepts a loaded model or a checkpoint name, forwards `conf`,
-`iou`, `imgsz`, `device`, `classes`, and `max_det` to `model.predict`, and runs
-through FiftyOne's own `apply_model`, so the progress bar, `skip_failures`, and
-batching behave the way FiftyOne users expect. `batch_size` maps onto
-LibreYOLO's batched list inference: one stacked forward per chunk.
+For open-vocabulary VLM prediction, configure the vocabulary on the loaded
+model first:
+
+```python
+from libreyolo import LibreVLM
+
+model = LibreVLM("qwen3-vl-4b")
+model.set_classes(["forklift", "worker"])
+apply_model(dataset, model, label_field="vlm_predictions", batch_size=8)
+```
+
+`apply_model` accepts a loaded model, a detector checkpoint/name, a VLM alias,
+or a VLM checkpoint directory. It forwards `conf`, `iou`, `device`, `classes`,
+and `max_det` to `model.predict`. Detector callers may also override `imgsz`;
+LibreVLM callers must omit it because each VLM's processor owns image resizing.
+The integration runs through FiftyOne's own `apply_model`, so the progress bar
+and `skip_failures` behave the way FiftyOne users expect. `batch_size` maps onto
+LibreYOLO's batched list inference when the model supports it. Generative
+`LibreVLM` adapters currently opt out of batched prediction, so FiftyOne may
+still form chunks but the VLM generates one image at a time; increasing
+`batch_size` does not improve VLM throughput.
+
+`LibreVLM` detections use the same standard `Results` conversion shown below.
+Generic chat VLMs currently expose an uncalibrated constant confidence of
+`1.0`, so confidence ranking, mistakenness, and threshold-based curation are not
+meaningful for those families until their documented real-data score gate
+passes. The candidate Qwen3-VL token score is internal and is not exposed to
+FiftyOne. Box geometry, labels, numeric class filtering, and visualization
+remain valid.
 
 To use the model with any other FiftyOne API that takes a model, wrap it
 directly: