feat(models): Florence-2 end-to-end integration and real-checkpoint validation

docs/supported-models.md

@@ -118,6 +118,7 @@ Implemented VLM variants include:
 - MiniCPM-O
 - Moondream 3
 - Moondream 2 (`moondream2` / `moondream1`): reuses Moondream3's linear-patch ViT vision tower and overlap-crop preprocessor, paired with a Phi-1.5-style dense text decoder (fused QKV, partial rotary embedding, parallel attention/MLP, tanh-GELU) instead of Moondream3's sparse-MoE decoder. Images are split into a resized global crop and a grid of overlapping local crops; the local crop features are trimmed of their overlap margins, stitched, and adaptively average-pooled back to the 27x27 encoder grid before being concatenated with the global features and projected to the text hidden size. The BOS token and the 729 projected image tokens form a bidirectional prefix ahead of the causal text prompt. Checkpoint revisions from 2025-06-21 onwards are trained against the `moondream/starmie-v1` tokenizer with Moondream3-style control-token templates (bos = eos = 0), while the official repository still ships the older GPT-2 `tokenizer.json` next to them; mlxcel detects the revision from the bundled `moondream.py`, resolves the starmie tokenizer from the Hub (cached after the first fetch, or place starmie's `tokenizer.json` in the model directory when offline), and keeps the GPT-2 tokenizer with `Question:`/`Answer:` framing for the 2025-01-09 .. 2025-04-14 revisions where that contract is the correct one.
+- Florence-2 (`florence2`): Microsoft's task-prompted vision foundation model, an encoder-decoder (seq2seq) VLM. A DaViT tower (4 stages, window + grid attention) encodes the 768x768 image into a `[1, 576, 1024]` feature grid; a learned 2D position embedding, a cosine temporal embedding, and the `image_feature_source` pooling recipe produce 577 projected tokens that are concatenated in front of the task-prompt embeddings (no image placeholder token). The fused sequence runs through a BART encoder, and a causal BART decoder with encoder cross-attention generates the answer greedily from `decoder_start_token_id`. The `-p/--prompt` string selects one of fifteen task modes, for example `<CAPTION>`, `<DETAILED_CAPTION>`, `<OCR>`, `<OCR_WITH_REGION>`, `<OD>`, `<DENSE_REGION_CAPTION>`, `<REGION_PROPOSAL>`, or `<CAPTION_TO_PHRASE_GROUNDING> some text` for the seven tasks that take input text; markers are case-insensitive and the angle brackets are optional. Spatial answers are decoded through the checkpoint's 1000-bin `<loc_*>` tokens and parsed into pixel-space boxes, quad boxes, or polygons against the original image size. Because generation is seq2seq (cross-attention against cached encoder output), the family runs through its own CLI pipeline (`mlxcel generate -m <model> --image <img> -p '<OD>'`) rather than the shared autoregressive loop, and `mlxcel-server` refuses the checkpoint at startup for now. bf16/f16 exports only, one image per request: use `mlx-community/Florence-2-base-ft-bf16` (or `-large-ft-bf16`). The 4-bit / 8-bit / 6-bit / 3-bit mlx-community conversions are rejected at load with a named error, because neither the BART stack nor the DaViT tower has a quantized code path yet.
 - Phi-3 Vision, Phi4MM, Phi4 SigLIP VLM
 - Molmo2 and Molmo-Point
 - Nemotron-H Nano Omni: ships a Conformer/Parakeet audio encoder and accepts spoken audio from the CLI with `--audio <path>`. Input audio is resampled to 16 kHz before the encoder. Server-side `input_audio` in `POST /v1/chat/completions` is also supported: the audio block is spliced inside the last user turn, before the `<|im_end|>` end-of-turn marker that the ChatML template uses (id 151 in the released checkpoint).

src/commands/generate.rs

@@ -2077,6 +2077,18 @@ fn run_generate_once(mut args: GenerateArgs) -> Result<()> {
                 &user_prompt,
             );
         }
+        // Florence-2 is an encoder-decoder (seq2seq) VLM: the decoder
+        // cross-attends to cached encoder output over the fused image+prompt
+        // sequence, so route it to its task pipeline before the
+        // autoregressive loop (issue #856). The raw `-p` string is the task
+        // prompt; the tokenized chat-template form above does not apply.
+        if let mlxcel::LoadedModel::Florence2VLM(florence2_model) = &model {
+            return super::generate_florence2::run_florence2_generation(
+                florence2_model,
+                &args,
+                &user_prompt,
+            );
+        }
         // Reject an off-ladder `--image-soft-tokens` before loading any image:
         // the budget drives the resize target, so an unsupported value is a
         // user error, not something to clamp silently.

src/commands/generate_florence2.rs

@@ -0,0 +1,286 @@
+// Copyright 2025-2026 Lablup Inc. and Jeongkyu Shin
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! CLI driver for Florence-2 task generation (issue #856).
+//!
+//! Florence-2 is an encoder-decoder (seq2seq) VLM: the encoder consumes the
+//! fused image-plus-prompt sequence and the decoder cross-attends to it, so
+//! it cannot run on the autoregressive loop `run_generation_mode` drives.
+//! `run_generate_once` routes `LoadedModel::Florence2VLM` here before that
+//! loop, mirroring the DiffusionGemma / LLaDA-2 early exits.
+//!
+//! The `-p/--prompt` string selects one of the fifteen task modes
+//! (`<CAPTION>`, `<OCR>`, `<OD>`, ...), optionally followed by the input
+//! text the task interpolates. Image decoding routes through the shared
+//! [`mlxcel::ImageInputLimits`] admission bounds so an oversized or
+//! decompression-bomb payload is rejected before any pixel work.
+
+use std::time::Instant;
+
+use anyhow::{Context, Result, anyhow, ensure};
+
+use mlxcel::models::florence2::{Florence2TaskResult, Florence2VlmModel, parse_task_prompt};
+
+use super::generate::print_generation_preamble;
+use crate::GenerateArgs;
+
+/// Run one Florence-2 task from the CLI flag surface and print the parsed
+/// answer plus a generation-stats line.
+pub(crate) fn run_florence2_generation(
+    model: &Florence2VlmModel,
+    args: &GenerateArgs,
+    user_prompt: &str,
+) -> Result<()> {
+    ensure!(
+        args.generation.audio.is_none(),
+        "Florence-2 does not take --audio input"
+    );
+    ensure!(
+        args.generation.video.is_empty(),
+        "Florence-2 does not take --video input"
+    );
+    ensure!(
+        !args.generation.image.is_empty(),
+        "Florence-2 is an image-task model: pass --image <path> together with a task prompt \
+         such as -p '<CAPTION>', -p '<OCR>', or -p '<OD>'"
+    );
+    ensure!(
+        args.generation.image.len() == 1,
+        "Florence-2 processes one image per request; got {} --image paths",
+        args.generation.image.len()
+    );
+
+    let (task, input) = parse_task_prompt(user_prompt).map_err(|e| anyhow!("-p/--prompt: {e}"))?;
+
+    // Decode through the shared admission limits (decompression-bomb
+    // defense, issue #855 handoff): `preprocess_with_sizes` takes an
+    // already-decoded image, so the bound has to hold at this boundary.
+    let image_path = &args.generation.image[0];
+    let bytes = std::fs::read(image_path)
+        .with_context(|| format!("Failed to read image {image_path:?}"))?;
+    let mut images =
+        mlxcel::decode_image_payloads_with_limits(&[bytes], mlxcel::current_image_input_limits())
+            .with_context(|| format!("Failed to decode image {image_path:?}"))?;
+    let image = images
+        .pop()
+        .ok_or_else(|| anyhow!("image decoding returned no image for {image_path:?}"))?;
+
+    print_generation_preamble(user_prompt)?;
+    println!();
+
+    let started = Instant::now();
+    let run = model.run_task(task, input.as_deref(), &image, args.generation.max_tokens)?;
+    let elapsed = started.elapsed().as_secs_f64();
+
+    println!(
+        "{}",
+        render_task_result(&run.output.result, &run.output.raw_text)
+    );
+    println!();
+    let tps = if elapsed > 0.0 {
+        run.generated_tokens as f64 / elapsed
+    } else {
+        0.0
+    };
+    println!(
+        "[Generated {} tokens in {:.2}s = {:.2} tok/s]",
+        run.generated_tokens, elapsed, tps
+    );
+    if args.generation.profile {
+        println!("[Raw answer] {}", run.output.raw_text);
+    }
+
+    mlxcel_core::clear_memory_cache();
+    Ok(())
+}
+
+/// Render a parsed task result for the terminal.
+///
+/// Spatial results are printed one instance per line as
+/// `label: [coordinates]` in original-image pixels. An answer that parsed to
+/// nothing falls back to the raw decoded text, which is the only way to tell
+/// "the model found nothing" from "the parser rejected the answer".
+fn render_task_result(result: &Florence2TaskResult, raw_text: &str) -> String {
+    let rendered = match result {
+        Florence2TaskResult::Text(text) => text.trim().to_string(),
+        Florence2TaskResult::Boxes { boxes, labels } => boxes
+            .iter()
+            .zip(labels)
+            .map(|(bbox, label)| {
+                format!(
+                    "{}: [{:.1}, {:.1}, {:.1}, {:.1}]",
+                    display_label(label),
+                    bbox.xmin,
+                    bbox.ymin,
+                    bbox.xmax,
+                    bbox.ymax
+                )
+            })
+            .collect::<Vec<_>>()
+            .join("\n"),
+        Florence2TaskResult::QuadBoxes { quad_boxes, labels } => quad_boxes
+            .iter()
+            .zip(labels)
+            .map(|(quad, label)| {
+                format!("{}: [{}]", display_label(label), join_points(&quad.points))
+            })
+            .collect::<Vec<_>>()
+            .join("\n"),
+        Florence2TaskResult::Polygons { polygons, labels } => {
+            render_polygon_instances(polygons, labels)
+        }
+        Florence2TaskResult::BoxesOrPolygons {
+            boxes,
+            box_labels,
+            polygons,
+            polygon_labels,
+        } => {
+            let mut lines: Vec<String> = boxes
+                .iter()
+                .zip(box_labels)
+                .map(|(bbox, label)| {
+                    format!(
+                        "{}: [{:.1}, {:.1}, {:.1}, {:.1}]",
+                        display_label(label),
+                        bbox.xmin,
+                        bbox.ymin,
+                        bbox.xmax,
+                        bbox.ymax
+                    )
+                })
+                .collect();
+            let polygon_block = render_polygon_instances(polygons, polygon_labels);
+            if !polygon_block.is_empty() {
+                lines.push(polygon_block);
+            }
+            lines.join("\n")
+        }
+        // `Florence2TaskResult` is `#[non_exhaustive]`: render future
+        // variants through the raw-text fallback below.
+        _ => String::new(),
+    };
+    if rendered.is_empty() {
+        format!("[no parsed instances] raw answer: {}", raw_text.trim())
+    } else {
+        rendered
+    }
+}
+
+fn render_polygon_instances(
+    polygons: &[Vec<mlxcel::models::Florence2Polygon>],
+    labels: &[String],
+) -> String {
+    polygons
+        .iter()
+        .zip(labels)
+        .map(|(outlines, label)| {
+            let rendered_outlines = outlines
+                .iter()
+                .map(|polygon| format!("[{}]", join_points(&polygon.points)))
+                .collect::<Vec<_>>()
+                .join(", ");
+            format!("{}: {}", display_label(label), rendered_outlines)
+        })
+        .collect::<Vec<_>>()
+        .join("\n")
+}
+
+fn join_points(points: &[f32]) -> String {
+    points
+        .iter()
+        .map(|point| format!("{point:.1}"))
+        .collect::<Vec<_>>()
+        .join(", ")
+}
+
+/// `<REGION_PROPOSAL>` predicts unnamed regions; keep the line shape stable.
+fn display_label(label: &str) -> &str {
+    if label.is_empty() { "(region)" } else { label }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use mlxcel::models::{Florence2BoundingBox, Florence2Polygon, Florence2QuadBox};
+
+    #[test]
+    fn renders_text_result() {
+        let result = Florence2TaskResult::Text("A car parked on the street.".to_string());
+        assert_eq!(
+            render_task_result(&result, ""),
+            "A car parked on the street."
+        );
+    }
+
+    #[test]
+    fn renders_boxes_one_instance_per_line() {
+        let result = Florence2TaskResult::Boxes {
+            boxes: vec![
+                Florence2BoundingBox {
+                    xmin: 1.0,
+                    ymin: 2.0,
+                    xmax: 3.0,
+                    ymax: 4.0,
+                },
+                Florence2BoundingBox {
+                    xmin: 5.0,
+                    ymin: 6.0,
+                    xmax: 7.0,
+                    ymax: 8.0,
+                },
+            ],
+            labels: vec!["car".to_string(), String::new()],
+        };
+        let rendered = render_task_result(&result, "");
+        assert_eq!(
+            rendered,
+            "car: [1.0, 2.0, 3.0, 4.0]\n(region): [5.0, 6.0, 7.0, 8.0]"
+        );
+    }
+
+    #[test]
+    fn renders_quad_boxes_and_polygons() {
+        let quads = Florence2TaskResult::QuadBoxes {
+            quad_boxes: vec![Florence2QuadBox {
+                points: [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0],
+            }],
+            labels: vec!["HELLO".to_string()],
+        };
+        assert_eq!(
+            render_task_result(&quads, ""),
+            "HELLO: [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0]"
+        );
+
+        let polygons = Florence2TaskResult::Polygons {
+            polygons: vec![vec![Florence2Polygon {
+                points: vec![1.0, 2.0, 3.0, 4.0],
+            }]],
+            labels: vec!["dog".to_string()],
+        };
+        assert_eq!(
+            render_task_result(&polygons, ""),
+            "dog: [1.0, 2.0, 3.0, 4.0]"
+        );
+    }
+
+    #[test]
+    fn empty_result_falls_back_to_raw_text() {
+        let result = Florence2TaskResult::Boxes {
+            boxes: vec![],
+            labels: vec![],
+        };
+        let rendered = render_task_result(&result, "<s></s>");
+        assert!(rendered.contains("raw answer"), "got: {rendered}");
+    }
+}

src/commands/mod.rs

@@ -23,6 +23,7 @@ pub(crate) mod detect;
 pub(crate) mod download;
 pub(crate) mod generate;
 mod generate_diffusion;
+mod generate_florence2;
 mod generate_llada2;
 mod generate_vlm;
 pub(crate) mod inspect;

src/distributed/tensor_parallel/inference.rs

@@ -400,6 +400,11 @@ fn fallback_architecture(model_type: ModelType) -> &'static str {
         // worker loop, never routed to tensor-parallel text inference; the
         // planner's supported-architecture validation rejects this string.
         ModelType::Llada2Moe => "llada2_moe",
+        // Florence-2 is an encoder-decoder (seq2seq) VLM served through its
+        // own CLI task pipeline, never routed to tensor-parallel text
+        // inference; the planner's supported-architecture validation rejects
+        // this string. Placeholder keeps the dispatch table total.
+        ModelType::Florence2VLM => "florence2",
         // Whisper is an ASR model served through the audio endpoints, never
         // routed to tensor-parallel text inference; the loader rejects it
         // earlier. Return a placeholder so the dispatch table stays total.

src/loaded_model.rs

@@ -112,6 +112,11 @@ pub enum LoadedModel {
     MiniCPMV46VLM(vision::MiniCPMV46VLModel),
     Moondream3VLM(vision::Moondream3VLModel),
     Moondream2VLM(vision::Moondream2VLModel),
+    /// Florence-2 (issue #856): DaViT tower + BART encoder-decoder text
+    /// stack. Seq2seq, so the CLI routes it to the Florence-2 task pipeline
+    /// before the autoregressive loop and the server refuses it at startup;
+    /// the `LanguageModel` delegation below exists for trait completeness.
+    Florence2VLM(models::Florence2VlmModel),
     Gemma3n(models::Gemma3nModel),
     Gemma3nVLM(vision::Gemma3nVLModel),
     Phi(models::PhiModel),
@@ -271,6 +276,7 @@ macro_rules! delegate_language_model {
             LoadedModel::MiniCPMV46VLM(inner) => LanguageModel::$method(inner, $($arg),*),
             LoadedModel::Moondream3VLM(inner) => LanguageModel::$method(inner, $($arg),*),
             LoadedModel::Moondream2VLM(inner) => LanguageModel::$method(inner, $($arg),*),
+            LoadedModel::Florence2VLM(inner) => LanguageModel::$method(inner, $($arg),*),
             LoadedModel::Gemma3n(inner) => LanguageModel::$method(inner, $($arg),*),
             LoadedModel::Gemma3nVLM(inner) => LanguageModel::$method(inner, $($arg),*),
             LoadedModel::Phi(inner) => LanguageModel::$method(inner, $($arg),*),

src/loading/mod.rs

@@ -217,6 +217,7 @@ fn try_load_vlm_model_from_dir(
         ModelType::MiniCPMV46VLM => Some(load_minicpmv4_6_vlm(model_path)?),
         ModelType::Moondream3VLM => Some(load_moondream3_vlm(model_path)?),
         ModelType::Moondream2VLM => Some(load_moondream2_vlm(model_path)?),
+        ModelType::Florence2VLM => Some(load_florence2_vlm(model_path)?),
         ModelType::Gemma3nVLM => Some(load_gemma3n_vlm(model_path)?),
         ModelType::Phi4MMVLM => Some(load_phi4mm_vlm(model_path)?),
         ModelType::Phi4SigLipVLM => Some(load_phi4_siglip_vlm(model_path)?),

src/loading/vlm.rs

@@ -47,6 +47,8 @@ mod dots_ocr;
 mod ernie4_5_vl;
 #[path = "vlm_fastvlm.rs"]
 mod fastvlm;
+#[path = "vlm_florence2.rs"]
+mod florence2;
 #[path = "vlm_gemma.rs"]
 mod gemma;
 #[path = "vlm_gemma_unified.rs"]
@@ -97,6 +99,7 @@ pub(crate) use deepseekocr::{
 pub(crate) use dots_ocr::load_dots_ocr_vl;
 pub(crate) use ernie4_5_vl::load_ernie4_5_moe_vlm;
 pub(crate) use fastvlm::load_fastvlm_vlm;
+pub(crate) use florence2::load_florence2_vlm;
 pub(crate) use gemma::{load_gemma3_vlm, load_gemma3n_vlm, load_gemma4_vlm};
 pub(crate) use gemma_unified::load_gemma4_unified;
 pub(crate) use granite_vision::load_granite_vision_vlm;