Repository navigation
epic: add Florence-2 (florence2) VLM support #850
Description
Summary
Florence-2 (microsoft/Florence-2-large / -base) is a unified multi-task vision seq2seq model. IMPORTANT: it is architecturally new for mlxcel. A DaViT vision backbone (model_type: davit) produces image tokens that are concatenated with the task-prompt tokens and fed to a BART-style encoder-decoder transformer (a Florence2Encoder plus a Florence2Decoder with cross-attention from decoder to encoder outputs), NOT a decoder-only stack. One text-generation interface covers captioning, OCR, detection, dense region captioning, grounding, and segmentation; task-prompt tokens select the mode, and spatial tasks emit location tokens that post-process into boxes / regions. This is the highest-risk item in the batch and should be scoped as its own epic.
Concretizes the Florence-2 entry recorded as out-of-scope-pending-seq2seq groundwork in #524.
This is the tracking epic for Florence-2 support. This repository has no type:epic label, so the issue keeps type:enhancement while serving as the epic. GitHub native sub-issue linking is not used here; the sub-issues below are tracked by the checklist in this issue.
Sub-issues (execution order)
Phase 1 (parallel foundations)
- feat(models): Florence-2 BART-style seq2seq encoder-decoder engine + text core (sub of #850) #852 BART-style seq2seq encoder-decoder engine + text core
- feat(models): Florence-2 DaViT vision backbone (sub of #850) #853 DaViT vision backbone
Phase 2 (fusion)
- feat(models): Florence-2 vision-language fusion + full weight loading (sub of #850) #854 Vision-language fusion + full weight loading (needs feat(models): Florence-2 BART-style seq2seq encoder-decoder engine + text core (sub of #850) #852, feat(models): Florence-2 DaViT vision backbone (sub of #850) #853)
Phase 3 (modes + capstone)
- feat(models): Florence-2 processor, task prompts, and location tokens (sub of #850) #855 Processor + task prompts + location tokens (needs feat(models): Florence-2 vision-language fusion + full weight loading (sub of #850) #854)
- feat(models): Florence-2 end-to-end integration and real-checkpoint validation (sub of #850) #856 End-to-end integration + real-checkpoint validation (needs feat(models): Florence-2 BART-style seq2seq encoder-decoder engine + text core (sub of #850) #852, feat(models): Florence-2 DaViT vision backbone (sub of #850) #853, feat(models): Florence-2 vision-language fusion + full weight loading (sub of #850) #854, feat(models): Florence-2 processor, task prompts, and location tokens (sub of #850) #855)
Upstream reference
- Directory: https://github.com/Blaizzy/mlx-vlm/tree/main/mlx_vlm/models/florence2
- Top-level model (encoder-decoder wiring): https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/models/florence2/florence2.py
- Language (BART-style encoder + decoder): https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/models/florence2/language.py
- Vision (DaViT backbone): https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/models/florence2/vision.py
- Processor (task prompts): https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/models/florence2/processing_florence2.py
- Config: https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/models/florence2/config.py
Public checkpoint
Confirmed available on HuggingFace:
microsoft/Florence-2-large,microsoft/Florence-2-base(originals)- mlx-community conversions confirmed:
mlx-community/Florence-2-large-ft-4bit/-8bit/-6bit/-bf16andmlx-community/Florence-2-base-ft-4bit/-8bit/-6bit/-3bit/-bf16(primary validation target: a-bf16or-4bitft variant).
Architecture notes
model_type: florence2with adavitvision sub-config.- Vision: DaViT encoder with windowed attention (window_partition / window_reverse), ChannelAttention, ConvEmbed, DepthWiseConv2d, learned absolute positions.
- Language: BART-style encoder-decoder (6 encoder + 6 decoder layers, learned absolute positions). Image tokens are concatenated with prompt tokens, the encoder runs over the joint sequence, and the decoder cross-attends to encoder outputs while decoding autoregressively.
- Task-prompt tokens drive the mode (caption / OCR / detect / ground / segment); spatial tasks emit location tokens post-processed into boxes or regions.
Implementation plan
- PROMINENT CONSTRAINT: mlxcel's generation engine is decoder-only today. Florence-2 requires a NEW encoder-decoder inference path: an encoder pass over the concatenated vision-plus-prompt sequence, then cross-attention decoding against the cached encoder outputs. The only in-tree encoder-decoder precedent is Whisper's dedicated ASR pipeline (
src/models/whisper/); use it as the structural template for a seq2seq decode loop and encoder-output caching, but note Florence-2 needs vision-plus-text encoder input and location-token output rather than audio-to-text. - DaViT backbone: new, but reuse existing window-attention helpers where possible (the windowed-attention patterns in the qwen2_5_vl / SAM-style encoders under
src/vision/encoders/); ConvEmbed / DepthWiseConv2d map to the existing conv layer helpers used by the FastViT / gemma3n conv paths. - Task-prompt handling + location-token post-processing (coordinate tokens to boxes / regions) is a dedicated processing stage; the coordinate / OCR post-processing in the OCR VLM modules is a partial reference.
- RECOMMENDED SCOPING: file this as an epic. Phase 1: seq2seq engine groundwork (encoder-decoder decode loop, cross-attention, encoder-output cache) validated on a minimal path. Phase 2: DaViT backbone. Phase 3: task-prompt modes + location-token post-processing + full multi-task validation. Land Phase 1 as reusable engine infrastructure before the model-specific work.
Touchpoints & acceptance criteria
- NEW encoder-decoder inference path (encoder pass, cross-attention decode, encoder-output cache) landed as reusable engine infrastructure (Whisper as template).
- Text config (BART encoder-decoder) + vision config (
davit) parse. - DaViT vision encoder (windowed attention, window_partition / window_reverse, ChannelAttention, ConvEmbed, DepthWiseConv2d).
- Image-token concatenation with prompt tokens (Florence-2 concatenates vision + prompt for the encoder rather than scattering into placeholders).
-
from_weights+sanitize: encoder-decoder weight naming, DaViT conv weights channels-last remap, mixed precision. - Task-prompt mode handling + location-token to box / region post-processing.
- Arch-string arm
"florence2"insrc/models/detection.rs. - Registration in
src/model_metadata.rs(for_each_model_registration!). -
generate_vlm(or a dedicated seq2seq VLM path) summary + TP / distributed arch-string if applicable. -
_tests.rsunit tests (DaViT shapes, cross-attention, task-prompt parse). -
docs/supported-models.mdupdated. - Real-checkpoint validation:
./target/release/mlxcel generate -m models/Florence-2-base-ft-bf16 --image <img> -p "<CAPTION or OCR or DETECTION task prompt>"produces correct output for at least caption + OCR + detection modes. Followdocs/adding-models.md.
Effort
HIGH, highest-risk item of the batch. The blocker is the seq2seq encoder-decoder engine path, which mlxcel does not have outside Whisper. Recommend an epic with the seq2seq groundwork landed first.
Activity
[-]feat(models): add Florence-2 (florence2) VLM support[/-][+]epic: add Florence-2 (florence2) VLM support[/+]feat(models): Florence-2 BART seq2seq engine and text core (#1060)
feat(models): Florence-2 end-to-end integration and real-checkpoint v…
inureyes commentedon Aug 7, 2026
Epic implementation complete
All five sub-issues are merged and closed. Florence-2 is reachable from the CLI and validated on a real checkpoint.
main:e9f1f191Real-checkpoint validation
Run by the orchestrator against
models/Florence-2-base-ft-bf16through the release binary, on real images rather than the synthetic fixture used inside the sub-issue PRs:<CAPTION>Two cats are sleeping on a pink blanket.<DETAILED_CAPTION>In this image there are two cats sleeping on the pink color cloth. Beside them there are remotes.<OD>cat: [17.0, 54.0, 318.4, 471.6],cat: [346.6, 25.7, 639.0, 372.7],couch: [0.3, 0.2, 639.0, 479.3],remote: [40.6, 73.7, 175.7, 116.9],remote: [333.1, 76.6, 370.2, 186.5]<CAPTION>A green car parked in front of a yellow building.<OCR>HELLOWORLD<OCR_WITH_REGION>HELLO: [14.5, 37.5, 565.5, 37.5, 565.5, 177.5, 14.5, 177.5],WORLD: [9.5, 248.5, 632.5, 248.5, 632.5, 387.5, 9.5, 387.5]mlxcel archreportsFlorence-2 (DaViT + BART seq2seq, task prompts, bf16/f16 only).This step was added because #856 validated against
tests/fixtures/test_image.png, a 679-byte synthetic PNG. Its answers (unanswerablefor caption, a single image-spanning box for detection) are byte-identical to the mlx-vlm reference, which proves the wiring is faithful but does not demonstrate the "correct output" the epic asks for. Wiring fidelity and output usefulness are separate claims and both are now evidenced.Verification on final
mainscripts/ci/check_cross_repo_refs.pyscripts/ci/check_crate_versions.pyscripts/ci/check_kernel_dtype_keys.pymake verify-fmtmake verify-clippy(--workspace --all-targets --features metal,accelerate -- -D warnings)make verify-test(--workspace --profile test-fast)The single failure is
loading::vlm::gemma_unified::tests::unified_sanitize_quantized_split_dequant_equivalence, filed as #1074. It is not caused by this epic: none of the five merged PRs touches gemma or quantization code, andsrc/loading/vlm_gemma_unified.rswas last modified by #400. It is also not the Rust 1.97.1 bump from #1066, which was the first hypothesis; compiling the same source withcargo +1.93.1reproduces it with byte-identical values.Corrections to the epic and sub-issue text
Every sub-issue found at least one error in its own issue body. Recording them so the next reader trusts the code over the prose:
vision_config.model_typeis the empty string on real checkpoints, not"davit". Gating on the literal would fail on every published Florence-2 export.hidden_size. The backbone emitsdim_embed[-1]of 1024;projection_dim768 is the fusion-stage projection.window_partition/window_reverselive only insrc/vision/encoders/deepseekocr_sam.rs, not in qwen2_5_vl or kimi_vl as feat(models): Florence-2 DaViT vision backbone (sub of #850) #853 suggested.mlxcel listreports florence2" resolves tomlxcel arch;listis the local model store listing.Two genuine upstream bugs in mlx-vlm were found and deliberately not copied:
ModelConfigreadsimage_feature_source,image_pos_embed, andvisual_temporal_embeddingfrom the top level of config.json while real checkpoints store them undervision_config, and itsimage_feature_sourcedefault is the reverse of the checkpoint's, which would relocate the pooled token from the front of the 577-token sequence to the end. Separately,post_process_generation's docstring callsimage_size"height x width" while the code proves it is(width, height), and the PATTERN inparse_od_from_text_and_spanshas a doubled backslash that makes it unmatchable.Known limitations, with follow-ups filed
docs/supported-models.mdstates this plainly.mlxcel-serverrefuses the checkpoint at startup (feat(server): serve Florence-2 through a seq2seq worker loop #1073). The server worker loop is decoder-only and would serve garbage for a seq2seq family, so the refusal is correct interim behavior. feat(server): serve Florence-2 through a seq2seq worker loop #1073 also carries the two security handoffs from feat(models): Florence-2 processor, task prompts, and location tokens (sub of #850) #855 and an open design question about where structured outputs (boxes, polygons, OCR regions) belong in the response schema.Execution note
The approved plan ran wave 0 as two parallel worktrees. That was abandoned after measurement: relocating
target/to a new path costs a full MLX C++ plus workspace rebuild per cargo profile (10m04s measured), and the MLX C++ build is per-profile, so making one worktree usable by a sub-agent costs roughly 20 minutes against a 600-second stream-idle watchdog. All five units ran sequentially in the pre-warmed primary working tree instead. No worktrees were created, so none needed cleanup.One unit still stalled: #1066 bumped the pinned Rust toolchain from 1.93.1 to 1.97.1 partway through the epic, invalidating the warmed cache, and #856's release build went cold and exceeded the watchdog during the silent fat-LTO link. The work was committed and the unit resumed without loss. The lesson for the next run is that a warmed cache is invalidated by the toolchain pin, not only by the commit range.
inureyes commentedon Aug 7, 2026
Epic stays open pending the two carved-out follow-ups
All five sub-issues (#852, #853, #854, #855, #856) are merged and closed, and Florence-2 works from the CLI on a real checkpoint. The epic itself stays open because two of its touchpoints are not met and were split into their own issues rather than dropped:
mlxcel-serverrefuses the checkpoint at startup, so "reachable from the CLI and server" is only half satisfied.Label moved from
status:donetostatus:in-progressto reflect that. Closing this epic waits on both.Not blocking this epic: #1074 is a pre-existing gemma4_unified defect unrelated to Florence-2, recorded here only because it is the one red test on
mainat the time of this run.